Text feature extraction method based on keyword frequency
By constructing keyword training sets, calculating feature similarity and frequency, sorting and matching analysis, the problem of insufficient precision of rare entry removal in the prior art is solved, and the accuracy of text feature extraction is improved.
Patent Information
- Application Number
- CN202411136995.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-19
- Publication Date
- 2024-11-08
AI Technical Summary
The existing text feature extraction method based on keyword frequency has insufficient accuracy in the process of removing rare entries, resulting in direct filtering of rare entries because they are below the threshold, affecting the accuracy of text feature extraction.
By constructing a training set of multiple categories of keywords, the similarity between features is calculated, the frequency of high-similarity feature is summarized, the low-similarity features are not summarized, and the keyword frequency is high and low-similarity features are sorted after sorting, and the characteristics of rare entries are analyzed according to matching. , determine whether to remove it and generate the final valid content.
The process of removing rare entries is optimized to prevent direct filtering due to threshold values, and to improve the accuracy of text feature extraction.
Smart Images

Figure CN118916787A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text feature extraction, and more specifically, to a text feature extraction method based on keyword frequency. Background Art
[0002] With the rapid development of Internet technology, big data is a manifestation or feature of the Internet development to the current stage. On the one hand, the development of the Internet provides more data, information and resources for the development of big data. On the other hand, the development of big data provides more support, services and applications for the development of the Internet. In the field of SMS applications, how to timely detect suspicious content in the massive SMS data and then intercept it in time is an important task of text processing. The existing text feature extraction method based on keyword frequency has the following shortcomings. Since the elimination process of rare terms is relatively simple, if the rare terms mainly appear in a certain category training set, they can well reflect the characteristics of the category training set, but because they are below a certain set threshold, they are directly filtered out, which affects the accuracy of text feature extraction. In view of this, we propose a text feature extraction method based on keyword frequency to solve the above problems. Summary of the invention
[0003] In order to overcome the deficiencies of the prior art, the present invention provides a text feature extraction method based on keyword frequency to solve the problems raised in the above background technology.
[0004] To achieve the above object, the present invention provides the following technical solutions: A text feature extraction method based on keyword frequency includes the following steps: SP01: Build a training set containing multiple category keywords, generate each feature in the set, and obtain the similarity between the features; SP02: For each feature in the set, calculate the number of times it appears in the entire keyword set. The feature keyword frequencies with high similarity are summarized to obtain the actual keyword frequency. The feature keyword frequencies with low similarity do not need to be summarized to obtain the actual keyword frequency. SP03: Sort the actual keyword frequencies from high to low, and select the features with high keyword frequencies to form a new feature set; SP04: Select the features with low keyword frequency as rare term features, and match and analyze the rare term features with the features of the training set of the category to which they belong; SP05: Determine whether to remove rare term features based on the matching analysis results; SP06: Generate final valid content based on the rare terms judgment results, and recommend the final valid content to users.
[0005] As a preferred technical solution of the present invention, the features in the generated set in SP01 are specifically converted into feature vector operations in the vector space by processing the keyword training set, and the semantic similarity is expressed by the spatial similarity.
[0006] As a preferred technical solution of the present invention, the feature vector includes a feature vector given by a user and an unknown feature vector, the feature vector given by the user is represented by Dp, the unknown feature vector is represented by Dx, the similarity is represented by Y, and the calculation formula of the similarity is as follows: ,in is the angle between Dp and Dx.
[0007] As a preferred technical solution of the present invention, the specific process of SP02 includes the following steps: SP11: Calculate the number of times each feature appears in the keyword training set; SP12: Calculate the number of occurrences of each feature to obtain the corresponding feature keyword frequency and analyze the similarity between each feature; SP13: If the similarity is higher than the similarity preset value, the similarity between the features is determined to be high, and the feature keyword frequencies are summarized to obtain the actual keyword frequency; SP14: If the similarity is lower than the similarity preset value, it is determined that the similarity between the features is low, and the feature keyword frequencies do not need to be summarized to obtain the actual keyword frequencies.
[0008] As a preferred technical solution of the present invention, the calculation formula of the characteristic keyword frequency is as follows: ,in is the number of times the feature appears in the keyword training set, The number of all terms in the keyword training set.
[0009] As a preferred technical solution of the present invention, the specific process of SP03 includes the following steps: SP21: Compare the actual keyword frequencies and process the actual keyword frequencies in a descending order; SP22: Analyze the actual keyword frequency after sorting to sort out the features with high keyword frequency and low keyword frequency; SP23: If the actual keyword frequency is higher than the preset value of the word frequency, the actual keyword frequency is determined to be high, and the features with high keyword frequency are selected to form a new feature set; SP24: If the actual keyword frequency is lower than the preset value of the word frequency, the actual keyword frequency is determined to be low, and the features of the low keyword frequency are selected.
[0010] As a preferred technical solution of the present invention, the specific process of SP04 includes the following steps: SP31: Take the low frequency features of keywords in SP24 as rare term features, and identify the rare term feature information and the feature information of the category training set; SP32: Based on the recognition results, the rare term features are correlated with the features of the training set of the category to be matched and analyzed to find out the correlation between the rare term features and the features of the training set of the category to be matched.
[0011] As a preferred technical solution of the present invention, the specific process of SP05 includes the following steps: SP41: Receive and read the matching analysis results in SP32, and compare the correlation between the rare term feature and the feature of the category training set with the correlation preset value; SP42: Generate corresponding judgment results based on the comparison information.
[0012] As a preferred technical solution of the present invention, when the correlation degree in SP41 is higher than the correlation preset value, the judgment result is to retain the rare term feature, and the final valid content generated in SP06 is the final valid content with the rare term feature.
[0013] As a preferred technical solution of the present invention, when the correlation degree in SP41 is lower than the preset correlation value, the judgment result is to remove the rare term features, and the final valid content generated in SP06 is the final valid content after filtering the rare term features.
[0014] Compared with the prior art, the present invention has the following beneficial effects: The present invention constructs a training set of keywords including multiple categories, generates each feature in the set, obtains the similarity between the features, calculates the number of times each feature in the set appears in the entire keyword set, summarizes the feature keyword frequencies with high similarity to obtain the actual keyword frequency, and does not need to summarize the feature keyword frequencies with low similarity to obtain the actual keyword frequency, sorts the actual keyword frequencies from high to low, selects the features with high keyword frequencies to form a new feature set, selects the features with low keyword frequencies as rare term features, matches and analyzes the rare term features with the features of the category training set, determines whether to remove the rare term features according to the matching analysis results, generates the final valid content according to the rare term judgment results, thereby optimizing the removal process of the rare terms, preventing the rare terms from being directly filtered out due to being below a set threshold, and is beneficial to improving the accuracy of text feature extraction. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] Figure 1A schematic block diagram of the steps of a text feature extraction method based on keyword frequency of the present invention; Figure 2 It is a schematic block diagram of the specific process steps of SP02 in a text feature extraction method based on keyword frequency of the present invention; Figure 3 It is a schematic block diagram of the specific process steps of SP03 in a text feature extraction method based on keyword frequency of the present invention; Figure 4 It is a schematic block diagram of the specific process steps of SP04 in a text feature extraction method based on keyword frequency of the present invention; Figure 5 The present invention is a schematic block diagram of the specific process steps of SP05 in a text feature extraction method based on keyword frequency. DETAILED DESCRIPTION
[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0017] like Figures 1 to 5 As shown, the present invention provides a text feature extraction method based on keyword frequency, comprising the following steps: SP01: Build a training set containing multiple category keywords, generate each feature in the set, and obtain the similarity between the features; SP02: For each feature in the set, calculate the number of times it appears in the entire keyword set. The feature keyword frequencies with high similarity are summarized to obtain the actual keyword frequency. The feature keyword frequencies with low similarity do not need to be summarized to obtain the actual keyword frequency. SP03: Sort the actual keyword frequencies from high to low, and select the features with high keyword frequencies to form a new feature set; SP04: Select the features with low keyword frequency as rare term features, and match and analyze the rare term features with the features of the training set of the category to which they belong; SP05: Determine whether to remove rare term features based on the matching analysis results; SP06: Generate final valid content based on the rare terms judgment results, and recommend the final valid content to users.
[0018] Among them, each feature in the generated set in SP01 specifically converts the processing of the keyword training set into a feature vector operation in the vector space, and expresses the semantic similarity with the spatial similarity.
[0019] The feature vector includes the feature vector given by the user and the unknown feature vector. The feature vector given by the user is represented by Dp, the unknown feature vector is represented by Dx, the similarity is represented by Y, and the calculation formula of the similarity is as follows: ,in is the angle between Dp and Dx.
[0020] Among them, Figure 2 As shown, the specific process of SP02 includes the following steps: SP11: Calculate the number of times each feature appears in the keyword training set; SP12: Calculate the number of occurrences of each feature to obtain the corresponding feature keyword frequency and analyze the similarity between each feature; SP13: If the similarity is higher than the similarity preset value, the similarity between the features is determined to be high, and the feature keyword frequencies are summarized to obtain the actual keyword frequency; SP14: If the similarity is lower than the similarity preset value, it is determined that the similarity between the features is low, and the feature keyword frequencies do not need to be summarized to obtain the actual keyword frequencies.
[0021] The calculation formula of the characteristic keyword frequency is as follows: ,in is the number of times the feature appears in the keyword training set, The number of all terms in the keyword training set.
[0022] Among them, Figure 3 As shown, the specific process of SP03 includes the following steps: SP21: Compare the actual keyword frequencies and sort them from high to low; SP22: Analyze the actual keyword frequency after sorting to sort out the features with high keyword frequency and low keyword frequency; SP23: If the actual keyword frequency is higher than the preset value of the word frequency, the actual keyword frequency is determined to be high, and the features with high keyword frequency are selected to form a new feature set; SP24: If the actual keyword frequency is lower than the preset value of the word frequency, the actual keyword frequency is determined to be low, and the features of the low keyword frequency are selected.
[0023] Among them, Figure 4 As shown, the specific process of SP04 includes the following steps: SP31: Take the low frequency features of keywords in SP24 as rare term features, and identify the rare term feature information and the feature information of the category training set; SP32: Based on the recognition results, the rare term features are correlated with the features of the training set of the category to be matched and analyzed to find out the correlation between the rare term features and the features of the training set of the category to be matched.
[0024] Among them, Figure 5 As shown, the specific process of SP05 includes the following steps: SP41: Receive and read the matching analysis results in SP32, and compare the correlation between the rare term feature and the feature of the category training set with the correlation preset value; SP42: Generate corresponding judgment results based on the comparison information.
[0025] Among them, if the correlation degree in SP41 is higher than the associated preset value, the judgment result is that the rare term features are retained, and the final valid content generated in SP06 is the final valid content with the rare term features. If the correlation degree in SP41 is lower than the associated preset value, the judgment result is that the rare term features are eliminated, and the final valid content generated in SP06 is the final valid content after filtering the rare term features.
[0026] In summary: The present invention constructs a training set of keywords including multiple categories, generates each feature in the set, obtains the similarity between the features, calculates the number of times each feature in the set appears in the entire keyword set, summarizes the feature keyword frequencies with high similarity to obtain the actual keyword frequency, and does not need to summarize the feature keyword frequencies with low similarity to obtain the actual keyword frequency, sorts the actual keyword frequencies from high to low, selects the features with high keyword frequencies to form a new feature set, selects the features with low keyword frequencies as rare term features, matches and analyzes the rare term features with the features of the category training set, determines whether to remove the rare term features according to the matching analysis results, generates the final valid content according to the rare term judgment results, thereby optimizing the removal process of the rare terms, preventing the rare terms from being directly filtered out due to being below a set threshold, and is beneficial to improving the accuracy of text feature extraction.
[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0028] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A text feature extraction method based on keyword frequency, characterized in that: The following steps are involved: SP01: Build a training set containing multiple category keywords, generate each feature in the set, and obtain the similarity between the features; SP02: For each feature in the set, calculate the number of times it appears in the entire keyword set. The feature keyword frequencies with high similarity are summarized to obtain the actual keyword frequency. The feature keyword frequencies with low similarity do not need to be summarized to obtain the actual keyword frequency. SP03: Sort the actual keyword frequencies from high to low, and select the features with high keyword frequencies to form a new feature set; SP04: Select the features with low keyword frequency as rare term features, and match and analyze the rare term features with the features of the training set of the category to which they belong; SP05: Determine whether to remove rare term features based on the matching analysis results; SP06: Generate final valid content based on the rare terms judgment results, and recommend the final valid content to users.
2. The text feature extraction method based on keyword frequency according to claim 1 is characterized in that: The generation of each feature in the set in SP01 specifically converts the processing of the keyword training set into a feature vector operation in the vector space, and expresses the semantic similarity with the spatial similarity.
3. The text feature extraction method based on keyword frequency according to claim 2 is characterized in that: The feature vector includes a feature vector given by a user and an unknown feature vector. The feature vector given by the user is represented by Dp, the unknown feature vector is represented by Dx, the similarity is represented by Y, and the calculation formula of the similarity is as follows: Y= ,in is the angle between Dp and Dx.
4. The text feature extraction method based on keyword frequency according to claim 3 is characterized in that: The specific process of SP02 includes the following steps: SP11: Calculate the number of times each feature appears in the keyword training set; SP12: Calculate the number of occurrences of each feature to obtain the corresponding feature keyword frequency and analyze the similarity between each feature; SP13: If the similarity is higher than the similarity preset value, the similarity between the features is determined to be high, and the feature keyword frequencies are summarized to obtain the actual keyword frequency; SP14: If the similarity is lower than the similarity preset value, it is determined that the similarity between the features is low, and the feature keyword frequencies do not need to be summarized to obtain the actual keyword frequencies.
5. The text feature extraction method based on keyword frequency according to claim 4 is characterized in that: The calculation formula of the characteristic keyword frequency is as follows: ,in is the number of times the feature appears in the keyword training set, The number of all terms in the keyword training set.
6. The text feature extraction method based on keyword frequency according to claim 4 is characterized in that: The specific process of SP03 includes the following steps: SP21: Compare the actual keyword frequencies and process the actual keyword frequencies in a descending order; SP22: Analyze the actual keyword frequency after sorting to sort out the features with high keyword frequency and low keyword frequency; SP23: If the actual keyword frequency is higher than the preset value of the word frequency, the actual keyword frequency is determined to be high, and the features with high keyword frequency are selected to form a new feature set; SP24: If the actual keyword frequency is lower than the preset value of the word frequency, the actual keyword frequency is determined to be low, and the features of the low keyword frequency are selected.
7. The text feature extraction method based on keyword frequency according to claim 6 is characterized in that: The specific process of SP04 includes the following steps: SP31: Take the low frequency features of keywords in SP24 as rare term features, and identify the rare term feature information and the feature information of the category training set; SP32: Based on the recognition results, the rare term features are correlated with the features of the training set of the category to be matched and analyzed to find out the correlation between the rare term features and the features of the training set of the category to be matched.
8. The text feature extraction method based on keyword frequency according to claim 7 is characterized in that: The specific process of SP05 includes the following steps: SP41: Receive and read the matching analysis results in SP32, and compare the correlation between the rare term feature and the feature of the category training set with the correlation preset value; SP42: Generate corresponding judgment results based on the comparison information.
9. The text feature extraction method based on keyword frequency according to claim 8 is characterized in that: When the relevance in the SP41 is higher than the relevance preset value, the judgment result is to retain the rare term feature, and the final valid content generated in the SP06 is the final valid content with the rare term feature.
10. The text feature extraction method based on keyword frequency according to claim 8, characterized in that: When the relevance in the SP41 is lower than the relevance preset value, the judgment result is to remove the rare term features, and the final valid content generated in the SP06 is the final valid content after filtering the rare term features.
