A text feature extraction method, a text classification method and a readable storage medium
By performing grammatical error correction, entity word extraction and dependency syntax analysis in text feature extraction, combined with normalized TF-IDF calculation, the problem of word segmentation errors and imbalance in text feature extraction and classification in automotive field is solved, and classification is improved accuracy and objectivity.
Patent Information
- Application Number
- CN202210754039.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-28
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-06-28
AI Technical Summary
In the automotive field, there are word participle errors in text feature extraction and classification, diverse synonyms expression forms, grammatical errors, and lack of syntactic analysis, resulting in uneven extraction of feature words, affecting the accuracy of classification.
Grammatical error correction, entity word extraction, dependency syntax analysis, and synonym expression consistency processing are adopted, and the dependency factor is introduced to calculate the TF-IDF value, and converted into the importance weight through normalization to optimize the importance ranking of characteristic words.
It improves the objectivity and accuracy of text feature extraction, ensures the balance of feature word extraction, and improves the accuracy and application effect of text classification.
Smart Images

Figure CN115062151B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of text information extraction, and in particular to a text feature extraction method, a text classification method and a readable storage medium. Background Art
[0002] In the context of the transformation of the automotive industry to intelligent manufacturing, a relatively complete data storage, management and application system has been gradually established in the automotive manufacturing, production, sales and after-sales service. However, due to the diversity of business fields and the diverse scenarios of knowledge generation, text information is difficult to unify into structured data. It is now mostly extracted in the form of documents, which also brings difficulties to the statistical analysis and information extraction of such information. Text extraction technology can be used as an effective solution to this problem.
[0003] The TF-IDF algorithm is a classic method in the field of text extraction technology. It combines word frequency TF and inverse text frequency IDF to calculate the importance of feature words in the target document. However, this method mainly relies on the frequency distribution of words and expresses limited information. Therefore, in actual use, new features or adjustment parameters are also introduced to optimize performance.
[0004] There has been a certain amount of technical research accumulation in the prior art on text feature extraction and text classification based on the TF-IDF algorithm. The Chinese patent with application number 2016108941746 discloses "a short text classification method based on TF-IDF feature extraction", which enhances the TF-IDF features of short texts by merging short texts into long texts, and generates a feature word list and a feature word dictionary by reducing the dimension; at the same time, when establishing a feature word list, a compensation mechanism is established for categories with relatively unclear features, and the weight of the text feature vector is enhanced, thereby improving the algorithm performance while ensuring the expression effect of the text features. The Chinese patent with application number 202011521853.1 discloses "a news keyword extraction method optimized based on the TF-IDF method", which adjusts the weights based on the distribution of the frequency of feature words in the corpus and the position information in the feature word document, so as to improve the accuracy of news data classification.
[0005] However, if the above methods are directly transferred to the text feature extraction and classification problems in the automotive field, there will be the following limitations: 1) There are many proper nouns in the automotive field, which may cause errors in the text segmentation stage; 2) Different descriptive expressions of synonyms are widely present in the texts in the automotive field, which affects the word frequency statistics of synonym features; 3) Incorrect segmentation caused by grammatical errors may cause incorrect segmentation words to occupy a higher weight, affecting the accuracy of text classification; 4) Lacking the combination with syntactic analysis, the weight of some modifiers may be higher than that of the core words, affecting the accuracy of text classification; 5) The volume of feature words and the number of long and short sentences contained in different texts may vary greatly, resulting in a large difference in the TF-IDF values of feature words in different texts, making it difficult to set a suitable TF-IDF threshold for screening text feature words, and it is difficult to ensure the balance of feature word extraction, which will also have an adverse effect on text classification. Summary of the invention
[0006] In view of the above-mentioned deficiencies in the prior art, the problem to be solved by the present invention is: how to provide a text feature extraction method so that it can have a better ability to distinguish key feature words in the text in the automotive field, so as to better ensure the objectivity and accuracy of the text feature word extraction.
[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions:
[0008] A text feature extraction method comprises the following steps:
[0009] S1: Obtain text and perform grammatical error recognition and grammar correction on the text;
[0010] S2: extract entity words from the text after grammatical correction, and add the extracted entity words to the feature dictionary;
[0011] S3: Use the feature dictionary to perform dependency syntactic analysis and word segmentation on the text after grammatical correction, and obtain feature words of word segmentation and dependency relationship information between feature words;
[0012] S4: Synonym search and synonym expression consistency processing are performed on the characteristic words obtained from text segmentation;
[0013] S5: Perform word frequency distribution statistics and dependency statistics on the feature words of the text after the synonym expression consistency processing, and then calculate the word frequency, inverse text frequency index and dependency distribution factor of each feature word as the word frequency distribution parameter of the feature word;
[0014] S6: Calculate the TF-IDF value of each feature word in the text according to the word frequency distribution parameters, and calculate the importance weight of each feature word in the text based on the TF-IDF value, mark and sort the feature words in the text according to their importance weight, and output them as a feature vector of the text.
[0015] In the above text feature extraction method, preferably, step S1 is specifically:
[0016] S1.1: Segment the text and store the corresponding relationship between each sentence and the text;
[0017] S1.2: Use a pre-trained grammatical error correction model to perform grammatical error recognition and grammatical correction processing on each sentence of the text, and output the text after grammatical correction processing.
[0018] In the above text feature extraction method, preferably, step S2 is specifically:
[0019] S2.1: Split the text after grammatical correction into single-word sequences and perform word vectorization to obtain single-word word vector features of the text;
[0020] S2.2: using the single-word vector features of the text as input to a pre-trained entity word boundary recognition model to perform entity word boundary recognition and marking processing on the text;
[0021] S2.3: Extract entity words from the text based on the entity word boundaries of the recognition mark and include them in the feature dictionary.
[0022] In the above text feature extraction method, preferably, step S3 is specifically:
[0023] S3.1: importing the feature dictionary into a language segmentation tool as a vocabulary of the language segmentation tool;
[0024] S3.2: Use the language segmentation tool to perform dependency syntactic analysis and segmentation on the text after grammatical correction to obtain the feature words of the segmentation and the dependency relationship information between the feature words.
[0025] In the above text feature extraction method, preferably, step 5 is specifically as follows:
[0026] S5.1: Count the total number of feature words obtained from word segmentation in the text and the number of times each feature word appears, and calculate the word frequency TF value of each feature word;
[0027] S5.2: Count the total number of texts and the number of texts containing each characteristic word, and calculate the inverse document frequency index IDF value of each characteristic word;
[0028] S5.3: According to the dependency information between feature words, the total number of dependency relationships and the number of times each feature word appears in the dependency relationship are counted, and the dependency distribution factor of each feature word is calculated:
[0029]
[0030] Among them, α i represents the dependency distribution factor of any i-th feature word, i∈{1,2,…,K}, K represents the total number of feature words obtained by word segmentation in the text; n i,c represents the number of times any i-th feature word appears in the subject-predicate dependency relationship and the verb-object dependency relationship of the text, n i,o Indicates the number of times any i-th feature word appears in other dependencies of the text; N c N represents the total number of occurrences of subject-predicate dependency and verb-object dependency in the text. o Indicates the total number of times other dependency relations appear in the text.
[0031] In the above text feature extraction method, preferably, step S6 is specifically as follows:
[0032] S6.1: Calculate the TF-IDF value of each feature word in the text according to the word frequency distribution parameter; wherein the TF-IDF value of any i-th feature word is calculated as follows:
[0033] TF-IDF i =TF i,j ×IDF i ×α i ;
[0034] Among them, TF-IDF i represents the TF-IDF value of any i-th feature word, i∈{1,2,…,K}, K represents the total number of feature words obtained by word segmentation in the text; TF i Indicates the word frequency TF value of any i-th feature word in the current text j; IDF i Represents the inverse text frequency index IDF value of any i-th feature word; α i Represents the dependency distribution factor of any i-th feature word;
[0035] S6.2: Calculate the normalized index value of the TF-IDF value as the importance weight of the feature word in the text; wherein the importance weight of any i-th feature word is calculated as follows:
[0036]
[0037] Among them, TF-IDFpi represents the importance weight of any i-th feature word; e is a natural constant;
[0038] S6.3: Extract feature words whose importance weights reach a preset importance weight threshold in the text and mark the corresponding importance weights. Sorting the extracted feature words in descending order of importance weights will result in a set, which will be output as a feature vector of the text.
[0039] Accordingly, the present invention also provides a text classification method, comprising the following steps:
[0040] Step A: Obtain the text to be classified, and use the above-mentioned text feature extraction method to extract feature words from the text to be classified to obtain a feature vector of the text to be classified;
[0041] Step B: Input the feature vector of the text to be classified into the textCNN model trained for text classification, and output the category prediction result of the text to be classified.
[0042] In the above text classification method, preferably, the textCNN model is obtained by text classification training in the following steps:
[0043] Step b1: obtaining a sample text data set from a text database, wherein the sample text data set includes a plurality of sample texts annotated with classification labels;
[0044] Step b2: using the aforementioned text feature extraction method to extract feature vectors of each sample text in the sample text data set;
[0045] Step b3: Select training samples and test samples from the text data set to form a training sample set and a test sample set respectively;
[0046] Step b4: taking the feature vector of each sample text in the training sample set as the input of the textCNN model, and taking the classification label of each sample text in the training sample set as the output verification label, performing classification prediction training on the textCNN model to adjust the text classification parameters of the textCNN model;
[0047] Step b5: Input the feature vector of each sample text in the test sample set into the textCNN model for classification prediction, and use the classification label of each sample text in the test sample set as the output verification label to compare and verify the classification prediction results of the textCNN model to evaluate the classification prediction performance of the textCNN model;
[0048] Step b6: If the classification prediction performance of the textCNN model does not reach the preset target, return to execute step b4; if the classification prediction performance of the textCNN model reaches the preset target, the training is completed and a textCNN model trained for text classification is obtained.
[0049] In the above-mentioned text classification method, preferably, in the step b3, the ratio of the number of texts selected as training samples and test samples is 8:2, and the selection method is random selection.
[0050] The present invention also discloses a readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the text classification method of the present invention are implemented.
[0051] Compared with the prior art, the present invention has the following beneficial effects:
[0052] The text feature extraction method of the present invention improves the text preprocessing process before word segmentation on the basis of the TF-IDF algorithm, including grammatical error correction, entity word extraction, syntactic dependency analysis, synonym expression consistency processing, etc., and introduces dependency factors in the calculation of TF-IDF to participate in the weighted calculation of key feature words in text sentences, optimizes the importance ranking of key feature words, and does not directly characterize the importance weight of feature words through TF-IDF values, but normalizes and converts TF-IDF values into importance ratios to characterize the weights, solves the problem that there may be large differences in the volume of feature words and the number of long and short sentences in different texts, better ensures the balance of text feature word extraction, and improves the objectivity and accuracy of classification when the extracted text features are applied to text problem classification applications. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to make the purpose, technical solution and advantages of the invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings, in which:
[0054] Figure 1 is a flow chart of the text feature extraction method of the present invention;
[0055] Figure 2 It is a flow chart of an implementation and application process of the text feature extraction method of the present invention;
[0056] Figure 3 is a flow chart of the text classification method of the present invention;
[0057] Figure 4 It is a flowchart of training and publishing the textCNN model used in the text classification method of the present invention;
[0058] Figure 5It is a schematic diagram of a technical implementation architecture structure for implementing the text classification method of the present invention. DETAILED DESCRIPTION
[0059] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but only represents selected embodiments of the present invention. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work belong to the scope of protection of the present invention.
[0060] It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings. In the description of the present invention, it should be noted that the orientation or position relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inside", "outside", etc. is based on the orientation or position relationship shown in the drawings, or the orientation or position relationship in which the invention product is usually placed when used, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance. In addition, the terms "horizontal", "vertical", etc. do not mean that the components are absolutely horizontal or suspended, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", and does not mean that the structure must be completely horizontal, but can be slightly tilted. In the description of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two elements. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0061] Embodiment 1:
[0062] In view of the limitations of the existing text feature extraction technology in the application of text feature extraction in the automotive field, the present invention provides a text feature extraction method. Before introducing the text feature extraction method of the present invention, the TF-IDF algorithm should be introduced first, which is a mature and commonly used text feature extraction method in the prior art and is also the basis of the method of the present invention.
[0063] TF-IDF (term frequency–inverse document frequency) is a commonly used weighting technique for information retrieval and text extraction. It uses statistical methods to evaluate the importance of a word to a document set or a document in a corpus. The importance of a word increases in direct proportion to the number of times it appears in the document and decreases in inverse proportion to the frequency of its appearance in the corpus. If a word appears frequently in one article and rarely appears in other articles, it is considered that this word or phrase has good category discrimination ability and is suitable for classification.
[0064] TF indicates the frequency of a feature word in a text. To prevent the word frequency from being biased towards long files, the TF value is usually normalized by dividing the word frequency by the total number of words in the article, as shown in Formula 1:
[0065]
[0066] Among them, TF i,j is the frequency TF value of word i in file j, n i,j is the number of occurrences, and the denominator is the sum of the number of occurrences of all words in file j;
[0067] The inverse document frequency index IDF can be obtained by dividing the total number of documents by the number of documents containing the word, and then taking the logarithm of the quotient. For example, formula 2:
[0068]
[0069] Among them, IDF i is the IDF value of word i, N is the total number of documents in the corpus, n i Represents the number of documents containing word i.
[0070] TF-IDF is actually: TF*IDF. When the frequency of a feature word in a file is high and the frequency of the word in the entire file collection is low, a high-weight TF-IDF can be generated. Therefore, TF-IDF tends to filter out common words and retain important words. Formula 3:
[0071]
[0072] The above is the TF-IDF algorithm.
[0073] The text feature extraction method of the present invention is as follows Figure 1 As shown, the following steps are included:
[0074] S1: Obtain text and perform grammatical error recognition and grammar correction on the text;
[0075] S2: extract entity words from the text after grammatical correction, and add the extracted entity words to the feature dictionary;
[0076] S3: Use the feature dictionary to perform dependency syntactic analysis and word segmentation on the text after grammatical correction, and obtain feature words of word segmentation and dependency relationship information between feature words;
[0077] S4: Synonym search and synonym expression consistency processing are performed on the characteristic words obtained from text segmentation;
[0078] S5: Perform word frequency distribution statistics and dependency statistics on the feature words of the text after the synonym expression consistency processing, and then calculate the word frequency, inverse text frequency index and dependency distribution factor of each feature word as the word frequency distribution parameter of the feature word;
[0079] S6: Calculate the TF-IDF value of each feature word in the text according to the word frequency distribution parameters, and calculate the importance weight of each feature word in the text based on the TF-IDF value, mark and sort the feature words in the text according to their importance weight, and output them as a feature vector of the text.
[0080] It should be noted that the text feature extraction method of the present invention can generate corresponding software codes or software services through program programming, and can then be run and implemented on a server and a computer.
[0081] The text feature extraction method of the present invention improves the text preprocessing process before word segmentation on the basis of the TF-IDF algorithm, including grammatical error correction, entity word extraction, syntactic dependency analysis, synonym expression consistency processing, etc., and introduces dependency factors in the calculation of TF-IDF to participate in the weighted calculation of key feature words in text sentences, optimizes the importance ranking of key feature words, and does not directly characterize the importance weight of feature words through TF-IDF values, but normalizes and converts TF-IDF values into importance ratios to characterize the weights, solves the problem that there may be large differences in the volume of feature words and the number of long and short sentences in different texts, better ensures the balance of text feature word extraction, and improves the objectivity and accuracy of classification when the extracted text features are applied to text problem classification applications.
[0082] Step S1 in the text feature extraction method of the present invention is specifically as follows:
[0083] S1.1: Segment the text and store the corresponding relationship between each sentence and the text;
[0084] S1.2: Use a pre-trained grammatical error correction model to perform grammatical error recognition and grammatical correction processing on each sentence of the text, and output the text after grammatical correction processing.
[0085] In the specific application implementation, the text can be segmented by period "." as the input of the downstream task. The correspondence between each segmented sentence and the text is stored in the Mysql library (relational database management system), which will be used again in the downstream task. The grammar correction model used can use grammar correction model tools such as pycorrector software. All dictionaries to be used are loaded from Redis (remote dictionary service); homophone dictionary, the content contains the candidate set of homophones and homophones in the dictionary for each word; load the similar dictionary, the content includes the candidate set of each word with common radicals in the dictionary; load the confusion dictionary, the content includes the data set corresponding to the discovered wrong words and correct words. The word segmentation model and the error correction model both use the pre-trained model in the Chinesebert module; the corrector function in the pycorrector toolkit can load the relevant dictionaries and pre-trained models in turn, complete the word segmentation, error recognition, and sorting of the error candidate set of the sentence, and output the error correction result, that is, the text after grammar correction processing is obtained.
[0086] Step S2 in the text feature extraction method of the present invention is specifically as follows:
[0087] S2.1: Split the text after grammatical correction into single-word sequences and perform word vectorization to obtain single-word word vector features of the text;
[0088] S2.2: using the single-word vector features of the text as input to a pre-trained entity word boundary recognition model to perform entity word boundary recognition and marking processing on the text;
[0089] S2.3: Extract entity words from the text based on the entity word boundaries of the recognition mark and include them in the feature dictionary.
[0090] In the specific application implementation, the grammatical correction tool pycorrector is used to complete the grammatical correction of the text in step S1, and the text is split into a sequence of single words. Each word is vectorized by calling the word vector model stored in Hbase (a distributed, column-oriented open source database) to complete the vectorized feature input; the pre-trained Bi-LSTM (bidirectional long short-term memory network) and CRF model (conditional random field model) are loaded to perform BIO sequence prediction on the word vector list, where B represents the starting word of the entity word, I represents the middle word of the entity word, and O represents other words as the boundary of the entity word; the words marked as BI, if not in the custom dictionary, are added to the feature dictionary.
[0091] Step S3 in the text feature extraction method of the present invention is specifically as follows:
[0092] S3.1: importing the feature dictionary into a language segmentation tool as a vocabulary of the language segmentation tool;
[0093] S3.2: Use the language segmentation tool to perform dependency syntactic analysis and segmentation on the text after grammatical correction to obtain the feature words of the segmentation and the dependency relationship information between the feature words.
[0094] In specific application implementation, the CustomDictionaryadd method of the Hanlp tool (Chinese language processing toolkit) can be used to import custom words; the JClass module of the Hanlp tool can be used to load the dependency syntactic analysis model, directly calculate the word segmentation of the sentence, and the dependency relationship between each word segmentation.
[0095] In step S4 of the text feature extraction method of the present invention, in a specific application implementation, a synonym table is imported, which includes common word pairs with synonymous relationships or semantically equivalent word pairs such as Chinese and English abbreviations; according to the synonym table, a synonym search is performed on the word segmentation list obtained by the text segmentation processing, and the synonyms hit by the search are replaced with standard words in the synonym table to achieve consistency in the expression of synonyms.
[0096] Step S5 in the text feature extraction method of the present invention is specifically as follows:
[0097] S5.1: Count the total number of feature words obtained from word segmentation in the text and the number of times each feature word appears, and calculate the word frequency TF value of each feature word;
[0098] S5.2: Count the total number of texts and the number of texts containing each characteristic word, and calculate the inverse document frequency index IDF value of each characteristic word;
[0099] S5.3: According to the dependency relationship information between the feature words, the total number of dependency relationships and the number of times each feature word appears in the dependency relationship are counted, and the dependency relationship distribution factor of each feature word is calculated.
[0100] The calculation of the word frequency TF value and the inverse text frequency index IDF value of text feature words is a mature technology and has been introduced before. Dependency Parsing (DP) reveals the syntactic structure by analyzing the dependency relationship between components within a language unit. Intuitively speaking, it is to analyze the grammatical components such as "subject, predicate, object", "attributive, adverbial, complement" in a sentence, and analyze the relationship between each component. Dependency syntax believes that the verb in the "predicate" is the core of a sentence, and other components are directly or indirectly related to the verb. Therefore, words in the "subject-predicate" and "verb-object" dependency relationships are usually important in the semantic expression of the sentence. The more dependencies there are, the more important the word is in the sentence. Based on this, the calculation idea of the dependency factor is proposed, as shown in Formula 4:
[0101]
[0102] Among them, α i represents the dependency distribution factor of any i-th feature word, i∈{1,2,…,K}, K represents the total number of feature words obtained by word segmentation in the text; n i,c represents the number of times any i-th feature word appears in the subject-predicate dependency relationship and the verb-object dependency relationship of the text, n i,o Indicates the number of times any i-th feature word appears in other dependencies of the text; N c N represents the total number of occurrences of subject-verb dependency and verb-object dependency in the text. o Indicates the total number of times other dependency relations appear in the text.
[0103] In the specific application implementation, the word list of the text is traversed, and the word is stored as the key in the form of a dictionary, and the accumulated frequency is the value. Finally, it is divided by the total number of word segments, and each value is updated to obtain the query dictionary of the feature words and TF values corresponding to the text. The word segmentation of all texts is traversed, and the word segmentation is stored as the key in the form of a dictionary. If the feature word appears in the cycle of each article, it is accumulated 1 time, and repeated appearances are not counted. Finally, the total number of articles is divided by the key value, and the natural logarithm is calculated to obtain the IDF value of the feature word in each document. The dependency list of the text is traversed, and the word segmentation is stored as the key in the form of a dictionary. The frequency of appearance in the dependency relationship is accumulated, where the frequency of subject-predicate and verb-object relationships is *2, and the frequency of other relationships is *1. Finally, it is divided by the weighted total frequency to obtain the query dictionary of the feature words and dependency factors corresponding to the text.
[0104] Step S6 in the text feature extraction method of the present invention is specifically as follows:
[0105] S6.1: Calculate the TF-IDF value of each feature word in the text according to the word frequency distribution parameter; wherein the TF-IDF value of any i-th feature word is calculated as follows, as shown in Formula 5:
[0106] TF-IDF i =TF i,j ×IDF i ×α i ;
[0107] Among them, TF-IDF i represents the TF-IDF value of any i-th feature word, i∈{1,2,…,K}, K represents the total number of feature words obtained by word segmentation in the text; TF i Indicates the word frequency TF value of any i-th feature word in the current text j; IDF i Represents the inverse text frequency index IDF value of any i-th feature word; α i Represents the dependency distribution factor of any i-th feature word;
[0108] S6.2: Calculate the normalized index value of the TF-IDF value as the importance weight of the feature vocabulary in the text;
[0109] Due to TF-IDF i The values vary greatly between different texts. In order to filter the feature words between texts through a unified threshold, the idea of normalized exponential function softmax is adopted to calculate the TF-IDF value within the text. i The value is converted and mapped to the interval [0,1], which is converted into the concept of proportion. In this way, the TF-IDF feature in the text i The sum of the values is 1. This makes it easier to screen the overall feature words of the document set based on the threshold of the percentage or the threshold of the cumulative percentage, and also avoids the disadvantage of imbalance caused by taking the same data feature words due to different sentence lengths. Therefore, in this embodiment, the importance percentage weight of any i-th feature word is calculated according to the following calculation logic, such as formula 6:
[0110]
[0111] Among them, TF-IDF pi represents the importance weight of any i-th feature word; e is a natural constant;
[0112] S6.3: Extract feature words whose importance weights reach a preset importance weight threshold in the text and mark the corresponding importance weights. Sorting the extracted feature words in descending order of importance weights will result in a set, which will be output as a feature vector of the text.
[0113] In general, the text feature extraction method of the present invention can be summarized as text preprocessing, feature statistics, and importance calculation. The feature vector of the text is extracted and can be used for some core applications of text problem classification. For example, common text problem classifications in the automotive field include consulting problems, fault problems, task instructions, and chat problems. The text after problem classification can be further applied to semantic analysis or big data statistics in these different types of problems. Its implementation and application process is as follows: Figure 2 shown.
[0114] Embodiment 2:
[0115] Based on the application of the text feature extraction method of the present invention, the present invention also proposes a text classification method for the problem of text question classification, such as Figure 3 As shown, the following steps are included:
[0116] Step A: obtaining a text to be classified, and extracting feature words from the text to be classified using the text feature extraction method of the present invention to obtain a feature vector of the text to be classified;
[0117] Step B: Input the feature vector of the text to be classified into the textCNN model trained for text classification, and output the category prediction result of the text to be classified.
[0118] The text to be classified refers to any text data that needs to be classified. After the feature vector is extracted from the text features, it is input into the textCNN model trained for text classification to perform question classification prediction and obtain the category prediction result of the text to be classified. The textCNN model is obtained through the text classification training in the following steps:
[0119] Step b1: obtaining a sample text data set from a text database, wherein the sample text data set includes a plurality of sample texts annotated with classification labels;
[0120] Step b2: using the text feature extraction method of the present invention to extract feature vectors of each sample text in the sample text data set;
[0121] Step b3: Select training samples and test samples from the text data set to form a training sample set and a test sample set respectively;
[0122] Step b4: taking the feature vector of each sample text in the training sample set as the input of the textCNN model, and taking the classification label of each sample text in the training sample set as the output verification label, performing classification prediction training on the textCNN model to adjust the text classification parameters of the textCNN model;
[0123] Step b5: Input the feature vector of each sample text in the test sample set into the textCNN model for classification prediction, and use the classification label of each sample text in the test sample set as the output verification label to compare and verify the classification prediction results of the textCNN model to evaluate the classification prediction performance of the textCNN model;
[0124] Step b6: If the classification prediction performance of the textCNN model does not reach the preset target, return to execute step b4; if the classification prediction performance of the textCNN model reaches the preset target, the training is completed and a textCNN model trained for text classification is obtained.
[0125] In specific application implementation, the trained textCNN model can be released to the public for problem classification prediction applications on the classified text through model calculation. In addition, the classification performance of the textCNN model can be monitored and retrained during the application process to continuously maintain and improve its classification effect. The training and release process of the textCNN model is as follows: Figure 4 As shown. The sample of the text data set with completed problem classification annotation refers to the process of labeling each text sample as a consulting problem, a fault problem, a task instruction problem, and a chat problem, and the label result is used as the target variable of the problem classification task; the feature vector extraction of the sample text mainly relies on the text feature extraction method of the present invention, and the initial value of the preset importance ratio threshold can be set to 50%. In the subsequent training process, the importance ratio threshold will also be optimized and adjusted according to the model performance evaluation results. During the training process of the textCNN model, the ratio of the number of texts selected for training samples and test samples can be designed to be 8:2, and the selection method can be designed to be random selection; the feature words of each sample text will call the word vector data set in Hbase in turn to obtain the word vector of the word segmentation; since the number of input feature words is uncertain, the padding method will be adopted here to perform long-cut and short-fill to ensure the consistency of the feature shape of the input model; the textCNN model framework is used to learn the data set, and the best model is selected with the smallest loss of the best validation set, and then the test set is predicted to evaluate whether the performance reaches the preset business goal based on the indicators; if the preset business goal is not achieved, the parameters in the process can be adjusted, and the training steps can be repeated until the performance reaches the preset goal; after the preset business goal is achieved, the textCNN model can be published online; in the specific implementation, the distributed model deployment platform can be used to distribute the model and provide interface services to the outside world.
[0126] As described above, the present invention improves the accuracy of word segmentation by adding a preprocessing process before TF-IDF calculation, further improves the weight of core words from the perspective of dependency syntactic analysis, proposes an improved TF-IDF calculation method, and takes the question classification task as an example to provide an application scenario after key question extraction.
[0127] The text classification method of the present invention can generate corresponding software codes or software services through program programming, and then can be run and implemented on servers and computers; Figure 5 As shown, its technical implementation framework can include the following parts:
[0128] Hardware layer: includes cloud services, computing engines such as CPU processors and GPU processors, and middleware, responsible for supporting the environment and computing resources;
[0129] Data layer: includes tools such as MySQL, HDFS, Hbase, and Redis, which are responsible for tasks such as relational data, text file storage, large-scale word vectors, and cache data storage;
[0130] Computation layer: includes tools such as Hanlp, pycorrector, chinesebert, Bi-LSTM, TF-IDF algorithm model, and textCNN model, responsible for tasks such as loading and computing models and tool modules;
[0131] Application layer: Taking question classification examples, it supports the prediction of question categories.
[0132] The hardware layer is responsible for the environment and computing resource support, among which the cloud service adopts Tencent Cloud, the CPU processor resources support basic logical calculations and indicator calculations, and the GPU processor resources support the operation of deep neural networks; the storage layer is responsible for the storage of indexes, relationships, text files, cached data, and large-scale word vector data, among which Mysql stores structured text data, field information, mapping relationships, etc., HDFS (Hadoop Distributed File System) stores a large number of text files, Redis caches and other information, and hbase stores Tencent's 8 million-level word vector data; the computing layer is responsible for the loading and operation of tools and models, among which the Hanlp tool participates in word segmentation and syntactic analysis, the pycorrector tool combines the Chinesebert model to participate in grammatical error correction, the Bi-LSTM model participates in named entity recognition, and the textCNN model participates in problem classification; application layer: supports specific business problems. The process of the present invention takes problem classification as an example to support the business needs of problem category prediction in the automotive field, such as consulting problems, fault problems, task instructions, and chat problems.
[0133] Embodiment three:
[0134] This embodiment discloses a readable storage medium.
[0135] A readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the text classification method of the present invention. The readable storage medium may be a device with a readable storage function such as a USB flash drive or a computer.
[0136] In summary, the text feature extraction method and text classification method of the present invention have the following beneficial effects:
[0137] 1. The text feature extraction method of the present invention proposes a specific preprocessing scheme for the word segmentation error problem with greater influencing factors in TF-IDF calculation, thereby improving the accuracy of word segmentation;
[0138] 2. The text feature extraction method of the present invention is too dependent on the frequency distribution of words in TF-IDF calculation, and has limited information expression ability. It proposes to introduce dependency factors to increase the weight of core words in sentences and alleviate the problem that the TF-IDF value of non-core words may be too large;
[0139] 3. In the text feature extraction method of the present invention, TF-IDF is an absolute value, which is difficult to divide the threshold for use in business. A softmax processing method is proposed to convert the absolute value into a percentage, thereby improving the flexibility of business use.
[0140] 4. The present invention further verifies and applies the important text extracted after TF-IDF calculation, and provides a text classification method, so that the objectivity and accuracy of the classification effect of the extracted text features in the text question classification application can be improved.
[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described with reference to the preferred embodiments of the present invention, those skilled in the art should understand that various changes can be made in form and details without departing from the spirit and scope of the present invention as defined.
Claims
1. A text feature extraction method, characterized in that: The steps include: S1: Obtain text and perform grammatical error recognition and grammar correction on the text; S2: extract entity words from the text after grammatical correction, and add the extracted entity words to the feature dictionary; S3: Use the feature dictionary to perform dependency syntactic analysis and word segmentation on the text after grammatical correction, and obtain feature words of word segmentation and dependency relationship information between feature words; S4: Synonym search and synonym expression consistency processing are performed on the characteristic words obtained from text segmentation; S5: Perform word frequency distribution statistics and dependency statistics on the feature words of the text after the synonym expression consistency processing, and then calculate the word frequency, inverse text frequency index and dependency distribution factor of each feature word as the word frequency distribution parameter of the feature word; step 5 is specifically as follows: S5.1: Count the total number of feature words obtained from word segmentation in the text and the number of times each feature word appears, and calculate the word frequency TF value of each feature word; S5.2: Count the total number of texts and the number of texts containing each characteristic word, and calculate the inverse document frequency index IDF value of each characteristic word; S5.3: According to the dependency information between feature words, the total number of dependency relationships and the number of times each feature word appears in the dependency relationship are counted, and the dependency distribution factor of each feature word is calculated: Among them, α i represents the dependency distribution factor of any i-th feature word, i∈{1,2,…,K}, K represents the total number of feature words obtained by word segmentation in the text; n i,c represents the number of times any i-th feature word appears in the subject-predicate dependency relationship and the verb-object dependency relationship of the text, n i,o Indicates the number of times any i-th feature word appears in other dependencies of the text; N c N represents the total number of occurrences of subject-verb dependency and verb-object dependency in the text. o Indicates the total number of occurrences of other dependency relations in the text; S6: Calculate the TF-IDF value of each feature word in the text according to the word frequency distribution parameters, and calculate the importance weight of each feature word in the text based on the TF-IDF value, mark and sort the feature words in the text according to their importance weight, and output them as a feature vector of the text.
2. The text feature extraction method according to claim 1, characterized in that: The step S1 is specifically as follows: S1.1: Segment the text and store the corresponding relationship between each sentence and the text; S1.2: Use a pre-trained grammatical error correction model to perform grammatical error recognition and grammatical correction processing on each sentence of the text, and output the text after grammatical correction processing.
3. The text feature extraction method according to claim 1, characterized in that: The step S2 is specifically as follows: S2.1: Split the grammatically corrected text into single-word sequences and perform word vectorization to obtain single-word word vector features of the text; S2.2: using the single-word vector features of the text as input to a pre-trained entity word boundary recognition model to perform entity word boundary recognition and marking processing on the text; S2.3: Extract entity words from the text based on the entity word boundaries of the recognition mark and include them in the feature dictionary.
4. The text feature extraction method according to claim 1, characterized in that: The step S3 is specifically as follows: S3.1: importing the feature dictionary into a language segmentation tool as a vocabulary of the language segmentation tool; S3.2: Use the language segmentation tool to perform dependency syntactic analysis and segmentation on the text after grammatical correction to obtain the feature words of the segmentation and the dependency relationship information between the feature words.
5. The text feature extraction method according to claim 1, characterized in that: The specific steps of step S6 are as follows: S6.1: Calculate the TF-IDF value of each feature word in the text according to the word frequency distribution parameter; wherein the TF-IDF value of any i-th feature word is calculated as follows: TF-IDF i =TF i,j ×IDF i ×α i ; Among them, TF-IDF i represents the TF-IDF value of any i-th feature word, i∈{1,2,…,K}, K represents the total number of feature words obtained by word segmentation in the text; TF i Indicates the word frequency TF value of any i-th feature word in the current text j; IDF i Represents the inverse text frequency index IDF value of any i-th feature word; α i Represents the dependency distribution factor of any i-th feature word; S6.2: Calculate the normalized index value of the TF-IDF value as the importance weight of the feature word in the text; wherein the importance weight of any i-th feature word is calculated as follows: Among them, TF-IDF pi represents the importance weight of any i-th feature word; e is a natural constant; S6.3: Extract feature words whose importance weights reach a preset importance weight threshold in the text and mark the corresponding importance weights. Sorting the extracted feature words in descending order of importance weights will result in a set, which will be output as a feature vector of the text.
6. A text classification method, characterized in that: The steps include: Step A: obtaining a text to be classified, and extracting feature words from the text to be classified using the text feature extraction method described in any one of claims 1 to 5 to obtain a feature vector of the text to be classified; Step B: Input the feature vector of the text to be classified into the textCNN model trained for text classification, and output the category prediction result of the text to be classified.
7. The text classification method according to claim 6, characterized in that: The textCNN model is obtained through text classification training in the following steps: Step b1: obtaining a sample text data set from a text database, wherein the sample text data set includes a plurality of sample texts annotated with classification labels; Step b2: using the text feature extraction method described in any one of claims 1 to 5 to respectively extract feature vectors of each sample text in the sample text data set; Step b3: Select training samples and test samples from the text data set to form a training sample set and a test sample set respectively; Step b4: taking the feature vector of each sample text in the training sample set as the input of the textCNN model, and taking the classification label of each sample text in the training sample set as the output verification label, performing classification prediction training on the textCNN model to adjust the text classification parameters of the textCNN model; Step b5: Input the feature vector of each sample text in the test sample set into the textCNN model for classification prediction, and use the classification label of each sample text in the test sample set as the output verification label to compare and verify the classification prediction results of the textCNN model to evaluate the classification prediction performance of the textCNN model; Step b6: If the classification prediction performance of the textCNN model does not reach the preset target, return to execute step b4; if the classification prediction performance of the textCNN model reaches the preset target, the training is completed and a textCNN model trained for text classification is obtained.
8. The text classification method according to claim 7, characterized in that: In step b3, the ratio of the number of texts of the training sample and the test sample is 8:2, and the selection method is random selection.
9. A readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the text classification method according to any one of claims 6 to 8 are implemented.
