Text classification method, system, electronic device and storage medium
Quickly searching for the recent template titles through vectorized representation and Ball Tree data structure, solving the problem of high time complexity in existing text classification methods and achieving efficient text classification.
Patent Information
- Application Number
- CN202210190148.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-02-28
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2042-02-28
AI Technical Summary
The existing text classification method requires comparing the titles in the text to be classified with all titles in the knowledge base one by one, resulting in high time complexity and affecting efficiency.
By vectorizing the titles and template titles to be classified, use the Ball Tree data structure to quickly find the nearest N template titles, and determine the text category based on similarity comparison to reduce time complexity.
Without affecting the classification accuracy, the efficiency of text classification is significantly improved and the calculation complexity is reduced.
Smart Images

Figure CN114610880B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of natural language processing, and in particular to a text classification method, system, electronic device and storage medium. Background Art
[0002] As a basic task in natural language processing, text classification has high practical application value in spam identification, online public opinion monitoring, topic classification and other fields. Its related research has attracted widespread attention in the field of artificial intelligence.
[0003] In the related art, the text classification problem is usually transformed into the problem of finding which type of title template is most similar to the title in the text to be classified, and then the category corresponding to the most similar title template is used as the category of the text to be classified. However, the above text classification method requires comparing the title in the text to be classified with all the titles in the knowledge base one by one, and the time complexity of the calculation process is high, which affects the efficiency of text classification.
[0004] Therefore, how to improve the efficiency of text classification without affecting the classification accuracy is a technical problem that technical personnel in this field currently need to solve. Summary of the invention
[0005] The purpose of this application is to provide a text classification method, system, electronic device and storage medium, which can improve the efficiency of text classification without affecting the classification accuracy.
[0006] In order to solve the above technical problems, the present application provides a text classification method, which includes:
[0007] Determine the text to be classified, and vectorize the title to be classified in the text to be classified to obtain a vectorized title to be classified;
[0008] Set the N vectorized template titles closest to each of the vectorized titles to be classified as candidate titles, and add marks to the text categories corresponding to the candidate titles; wherein the vectorized template titles are obtained by vectorizing the template titles in the template classification knowledge base;
[0009] The M text categories with the most number of times the mark is added are set as candidate text categories;
[0010] The template files belonging to the candidate text category in the template classification knowledge base are set as candidate template files, and the text category of the text to be classified is determined according to the similarity comparison results between all the candidate template files and the text to be classified.
[0011] Optionally, vectorizing the title to be classified in the text to be classified to obtain a vectorized title to be classified includes:
[0012] Performing word segmentation processing on the title to be classified in the text to be classified to obtain a first word segmentation result;
[0013] The first word segmentation result is mapped to a vector space to obtain a vectorized title to be classified of the title to be classified.
[0014] Optionally, mapping the first word segmentation result to a vector space to obtain a vectorized title to be classified of the title to be classified includes:
[0015] Inputting the first word segmentation result of the title to be classified into the word vector model Word2vec to obtain a vectorized representation result of the first word segmentation result;
[0016] The vectorized title to be classified of the title to be classified is determined according to the vectorized representation results of all the first word segmentation results.
[0017] Optionally, before setting the N vectorized template titles closest to each of the vectorized titles to be classified as candidate titles, the method further includes:
[0018] Performing word segmentation processing on the template title to obtain a second word segmentation result;
[0019] Mapping the second word segmentation result to a vector space to obtain a vectorized representation result of the second word segmentation result;
[0020] Determine the vectorized title to be classified of the title to be classified according to the vectorized representation results of all the second word segmentation results;
[0021] Constructing a data structure Ball Tree of all the vectorized titles to be classified;
[0022] Accordingly, the N vectorized template titles closest to each of the vectorized titles to be classified are set as candidate titles, including:
[0023] The distance between the vectorized title to be classified and the vectorized template title is calculated by using the data structure Ball Tree, and the N vectorized template titles closest to each of the vectorized title to be classified are set as the candidate titles.
[0024] Optionally, determining the text category of the text to be classified according to the similarity comparison results between all the candidate template files and the text to be classified includes:
[0025] Calculating the longest common subsequence length between the template title of the candidate template file and the title to be classified of the text to be classified, and determining the first similarity between the text to be classified and each of the candidate template files according to the longest common subsequence length;
[0026] Calculating the edit distance between the template title of the candidate template file and the title to be classified of the text to be classified, and determining the second similarity between the text to be classified and each of the candidate template files according to the edit distance;
[0027] Performing weighted calculation on the first similarity and the second similarity to obtain a weighted similarity between the text to be classified and each of the candidate template files;
[0028] Determining whether the maximum value of the weighted similarity is greater than a preset value;
[0029] If so, the candidate template file with the highest weighted similarity to the text to be classified is set as the target template file, and the text category of the text to be classified is set as the text category of the target template file.
[0030] Optionally, determining the first similarity between the text to be classified and each of the candidate template files according to the longest common subsequence length includes:
[0031] Calculate the title similarity between each title text to be classified and each vectorized template title according to the longest common subsequence length, and construct a similarity matrix between the text to be classified and each candidate template file according to the title similarity;
[0032] The maximum match of the similarity matrix is calculated using the Hungarian algorithm, and the mean value of the maximum match is set as the first similarity between the text to be classified and the candidate template file.
[0033] Optionally, after determining the text category of the text to be classified according to the similarity comparison results between all the candidate template files and the text to be classified, the method further includes:
[0034] The correspondence between the text to be classified and the text category of the text to be classified is updated to the template classification knowledge base.
[0035] The present application also provides a text classification system, which includes:
[0036] A vectorization representation module is used to determine the text to be classified, and to perform vectorization representation on the title to be classified in the text to be classified to obtain a vectorized title to be classified;
[0037] A frequency statistics module, used for setting the N vectorized template titles closest to each of the vectorized titles to be classified as candidate titles, and adding marks to the text categories corresponding to the candidate titles; wherein the vectorized template titles are obtained by vectorizing the template titles in the template classification knowledge base;
[0038] A candidate category screening module, used for setting the M text categories with the most mark addition times as candidate text categories;
[0039] The category determination module is used to set the template files belonging to the candidate text category in the template classification knowledge base as candidate template files, and determine the text category of the text to be classified based on the similarity comparison results between all the candidate template files and the text to be classified.
[0040] The present application also provides a storage medium on which a computer program is stored. When the computer program is executed, the steps of the above-mentioned text classification method are implemented.
[0041] The present application also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above-mentioned text classification method when calling the computer program in the memory.
[0042] The present application provides a text classification method, comprising: determining a text to be classified, and vectorizing the titles to be classified in the text to be classified to obtain vectorized titles to be classified; setting N vectorized template titles closest to each of the vectorized titles to be classified as candidate titles, and adding marks to the text categories corresponding to the candidate titles; wherein the vectorized template titles are obtained by vectorizing the template titles in a template classification knowledge base; setting the M text categories with the most times of adding the marks as candidate text categories; setting the template files in the template classification knowledge base belonging to the candidate text categories as candidate template files, and determining the text category of the text to be classified based on the similarity comparison results between all the candidate template files and the text to be classified.
[0043] The present application vectorizes the title to be classified in the text to be classified to obtain the vectorized title to be classified, and adds a mark to the text category of the N vectorized template titles closest to each vectorized title to be classified according to the distance between the vectorized title to be classified and the vectorized template title. The vectorized template title is the result of vectorizing the target title in the template classification knowledge base. The template classification knowledge base includes multiple template clusters, each template cluster corresponds to a text category, each template cluster includes multiple template texts, and each template text includes multiple template titles. Therefore, each vectorized template title has its corresponding text category. The present application uses the M text categories with the most number of mark additions as candidate text categories. The present application determines the text category of the text to be classified based on the similarity comparison results of all the candidate template files and the text to be classified. Since the number of template titles included in the candidate text category is less than the template classification knowledge base, the text classification scheme of the present application can improve the text classification efficiency without affecting the classification accuracy. The present application also provides a text classification system, an electronic device and a storage medium, which have the above-mentioned beneficial effects and are not repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0045] Figure 1 A flowchart of a text classification method provided in an embodiment of the present application;
[0046] Figure 2 A structural diagram of a text classification system provided in an embodiment of the present application. DETAILED DESCRIPTION
[0047] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0048] See below Figure 1 , Figure 1 A flowchart of a text classification method provided in an embodiment of the present application.
[0049] Specific steps may include:
[0050] S101: determining a text to be classified, and performing vectorization on a title to be classified in the text to be classified to obtain a vectorized title to be classified;
[0051] Among them, the text to be classified in this embodiment refers to the text of unknown text type that needs to be classified, and the title to be classified refers to the title of the text to be classified. After determining the text to be classified, this step can perform a title extraction operation on the text to be classified to obtain the title to be classified, or determine the title to be classified according to the directory of the text to be classified. This embodiment determines the similarity between titles based on the distance between text vectors, so the title to be classified in the text to be classified can be vectorized, so as to map the text of the title to be classified to the vector space and obtain the vectorized title to be classified. The above-mentioned vectorized title to be classified refers to: the vectorized representation of the text of the title to be classified.
[0052] S102: setting N vectorized template titles closest to each of the vectorized titles to be classified as candidate titles, and adding marks to the text categories corresponding to the candidate titles;
[0053] Before this step, there may be an operation of vectorizing the template title in the template classification knowledge base to obtain a vectorized template title. The template classification knowledge base includes template texts of multiple text categories, each text category corresponds to a template set, and the template set includes multiple template texts belonging to the same text category, and the template title is the title of the template text. The above-mentioned vectorized template title refers to: the vectorized representation of the text of the template title.
[0054] This step can calculate the distance between the vectorized title to be classified and the vectorized template title, and then determine the N vectorized template titles closest to each vectorized title to be classified, and add marks to the text categories described therein. In this embodiment, the same text category can be repeatedly marked, so that after all marks are added, the number of times the mark is added for each text category is counted.
[0055] Specifically, this embodiment can select the current vectorized title to be classified according to the order in which the title to be classified appears in the text to be classified, calculate the distance between all vectorized template titles and the current vectorized title to be classified, and sort the distances from far to near, so as to add marks to the text classifications corresponding to the top N vectorized template titles in the sorting, determine the new current vectorized title to be classified, and execute the above steps until all the current vectorized titles to be classified are processed.
[0056] S103: setting the M text categories with the most mark addition times as candidate text categories;
[0057] Among them, each vectorized template title has its corresponding text category. The higher the number of times the mark of the vectorized template title is added, the greater the probability that the text to be classified and the template text where the vectorized template title is located belong to the same text category.
[0058] In this embodiment, the number of times each text category in the template classification knowledge base is marked can be counted, and the number of times can be ranked in descending order, and the M text categories with the most times of marking can be set as candidate text categories. In this way, the template texts in the template classification knowledge base can be preliminarily screened, so that in S104, only the template texts in the template classification knowledge base belonging to the candidate text category are similarity calculated.
[0059] S104: setting the template files belonging to the candidate text category in the template classification knowledge base as candidate template files, and determining the text category of the text to be classified according to the similarity comparison results between all the candidate template files and the text to be classified.
[0060] Among them, the candidate text category is the text category to which the text to be classified may belong. On this basis, all template files belonging to the candidate text category in the template classification knowledge base can be set as candidate template files, and then the similarity between the candidate template files and the text to be classified is calculated to obtain the similarity comparison result. The similarity comparison result includes the similarity between the text to be classified and each candidate template file, so the text category of the candidate template file with the highest similarity can be used as the text category of the text to be classified.
[0061] In this embodiment, the title to be classified in the text to be classified is vectorized to obtain the vectorized title to be classified, and according to the distance between the vectorized title to be classified and the vectorized template title, the text category of the N vectorized template titles closest to each vectorized title to be classified is marked. The vectorized template title is the result of vectorizing the target title in the template classification knowledge base. The template classification knowledge base includes multiple template clusters, each template cluster corresponds to a text category, each template cluster includes multiple template texts, and each template text includes multiple template titles. Therefore, each vectorized template title has its corresponding text category. In this embodiment, the M text categories with the most number of marks added are used as candidate text categories. In this embodiment, the text category of the text to be classified is determined based on the similarity comparison results between all the candidate template files and the text to be classified. Since the number of template titles included in the candidate text category is smaller than that in the template classification knowledge base, the text classification scheme of this embodiment can improve the text classification efficiency without affecting the classification accuracy.
[0062] As a feasible implementation, after determining the text category of the text to be classified according to the similarity comparison results between all the candidate template files and the text to be classified, the corresponding relationship between the text to be classified and the text category of the text to be classified can also be updated to the template classification knowledge base. Specifically, the template classification knowledge base includes multiple template clusters, each template cluster corresponds to a text category, and the text to be classified can be added to the template cluster corresponding to the text category to enrich the template classification knowledge base.
[0063] As a feasible implementation method, the above embodiment can obtain the vectorized title to be classified in the following manner: perform word segmentation processing on the title to be classified in the text to be classified to obtain a first word segmentation result; map the first word segmentation result to the vector space to obtain the vectorized title to be classified of the title to be classified. Specifically, after obtaining the first word segmentation result of the title to be classified, the first word segmentation result of the title to be classified can be input into the word vector model Word2vec to obtain the vectorized representation result of the first word segmentation result; determine the vectorized title to be classified of the title to be classified based on the vectorized representation results of all the first word segmentation results.
[0064] It can be understood that before setting the N vectorized template titles closest to each of the vectorized titles to be classified as alternative titles, a data structure corresponding to the titles to be classified can also be constructed so as to determine the distance between the vectorized titles to be classified and the vectorized template titles based on the data structure. The specific process is as follows: performing word segmentation processing on the template title to obtain a second word segmentation result; mapping the second word segmentation result to the vector space to obtain a vectorized representation result of the second word segmentation result; determining the vectorized title to be classified of the title to be classified based on the vectorized representation results of all the second word segmentation results; and constructing a data structure Ball Tree of all the vectorized titles to be classified.
[0065] Accordingly, this embodiment can determine the alternative titles in the following manner: calculate the distance between the vectorized title to be classified and the vectorized template title through the data structure Ball Tree, and set the N vectorized template titles closest to each of the vectorized title to be classified as the alternative titles.
[0066] The process described in the above embodiment is explained below through an embodiment in actual application.
[0067] Text classification technology is a technology that models the relationship between document features and document categories based on a set of annotated training documents, and then classifies new documents based on the learned relationship model.
[0068] At present, a variety of systems for text recognition have been developed, including knowledge-based methods, statistical and traditional machine learning methods, and deep learning-based methods. Knowledge-based methods mainly extract a series of logical rules from a given text set based on the experience of domain experts as the basis for computer automatic text classification. This type of method has the advantages of clear knowledge and strong interpretability, but it is time-consuming and labor-intensive to build a knowledge base. The basic steps of statistical and traditional machine learning methods include text feature extraction, text feature representation, and classifier construction. It extracts statistical features such as term frequency (TF), document frequency (DF), information gain (IG), etc., maps them to vector space, and then constructs a classifier through traditional machine learning models such as support vector machine (SVM) and K-Nearest Neighbor (KNN). Compared with knowledge-based methods, this type of method has stronger generalization ability but weaker interpretability. The deep learning-based method converts words in the text into word vectors, and automatically selects features and learns classification models by building neural network models such as convolutional neural networks (CNN) and long short-term memory artificial neural networks (LSTM). This type of method is data-driven and can learn good features on large-scale data, but it is difficult to train the model when the amount of data is small and the number of different types of data is unbalanced.
[0069] According to the length of the processed text, text classification can be divided into short text classification and long text classification tasks. For example, classifying the sentiment of microblog text is a typical short text classification task, and classifying the topic of news content is a long text classification task. A major challenge in short text classification is that each text has few words and provides limited information. The word frequency or other feature matrix constructed based on the word segmentation results is usually very sparse. Long text classification, due to the large number of words and rich information, faces the difficult problems of eliminating the influence of information irrelevant to classification features and analyzing the long-time feature dependency between contexts.
[0070] In the actual scenario of text classification, some word template files with marked categories are usually given, such as patent technology briefing templates, project application templates, labor contract templates, etc., to determine which category the file to be classified belongs to. The information contained in the template file is mainly the title of each chapter, and the explanatory text of the content to be filled under the title, while the file to be classified is a filled file, including the title and content. In this task context, the number of training data (template files) under each category is small and the content is not much different, but the difference between classes is more obvious. In this dataset with very little data and sparse data features, it is difficult to obtain good learning results whether it is based on traditional machine learning methods or deep learning methods. However, the titles of each level in the template file are naturally a kind of knowledge for determining what category a text is. There is no need to rely on experts to build a knowledge base, which is very suitable for processing using knowledge-based text classification methods. Based on the above analysis, we can remove factors unrelated to classification such as the extraction of word document text titles and the construction of knowledge bases, and describe the task as follows: There is a template classification knowledge base K = {A1, A2, ..., A M}, which contains M categories, and each category has a template set A i ={B1,B2,...,B a}, each template set has a certain number of templates B i =[τ1,τ2,...,τ b ], the template consists of the corresponding word file template in the title of each level τ i When given a text T to be classified, there are n titles T = [t1, t2, ..., t n ], the text to be classified is classified into a text category in the template classification knowledge base K.
[0071] Based on the above task description, the problem of text classification can be transformed into the problem of finding which type of title template is most similar to the title in the text to be classified, and the category of the most similar template is obtained as the category of the text to be classified. In the calculation of text similarity, commonly used calculation methods, such as: 1) Edit distance similarity: Edit distance is a quantitative measurement of the degree of difference between two strings. The measurement method is to see how many "insertions", "replacements" or "deletions" are required to transform one string into another. The smaller the edit distance between two strings, the more similar they are literally. 2) Longest common subsequence similarity: The longest common subsequence is a problem of finding the longest subsequence among all sequences in two sequences. A sequence, if it is a subsequence of two or more known sequences and is the longest of all sequences that meet this condition, is called the longest common subsequence of the known sequences. The larger the proportion of the longest common subsequence to the total length of the text, the more similar the two texts are literally. 3) Latent semantic similarity: Latent semantic similarity mainly refers to the use of latent semantic analysis technology (LSA / LSI) to map documents in high-dimensional vector space model representation into low-dimensional latent semantic space, and calculate their similarity by calculating the angle between vectors. The smaller the vector angle, the greater the latent semantic similarity, and the more semantically similar the two texts are. 4) Word vector similarity: It mainly refers to the use of word2vec, GloVe and other word vector models or other deep learning methods to convert text into vector form and calculate similarity in vector space. The smaller the vector angle, the greater the word vector similarity, and the more semantically similar the two texts are. In general, these methods focus on two aspects of text similarity: literal similarity and semantic similarity. Some texts, such as "I like you" and "You like me", are similar in words but not in semantics, and vice versa. The task source of this embodiment is to determine which type the new file belongs to based on the template file. The reason why the template file is called a template means that the title of the file edited based on it is basically consistent with it in terms of mode and content. For example, the "technical field to which the present invention belongs" in the patent technology briefing template may be written as "the technical field of the present invention", "the technical field to which the present invention relates", etc., which have a high degree of similarity both semantically and literally. To judge the similarity of these two aspects, a mainstream approach is to take one method of calculating literal similarity and semantic similarity, calculate the similarity separately, and then use weighted average, conditional constraints and other methods to fuse the calculation results to obtain the final similarity.
[0072] The disadvantage of the above mainstream approach is its huge time complexity: Assume that the template classification knowledge base of the text title K = {A1, A2, ..., A M}, which contains M text categories, and each text category has a template set A i ={B1,B2,...,Ba}, each template set has an average of a templates, and each template B i =[τ1,τ2,...,τ b ] Averaged over b title texts τ i Then there are m = M × a × b title texts in the text title template classification knowledge base, and each text to be classified T = [t1, t2, ..., t n ] has an average of n titles t i , assuming that the average number of words in each title in the template and each title in the text to be classified is c. To find the template closest to T, the algorithm needs to compare each t in M categories one by one i and each τ i If the similarity between the two is calculated using the longest common subsequence or edit distance, the time complexity of the dynamic programming algorithm is O(mnc 2 ), if the similarity calculation based on word vector is used, and the dimension of word vector is d, the time complexity is O(mndc). In practical applications, the dimension d of word vector is fixed, and the value range is generally an integer in the range of [50,300]. The number of words c in each title generally does not exceed 50. The number of titles n in the text to be classified can be constrained within a certain number by setting a threshold and other methods, but the total number of titles m in the text title template classification knowledge base may reach tens of thousands or even more, which becomes a bottleneck affecting the calculation speed. In view of the technical problems existing in the related art, this application proposes to use a two-stage feature matching algorithm for optimization, which significantly reduces the time complexity and improves the time efficiency of the calculation without reducing the accuracy.
[0073] In order to solve the above problems, this embodiment proposes a two-stage text classification method. The general idea is to divide the entire classification process into a pre-selection stage and a selection stage. In the pre-selection stage, a time-efficient algorithm is used to quickly pre-select several most likely categories from the text title template classification knowledge base, and then in the selection stage, other methods are used to find the most similar category from the pre-selected categories. In other words, in the selection stage, the algorithm with high computational complexity only needs to calculate the similarity between the text to be classified and the pre-selected types of templates. Compared with calculating on the original full set of the text title template classification knowledge base, the time spent on this subset will be shortened. In the pre-selection stage, the algorithm still needs to be calculated on the full set. At this time, how to reduce the time complexity of the pre-selection stage becomes a key issue.
[0074] This embodiment takes into account both the literal similarity and semantic similarity of the text. Therefore, the literal similarity can be calculated based on the edit distance and the longest common subsequence, and the semantic similarity can be calculated using the word2vec word vector model. According to the analysis above, the time complexity of the edit distance and the longest common subsequence is O(mnc2 ), the time complexity of the method based on word vector is O(mndc), this embodiment improves the method based on word vector, reduces its complexity to O(log(m)ndc), and uses it as an algorithm in the pre-selection stage. Specifically, the improvement of this embodiment can be implemented based on the data structure Ball Tree. Ball Tree is similar to K-dimensional Tree. Both are tree data structures that store instance points in k-dimensional space for rapid retrieval. They are mainly used for searching key data in multidimensional space. Its optimization effect on retrieval efficiency in the implementation of KNN algorithm has been widely recognized. The text similarity based on word vector itself also vectorizes the text and maps it to a high-dimensional vector space, which can compare each t one by one. i and each τ i The similarity problem between them is transformed into two steps: the first step is to transform all τ i Map to vector space and construct a Ball Tree; the second step is to transform each t i Map it to the vector space and find the k closest τ based on the constructed Ball Tree i vector.
[0075] The specific implementation process of the two-stage text classification method of the present invention is described below:
[0076] The initial task of this embodiment is to determine which category the document to be classified belongs to, given some Word template files that have been marked with categories, such as patent technology briefing templates, national fund project application templates, labor contract templates, etc. This solution only focuses on the part related to text features, and abstracts the problem into: there is a template classification knowledge base K = {A1, A2, ..., A M}, which contains M categories, and each category has a template set A i ={B1,B2,...,B a}, each template set has a certain number of templates, template B i =[τ1,τ2,...,τ b ] by the corresponding word document template in the various levels of title τ i When given a text T to be classified, there are n titles T = [t1, t2, ..., t n ], the text to be classified is classified into a type in the knowledge base.
[0077] Step 1: Segment the title text;
[0078] In this step, the title to be classified in the to-be-classified text can be segmented to obtain a first segmentation result, and the template title can also be segmented to obtain a second segmentation result; for example, a segmentation tool such as "Jieba Segmentation" can be used to segment each t i and each τ i Participle.
[0079] Step 2: Vectorize the segmented text;
[0080] Among them, this step can map the first word segmentation result to the vector space to obtain the vectorized title to be classified of the title to be classified; specifically, the first word segmentation result of the title to be classified can be input into the word vector model Word2vec to obtain the vectorized representation result of the first word segmentation result; the vectorized title to be classified of the title to be classified is determined according to the vectorized representation results of all the first word segmentation results. This embodiment can also map the second word segmentation result to the vector space to obtain the vectorized representation result of the second word segmentation result, and determine the vectorized title to be classified of the title to be classified according to the vectorized representation results of all the second word segmentation results.
[0081] Specifically, this embodiment can use a word vector model trained on a large-scale corpus for vectorization, such as the Word2vec word vector model. c ], and the c word segmentation results w i Input the word vector model in turn to obtain the vector representation of each word e1, e2, ..., e c (i.e., the vectorized representation of the first word segmentation result), forming a word vector list E = [e1, e2, ..., e c ], then the vectorized representation result t of the title to be classified t E (i.e., the vectorized title to be classified) is the mean of all word vectors in the title, i.e. The same method can be used to vectorize each template title in the template text to obtain the vectorized representation result of the template title. (i.e., vectorized template title). The i in the above text represents any value between 1 and the maximum value.
[0082] Step 3: Build and store the Ball Tree based on the vectorized representation of the template title:
[0083] Among them, this embodiment can construct all the vectorized titles to be classified The operation of this step can be implemented based on the open source KNN algorithm tool, for example, setting k=5, and setting the label of each data (ie, the vectorized title to be classified) to the text category l to which its template belongs.
[0084] Step 4: For the text to be classified that has been converted into vector form, E =[t1 E ,t2 E ,...,t n E ], take each title vector t in turn i E , find the k samples closest to it through the Ball Tree constructed in step 3, and obtain the k labels [l1,...,l k ], count the number of times different categories of labels appear. i E After searching and counting, take the first κ categories with the most occurrences as the pre-selected results, for example, take κ = 3;
[0085] Among them, step 4 is the preliminary selection stage of text categories. The above method can effectively reduce the amount of calculation in the selection stage and improve the efficiency of text classification.
[0086] Step 5: For the κ pre-selected categories obtained in step 4, calculate T = [t1, t2, ..., t n ] Similarity to each template in each category:
[0087] Among them, step 5 is the stage of selecting text categories, and the text category can be determined specifically in the following manner: calculating the longest common subsequence length between the template title of the candidate template file and the title to be classified of the text to be classified, and determining the first similarity between the text to be classified and each of the candidate template files according to the longest common subsequence length; calculating the edit distance between the template title of the candidate template file and the title to be classified of the text to be classified, and determining the second similarity between the text to be classified and each of the candidate template files according to the edit distance; performing weighted calculation on the first similarity and the second similarity to obtain the weighted similarity between the text to be classified and each of the candidate template files; judging whether the maximum value of the weighted similarity is greater than a preset value; if so, setting the candidate template file with the highest weighted similarity to the text to be classified as the target template file, and setting the text category of the text to be classified as the text category of the target template file.
[0088] Specifically, this embodiment can use the Hungarian algorithm to calculate the first similarity, and the process is as follows: calculate the title similarity between each title text to be classified and each vectorized template title based on the longest common subsequence length, and construct a similarity matrix between the text to be classified and each alternative template file based on the title similarity; use the Hungarian algorithm to calculate the maximum match of the similarity matrix, and set the mean of the maximum match as the first similarity between the text to be classified and the alternative template file.
[0089] An example to illustrate the above process:
[0090] (1) Using the longest common subsequence as the metric, calculate the text to be classified T = [t1, t2, ..., t n ] with each template B in each category i =[τ1,τ2,...,τ b ] (i.e. the first similarity):
[0091] Specifically, this embodiment can calculate each t based on the classic longest common subsequence algorithm i and each τ j The longest common subsequence length between them; convert the longest common subsequence length into similarity: let t i and τ j The longest subsequence length between ij , their word counts are c t and c τ , then the similarity between them All s ij Together they form a similarity matrix S of size n×b; Calculate the maximum similarity match: Use the Hungarian algorithm to find the maximum match of the similarity matrix S Find T and B i When the longest common subsequence similarity between the two, the mean of the maximum match can be taken as the final similarity, that is,
[0092] (2) Calculate text similarity (i.e., second similarity) using edit distance as a metric;
[0093] This step is similar to (1), using edit distance as the metric to calculate T and B i The edit distance similarity between EDIT , the difference is that t i and τ j When the edit distance between is converted into similarity: let t i and τ j The edit distance between Their word counts are c t and c τ, then the similarity between them
[0094] (3) Calculate the classification results:
[0095] T and B i Similarity between (i.e. weighted similarity) Where λ is the weight of the longest common subsequence similarity. If there are κ×b templates in κ categories, κ×b similarities will be obtained, and the category with the highest similarity that exceeds the predetermined threshold is the final classification result. t
[0096] This embodiment uses a text matching method based on word2vec word vectors to reduce the time complexity from O(mndc) to O(log(m)ndc), where m is the total number of template sentences in the existing classification knowledge base, n is the total number of sentences in the sample to be matched, d is the dimension of the word vector, and c is the average number of words in the sentence to be matched. This embodiment will calculate each t one by one i and each τ i The similarity problem between them is transformed into two steps: the first step is to transform all τ i Map to vector space and construct a BallTree; the second step is to transform each t i Map it to the vector space and find the k closest τ based on the constructed Ball Tree i vector.
[0097] This embodiment proposes a two-stage text classification scheme, which divides the entire classification process into a pre-selection stage and a selection stage: in the pre-selection stage, the word2vec word vector text matching method is used to quickly pre-select several template categories with the highest semantic similarity to the text to be classified; in the selection stage, based on the longest common subsequence similarity and the edit distance similarity, the template with the highest literal similarity to the text to be classified is found from the pre-selected categories. If the similarity is higher than a predetermined threshold, it is considered that the category to which the template belongs is the category to which the text to be classified belongs. This embodiment significantly reduces the time complexity of the algorithm without affecting the classification accuracy, achieves the technical effect of improving time efficiency, and is more in line with the needs of practical applications.
[0098] See also Figure 2 , Figure 2 A structural diagram of a text classification system provided in an embodiment of the present application, the system may include:
[0099] The vectorization representation module 201 is used to determine the text to be classified, and to perform vectorization representation on the title to be classified in the text to be classified to obtain a vectorized title to be classified;
[0100] The frequency statistics module 202 is used to set the N vectorized template titles closest to each of the vectorized titles to be classified as candidate titles, and add marks to the text categories corresponding to the candidate titles; wherein the vectorized template titles are obtained by vectorizing the template titles in the template classification knowledge base;
[0101] A candidate category screening module 203 is used to set the M text categories with the most mark addition times as candidate text categories;
[0102] The category determination module 204 is used to set the template files belonging to the candidate text category in the template classification knowledge base as candidate template files, and determine the text category of the text to be classified based on the similarity comparison results between all the candidate template files and the text to be classified.
[0103] In this embodiment, the title to be classified in the text to be classified is first vectorized to obtain the vectorized title to be classified, and according to the distance between the vectorized title to be classified and the vectorized template title, the text category of the N vectorized template titles closest to each vectorized title to be classified is marked. The vectorized template title is the result of vectorizing the target title in the template classification knowledge base. The template classification knowledge base includes multiple template clusters, each template cluster corresponds to a text category, each template cluster includes multiple template texts, and each template text includes multiple template titles. Therefore, each vectorized template title has its corresponding text category. In this embodiment, the M text categories with the most number of marks added are used as candidate text categories. In this embodiment, the text category of the text to be classified is determined based on the similarity comparison results between all the candidate template files and the text to be classified. Since the number of template titles included in the candidate text category is smaller than that in the template classification knowledge base, the text classification scheme of this embodiment can improve the text classification efficiency without affecting the classification accuracy.
[0104] Furthermore, the vectorized representation module 201 includes:
[0105] A word segmentation unit is used to perform word segmentation processing on the title to be classified in the text to be classified to obtain a first word segmentation result;
[0106] A vector mapping unit is used to map the first word segmentation result to a vector space to obtain a vectorized title to be classified of the title to be classified.
[0107] Furthermore, the vector mapping unit is used to input the first word segmentation result of the title to be classified into the word vector model Word2vec to obtain the vectorized representation result of the first word segmentation result; and is also used to determine the vectorized title to be classified of the title to be classified based on the vectorized representation results of all the first word segmentation results.
[0108] Furthermore, it also includes:
[0109] A data structure construction module, used for performing word segmentation processing on the template title to obtain a second word segmentation result; further used for mapping the second word segmentation result to a vectorized representation result of the second word segmentation result; further used for determining the vectorized title to be classified of the title to be classified according to the vectorized representation results of all the second word segmentation results; further used for constructing a data structure Ball Tree of all the vectorized titles to be classified;
[0110] Correspondingly, the frequency statistics module 202 is used to calculate the distance between the vectorized title to be classified and the vectorized template title through the data structure Ball Tree, and set the N vectorized template titles closest to each of the vectorized title to be classified as the candidate titles.
[0111] Furthermore, the category determination module 204 includes:
[0112] A first similarity calculation unit is used to calculate the longest common subsequence length between the template title of the candidate template file and the title to be classified of the text to be classified, and determine the first similarity between the text to be classified and each of the candidate template files according to the longest common subsequence length;
[0113] A second similarity calculation unit is used to calculate the edit distance between the template title of the candidate template file and the title to be classified of the text to be classified, and determine the second similarity between the text to be classified and each of the candidate template files according to the edit distance;
[0114] A similarity weighting unit, configured to perform weighted calculation on the first similarity and the second similarity to obtain a weighted similarity between the text to be classified and each of the candidate template files;
[0115] The result output unit is used to determine whether the maximum value of the weighted similarity is greater than a preset value; if so, the candidate template file with the highest weighted similarity to the text to be classified is set as the target template file, and the text category of the text to be classified is set as the text category of the target template file.
[0116] Furthermore, the first similarity calculation unit is used to calculate the title similarity between each title text to be classified and each vectorized template title according to the longest common subsequence length, and construct a similarity matrix between the text to be classified and each alternative template file according to the title similarity; it is also used to calculate the maximum match of the similarity matrix using the Hungarian algorithm, and set the mean of the maximum match as the first similarity between the text to be classified and the alternative template file.
[0117] Furthermore, it also includes:
[0118] The knowledge base updating module is used to update the correspondence between the text to be classified and the text category of the text to be classified to the template classification knowledge base after determining the text category of the text to be classified according to the similarity comparison results between all the candidate template files and the text to be classified.
[0119] Since the embodiments of the system part correspond to the embodiments of the method part, please refer to the description of the embodiments of the method part for the embodiments of the system part, which will not be repeated here.
[0120] The present application also provides a storage medium on which a computer program is stored, and when the computer program is executed, the steps provided in the above embodiment can be implemented. The storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.
[0121] The present application also provides an electronic device, which may include a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps provided in the above embodiment may be implemented. Of course, the electronic device may also include various network interfaces, power supplies and other components.
[0122] The various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of this application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of this application.
[0123] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
Claims
1. A text classification method, characterized in that: include: Determine the text to be classified, and vectorize the title to be classified in the text to be classified to obtain a vectorized title to be classified; Set the N vectorized template titles closest to each of the vectorized titles to be classified as candidate titles, and add marks to the text categories corresponding to the candidate titles; wherein the vectorized template titles are obtained by vectorizing the template titles in the template classification knowledge base; The M text categories with the most number of times the mark is added are set as candidate text categories; The template files belonging to the candidate text category in the template classification knowledge base are set as candidate template files, and the text category of the text to be classified is determined according to the similarity comparison results between all the candidate template files and the text to be classified.
2. The text classification method according to claim 1, characterized in that: Vectorizing the title to be classified in the text to be classified to obtain a vectorized title to be classified, including: Performing word segmentation processing on the title to be classified in the text to be classified to obtain a first word segmentation result; The first word segmentation result is mapped to a vector space to obtain a vectorized title to be classified of the title to be classified.
3. The text classification method according to claim 2, characterized in that: Mapping the first word segmentation result to a vector space to obtain a vectorized title to be classified of the title to be classified includes: Inputting the first word segmentation result of the title to be classified into the word vector model Word2vec to obtain a vectorized representation result of the first word segmentation result; The vectorized title to be classified of the title to be classified is determined according to the vectorized representation results of all the first word segmentation results.
4. The text classification method according to claim 1, characterized in that: Before setting the N vectorized template titles closest to each of the vectorized titles to be classified as candidate titles, the method further includes: Performing word segmentation processing on the template title to obtain a second word segmentation result; Mapping the second word segmentation result to a vector space to obtain a vectorized representation result of the second word segmentation result; Determine the vectorized title to be classified of the title to be classified according to the vectorized representation results of all the second word segmentation results; Constructing a data structure Ball Tree of all the vectorized titles to be classified; Accordingly, the N vectorized template titles closest to each of the vectorized titles to be classified are set as candidate titles, including: The distance between the vectorized title to be classified and the vectorized template title is calculated by using the data structure Ball Tree, and the N vectorized template titles closest to each of the vectorized title to be classified are set as the candidate titles.
5. The text classification method according to claim 1, characterized in that: Determining the text category of the text to be classified according to the similarity comparison results between all the candidate template files and the text to be classified includes: Calculating the longest common subsequence length between the template title of the candidate template file and the title to be classified of the text to be classified, and determining the first similarity between the text to be classified and each of the candidate template files according to the longest common subsequence length; Calculating the edit distance between the template title of the candidate template file and the title to be classified of the text to be classified, and determining the second similarity between the text to be classified and each of the candidate template files according to the edit distance; Performing weighted calculation on the first similarity and the second similarity to obtain a weighted similarity between the text to be classified and each of the candidate template files; Determining whether the maximum value of the weighted similarity is greater than a preset value; If so, the candidate template file with the highest weighted similarity to the text to be classified is set as the target template file, and the text category of the text to be classified is set as the text category of the target template file.
6. The text classification method according to claim 5, characterized in that: Determining a first similarity between the text to be classified and each of the candidate template files according to the longest common subsequence length includes: Calculate the title similarity between each title text to be classified and each vectorized template title according to the longest common subsequence length, and construct a similarity matrix between the text to be classified and each candidate template file according to the title similarity; The maximum match of the similarity matrix is calculated using the Hungarian algorithm, and the mean value of the maximum match is set as the first similarity between the text to be classified and the candidate template file.
7. The text classification method according to any one of claims 1 to 6, characterized in that: After determining the text category of the text to be classified according to the similarity comparison results between all the candidate template files and the text to be classified, the method further includes: The correspondence between the text to be classified and the text category of the text to be classified is updated to the template classification knowledge base.
8. A text classification system, characterized in that: include: A vectorization representation module is used to determine the text to be classified, and to perform vectorization representation on the title to be classified in the text to be classified to obtain a vectorized title to be classified; A frequency statistics module, used for setting the N vectorized template titles closest to each of the vectorized titles to be classified as candidate titles, and adding marks to the text categories corresponding to the candidate titles; wherein the vectorized template titles are obtained by vectorizing the template titles in the template classification knowledge base; A candidate category screening module, used for setting the M text categories with the most mark addition times as candidate text categories; The category determination module is used to set the template files belonging to the candidate text category in the template classification knowledge base as candidate template files, and determine the text category of the text to be classified based on the similarity comparison results between all the candidate template files and the text to be classified.
9. An electronic device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the text classification method according to any one of claims 1 to 7 are implemented.
10. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the steps of the text classification method according to any one of claims 1 to 7 are implemented.
Citation Information
Patent Citations
Text classification method and device
CN108228758A
Text classification method and device, equipment and storage medium
CN114003682A
