Text classification method and device
By using the Chinese word vector model and cosine similarity calculation method, the classification text is accurately classified, which solves the problem that traditional methods cannot accurately capture the core content of the text, and achieves efficient and accurate text classification.
Patent Information
- Application Number
- CN202411542108.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-31
- Publication Date
- 2025-02-21
AI Technical Summary
When facing text with complex semantics and diverse expressions, traditional text classification methods cannot accurately capture the core content of the text, resulting in poor classification results.
By obtaining the text and the collection of categories to be classified, converting the categories into vectors using the Chinese word vector model, preprocessing and vector conversion of the text to be classified, calculating the cosine similarity between the category vector and the high-dimensional vector, and filtering out the Class vectors with the highest similarity are classified.
It realizes accurate classification of semantic complex and diverse texts, reduces incorrect division caused by comprehension deviations, and improves the accuracy and efficiency of text classification.
Smart Images
Figure CN119493865A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the technical field of natural language processing, and in particular to a text classification method and device. Background Art
[0002] With the rapid development of the Internet and information technology, text data is increasingly used in various fields, and the amount of data is also showing explosive growth. How to efficiently and accurately classify these text data is very important.
[0003] Traditional text classification methods mostly rely on keyword matching or simple statistical features, that is, determining the category of a text by identifying specific words that appear in the text, or analyzing some basic features of the text to help determine its classification. However, this simple classification method often fails to accurately capture the core content of a text with complex semantics and diverse expressions, resulting in poor text classification results.
[0004] Therefore, how to achieve accurate text classification has become a problem that needs to be urgently solved in this field. Summary of the invention
[0005] The present application provides a text classification method and device, the purpose of which is to achieve accurate text classification.
[0006] In order to achieve the above objectives, this application provides the following technical solutions:
[0007] A text classification method, comprising:
[0008] Acquire a text set to be classified and a category set; wherein the text set to be classified includes a plurality of texts to be classified, and the category set includes a plurality of categories;
[0009] Inputting the multiple categories into a Chinese word vector model to obtain multiple category vectors;
[0010] For each of the to-be-classified texts, data preprocessing is performed on the to-be-classified texts to obtain preprocessed to-be-classified texts;
[0011] Performing vector conversion on the preprocessed text to be classified to obtain a high-dimensional vector;
[0012] For each of the category vectors, calculating the cosine similarity between the category vector and the high-dimensional vector to obtain the similarity between the high-dimensional vector and each category vector;
[0013] From the similarities between the high-dimensional vector and each category vector, select the category vector with the highest similarity and mark it as the target category vector;
[0014] The text to be classified corresponding to the high-dimensional vector is added to the category corresponding to the target category vector.
[0015] Optionally, for each of the to-be-classified texts, performing data preprocessing on the to-be-classified text to obtain the preprocessed to-be-classified text, including:
[0016] For each text to be classified, delete the blank characters in the text to be classified to obtain the deleted text to be classified;
[0017] De-duplication processing is performed on the repeated characters in the deleted text to be classified to obtain a de-duplication text to be classified;
[0018] The non-Chinese characters in the deduplicated text to be classified are deleted to obtain the preprocessed text to be classified.
[0019] Optionally, performing vector conversion on the preprocessed text to be classified to obtain a high-dimensional vector includes:
[0020] Segment the preprocessed text to be classified to obtain multiple words;
[0021] For each of the words, the word frequency is calculated based on a word frequency calculation formula, the word and the preprocessed text to be classified to which the word belongs;
[0022] Calculate the rarity of a word based on the text set, the word, the preprocessed text data to which the word belongs, and an inverse text frequency calculation formula;
[0023] Determining the weight of the word according to the word frequency and the rarity of the word;
[0024] Input the words into the Chinese word vector model to obtain a word vector;
[0025] Multiplying the word vector by the weight of the word to obtain the product of the word;
[0026] The sum of the products of all words is calculated, and the sum of all the products is divided by the number of words to obtain a high-dimensional vector.
[0027] Optionally, determining the weight of a word according to the word frequency and the rarity of the word includes:
[0028] When the rarity of the word is greater than 0, calculating the text length of the preprocessed text to be classified to which the word belongs;
[0029] Calculate the average length of all the texts to be classified in the text collection;
[0030] The weight of the word is calculated based on the text length of the preprocessed text data to which the word belongs, the average length, the word frequency of the word, the rarity of the word, the preset adjustment parameters and the BM25 calculation formula;
[0031] When the rarity of the word is less than 0, 0 is determined as the weight of the word.
[0032] Optionally, also include:
[0033] Storing all pre-processed texts to be classified into a first preset file;
[0034] After the preprocessed text to be classified is stored, the plurality of categories and the category vector corresponding to each category are stored in a second preset file.
[0035] A text classification device, comprising:
[0036] An acquisition unit, used to acquire a text set to be classified and a category set; wherein the text set to be classified includes a plurality of texts to be classified, and the category set includes a plurality of categories;
[0037] An input unit, used for inputting the plurality of categories into a Chinese word vector model to obtain a plurality of category vectors;
[0038] A preprocessing unit, used for performing data preprocessing on each of the to-be-classified texts to obtain preprocessed to-be-classified texts;
[0039] A conversion unit, used for performing vector conversion on the preprocessed text to be classified to obtain a high-dimensional vector;
[0040] A calculation unit, configured to calculate, for each of the category vectors, a cosine similarity between the category vector and the high-dimensional vector, to obtain a similarity between the high-dimensional vector and each category vector;
[0041] A screening unit, configured to screen out a category vector with the highest similarity from the similarities between the high-dimensional vector and each category vector, and mark it as a target category vector;
[0042] The adding unit is used to add the to-be-classified text corresponding to the high-dimensional vector to the category corresponding to the target category vector.
[0043] Optionally, the preprocessing unit is specifically used for:
[0044] For each text to be classified, delete the blank characters in the text to be classified to obtain the deleted text to be classified;
[0045] De-duplication processing is performed on the repeated characters in the deleted text to be classified to obtain a de-duplication text to be classified;
[0046] The non-Chinese characters in the deduplicated text to be classified are deleted to obtain the preprocessed text to be classified.
[0047] Optionally, the conversion unit is specifically used for:
[0048] Segment the preprocessed text to be classified to obtain multiple words;
[0049] For each of the words, the word frequency is calculated based on a word frequency calculation formula, the word and the preprocessed text to be classified to which the word belongs;
[0050] Calculate the rarity of a word based on the text set, the word, the preprocessed text data to which the word belongs, and an inverse text frequency calculation formula;
[0051] Determining the weight of the word according to the word frequency and the rarity of the word;
[0052] Input the words into the Chinese word vector model to obtain a word vector;
[0053] Multiplying the word vector by the weight of the word to obtain the product of the word;
[0054] The sum of the products of all words is calculated, and the sum of all the products is divided by the number of words to obtain a high-dimensional vector.
[0055] Optionally, the conversion unit is specifically used for:
[0056] When the rarity of the word is greater than 0, calculating the text length of the preprocessed text to be classified to which the word belongs;
[0057] Calculate the average length of all the texts to be classified in the text collection;
[0058] The weight of the word is calculated based on the text length of the preprocessed text data to which the word belongs, the average length, the word frequency of the word, the rarity of the word, the preset adjustment parameters and the BM25 calculation formula;
[0059] When the rarity of the word is less than 0, 0 is determined as the weight of the word.
[0060] Optionally, also include:
[0061] A first storage unit, used for storing all pre-processed texts to be classified into a first preset file;
[0062] The second storage unit is used to store the plurality of categories and the category vector corresponding to each category in a second preset file after the preprocessed text to be classified is stored.
[0063] The technical solution provided by the present application obtains a set of texts to be classified and a set of categories; multiple categories are input into a Chinese word vector model to obtain multiple category vectors; for each text to be classified, data preprocessing is performed on the text to be classified; the preprocessed text vector to be classified is converted into a high-dimensional vector; for each category vector, the cosine similarity between the category vector and the high-dimensional vector is calculated; from the similarity between the high-dimensional vector and each category vector, the category vector with the highest similarity is selected and marked as the target category vector; the text to be classified corresponding to the high-dimensional vector is added to the category corresponding to the target category vector. In the process of vector conversion, the word information is fully analyzed, the conversion is performed according to the word information, and the cosine similarity between the category vector and the high-dimensional vector is calculated, and the category division is performed according to the similarity, which reduces the erroneous division caused by misunderstanding and realizes accurate text classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0065] Figure 1 A flowchart of a text classification method provided in an embodiment of the present application;
[0066] Figure 2 A flowchart of a method for preprocessing text to be classified provided in an embodiment of the present application;
[0067] Figure 3 A flowchart of a method for converting a text vector to be classified provided in an embodiment of the present application;
[0068] Figure 4 A schematic diagram of the architecture of a text classification device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0069] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0070] In this application, the terms "comprises", "comprising" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element.
[0071] like Figure 1 As shown, it is a flowchart of a text classification method provided in an embodiment of the present application, comprising the following steps:
[0072] S101: Obtain a set of texts to be classified and a set of categories.
[0073] The to-be-classified text set includes a plurality of to-be-classified texts, and the category set includes a plurality of categories (referring to categories or labels for classifying texts).
[0074] It should be noted that categories include but are not limited to single words (for example, literature, technology, history, education, entertainment, news, and reviews).
[0075] S102: Input multiple categories into the Chinese word vector model to obtain multiple category vectors.
[0076] The category vector indicates that the category is converted into a vector form.
[0077] It can be understood that the Chinese word vector model is a pre-trained model, which is pre-trained based on sample categories as input and sample category vectors manually annotated to the sample categories as training targets.
[0078] S103: For each text to be classified, data preprocessing is performed on the text to be classified to obtain the preprocessed text to be classified.
[0079] It can be understood that data preprocessing is performed on the text to be classified, including a series of data processing operations such as removing blank characters and removing duplicate data, to form a processed data set, that is, to obtain multiple preprocessed texts to be classified.
[0080] Optionally, in another embodiment of the present application, the specific implementation of step S103 is as follows: Figure 2 As shown, the following steps are included:
[0081] S201: For each text to be classified, delete the blank characters in the text to be classified to obtain the deleted text to be classified.
[0082] Optionally, use Python’s str.replace() method or regular expressions to remove whitespace characters from the text to be classified.
[0083] For example, the text to be classified is "I am from China", and the blank characters in the text to be classified are deleted, and the text to be classified after deletion is "I am from China".
[0084] It should be noted that due to the presence of blank characters, semantically consistent data such as "I am from China" and "I am from China" will be missed by the deduplication operation. Therefore, it is necessary to remove blank characters in the data before deduplication.
[0085] S202: performing deduplication processing on repeated characters in the deleted text to be classified to obtain a deduplicated text to be classified.
[0086] It is understandable that in order to ensure that the recommendation result is unique, it is necessary to deduplicate the texts to be classified in the text set to be classified during data preprocessing.
[0087] For example, the first text to be classified is "I am from China", and the second text to be classified is "I am from China". The repeated characters in the deleted text to be classified are deduplicated to obtain the deduplicated text to be classified "I am from China".
[0088] S203: Delete the non-Chinese characters in the deduplicated text to be classified to obtain the preprocessed text to be classified.
[0089] It is understandable that since the embodiments of the present application mainly classify texts based on Chinese semantic similarity, in order to prevent the classification effect from being affected, it is necessary to delete the non-Chinese characters in the deduplicated text to be classified to obtain the preprocessed text to be classified.
[0090] It should be noted that since the word segmentation operation in vectorization needs to divide the text into the smallest and meaningful units, it is necessary to preprocess the text to be classified before vectorization to remove or process the meaningless content in the text to be classified.
[0091] S104: Perform vector conversion on the preprocessed text to be classified to obtain a high-dimensional vector.
[0092] Among them, high-dimensional vectors refer to vectors with a very large number of features, such as vectors with hundreds or thousands of dimensions.
[0093] It can be understood that the preprocessed text to be classified is converted into a vector. Specifically, the preprocessed text to be classified is segmented to obtain multiple words; the multiple words are converted into multiple vectors; the weight of each word is calculated, and based on the weight of each word and the multiple vectors, a high-dimensional vector is determined.
[0094] Optionally, the dimension of the high-dimensional vector includes but is not limited to 300 dimensions.
[0095] Optionally, in another embodiment of the present application, the specific implementation of step S104 is as follows: Figure 3 As shown, the following steps are included:
[0096] S301: Segment the preprocessed text to be classified to obtain multiple words.
[0097] It is understood that the preprocessed text to be classified is segmented into smaller, meaningful units (usually called "words" or "tokens"). These units can be words, phrases, symbols or other language elements. The purpose of word segmentation is to convert continuous text into a discrete, easy-to-process form for subsequent text analysis and processing.
[0098] Optionally, the preprocessed text to be classified may be segmented using Chinese word segmentation technology, and then the TF-IDF weighting technology may be used to synthesize it into a high-dimensional vector that can represent the semantics of the entire phrase or the entire sentence.
[0099] For example, assuming that the preprocessed text to be classified is: "Natural language processing is an important branch of artificial intelligence.", the preprocessed text to be classified is segmented to obtain multiple words ['natural', 'language', 'processing', 'is', 'artificial intelligence', 'of', 'an', 'important', 'branch', '. '].
[0100] S302: For each word, the word frequency is calculated based on the word frequency calculation formula, the word and the pre-processed text to be classified to which the word belongs.
[0101] The word frequency of a word indicates the frequency of the word appearing in the preprocessed text to be classified to which the word belongs. Intuitively, the more frequently a word appears in a text, the more important it is to the topic of the text.
[0102] In step S302, the specific expression of the word frequency calculation formula is shown in formula (1).
[0103]
[0104] In formula (1), TF (t,d) is the frequency of the word, t is the word, d is the preprocessed text to be classified to which the word belongs, n (t,d) is the number of times t appears in d, and t' represents any word in d.
[0105] S303: Calculate the rarity of a word based on the text set, the word, the pre-processed text data to which the word belongs, and the inverse text frequency calculation formula.
[0106] Among them, the rarity of a word refers to the frequency of occurrence of the word in each text to be classified. If a word appears in many texts to be classified, then the word may be a common word and contributes little to the topic of a specific text.
[0107] In step S303, the specific expression of the inverse text frequency calculation formula is shown in formula (2).
[0108]
[0109] In formula (2), IDF (t,D) is the rarity of the word, |D| is the text set, and |{d∈D:t∈d}| is the number of texts to be classified that contain word t.
[0110] S304: Determine the weight of the word according to the frequency of the word and the rarity of the word.
[0111] The weight of a word indicates the importance of the word in the text collection.
[0112] Optionally, in another embodiment of the present application, the specific implementation of step S304 includes the following steps:
[0113] When the rarity of a word is greater than 0, the text length of the preprocessed text to be classified to which the word belongs is calculated.
[0114] The text length usually refers to the number of characters, including Chinese characters, punctuation marks, and spaces.
[0115] Optionally, the len() function in Python can be used to calculate the text length of the preprocessed text to be classified to which the word belongs.
[0116] Calculate the average length of all the texts to be classified in the text collection.
[0117] It can be understood that the average length of all the texts to be classified in the text set is calculated. Specifically, the total length of all the texts to be classified is calculated, and the total length is divided by the number of the texts to be classified to obtain the average length of all the texts to be classified.
[0118] The weight of the word is calculated based on the text length, average length, word frequency, word rarity, preset adjustment parameters and BM25 calculation formula of the preprocessed text data to which the word belongs.
[0119] The specific expression of the BM25 calculation formula is shown in formula (3).
[0120]
[0121] In formula (3), BM25 (t,d,D) is the weight of the word, len(d) is the text length of the preprocessed text to be classified to which the word belongs, avg_len is the average length of all texts to be classified, k1 and b are preset adjustment parameters, usually k1=1.5, b=0.75.
[0122] When the rarity of a word is less than 0, 0 is determined as the weight of the word.
[0123] It can be understood that, in combination with the content of formula (3), the calculation process of the word weight can refer to formula (4).
[0124]
[0125] As can be seen from formula (4), the calculation of TF-IDF is improved by introducing a saturation function and a document length factor for each word in the text to be classified. In the BM25 calculation formula, a saturation function is introduced to adjust the weight of the word frequency. This is to prevent a word from appearing too many times in a document and causing the weight to be too large. The BM25 calculation formula also takes into account the length of the text to be classified and introduces a document length factor, so that the effect of the length of the text to be classified on the weight is not linear. This can better adapt to texts of different lengths to be classified.
[0126] For example, text 1 to be classified: "Applejuice is made from apple.", text 2 to be classified: "iPhoneis invented by the Apple Company.", text 3 to be classified: "I like eating peach.", text 4 to be classified: "I am from China.", text 5 to be classified: "Panda is cute.", it can be seen that "apple" appears 2 times in text 1 to be classified, "juice" appears 1 time in text 1 to be classified, and the total number of words in text 1 to be classified is 6. Therefore, TF (apple,待分类文本1) =2 / 6=0.33, TF (juice,待分类文本1) =1 / 6=0.17. There are 5 texts in the text set, 2 of which contain "apple" and 1 contains "juice". Therefore, The length of text 1 is 6, and the average length of the text collection is To summarize, Apple's BM25 for text 1 is:
[0127] ; The BM25 of juice for text 1 is:
[0128] Therefore, according to the principle of BM25, juice is slightly more important to the classified text 1 than apple.
[0129] S305: Input the word into the Chinese word vector model to obtain the word vector.
[0130] Among them, the word vector is a high-dimensional vector.
[0131] S306: Multiply the word vector by the word weight to obtain the product of the word.
[0132] The specific expression of multiplying the word vector by the word weight is shown in formula (5).
[0133] Product of words = word vector * word weight (5)
[0134] S307: Calculate the sum of the products of all words, and divide the sum of all products by the number of words to obtain a high-dimensional vector.
[0135] Among them, calculating the sum of the products of all words and dividing the sum of all products by the number of words is actually calculating the average value of the products of all words, and determining the average value as a high-dimensional vector expressing the semantics of the text.
[0136] It should be noted that each word is represented by a high-dimensional vector, and each dimension of the vector is a real number. High-dimensional vectors can capture the semantic and grammatical information of words, making it possible to better understand and process natural language. These high-dimensional vectors are usually located in a high-dimensional space, and the goal is to make semantically similar words closer in the vector space.
[0137] It can be seen that the advantages of high-dimensional vectors are: ① Powerful semantic representation ability: high-dimensional vectors can capture the semantic information of words, so that semantically similar words are close in position in the vector space; ② Dealing with sparsity problems: Traditional methods (such as one-hot encoding) will cause high-dimensional vectors to be sparse, while high-dimensional vectors can effectively reduce dimensions and reduce sparsity; ③ Flexibility: high-dimensional vectors can be extended to higher-level text representations, such as sentence vectors and document vectors.
[0138] Taking advantage of the above advantages, the embodiment of the present application segments phrases and sentences, then weights the vector of each word, and finally synthesizes a sentence vector that can represent the semantics of the phrase and sentence.
[0139] S105: For each category vector, calculate the cosine similarity between the category vector and the high-dimensional vector to obtain the similarity between the high-dimensional vector and each category vector.
[0140] The specific form of calculating the cosine similarity between the category vector and the high-dimensional vector is shown in formula (6).
[0141]
[0142] In formula (6), similarity is the similarity between the high-dimensional vector and each category vector, A is the category vector, and B is the high-dimensional vector.
[0143] S106: Filter out the category vector with the highest similarity from the similarities between the high-dimensional vector and each category vector, and mark it as the target category vector.
[0144] It can be understood that from the similarities between the high-dimensional vector and each category vector, the category vector with the highest similarity is screened out. Specifically, the similarities between the high-dimensional vector and each category vector are sorted in order from high to low similarities to obtain a similarity sequence; the similarity with the smallest sequence number is screened out from the similarity sequence, and the category vector corresponding to the similarity with the smallest sequence number is represented as the target category vector.
[0145] For example, for a text to be classified and three categories, the calculated similarities are as follows: category vector A: similarity 0.8, category vector B: similarity 0.5, category vector C: similarity 0.9. From the similarity between the high-dimensional vector and each category vector, the category vector with the highest similarity is selected. The similarity between the text and category C is the highest (0.9) and is represented as the target category vector.
[0146] S107: Add the text to be classified corresponding to the high-dimensional vector to the category corresponding to the target category vector.
[0147] For example, the category vector with the highest similarity to the text to be classified corresponding to the high-dimensional vector is the category vector C, that is, the target category vector, and the text to be classified corresponding to the high-dimensional vector is added to the category corresponding to the target category vector.
[0148] Optionally, if the data volume of the text set to be classified and the category set is large, data preprocessing may take a lot of time. In order to reduce the time of data preprocessing and facilitate the subsequent use of the above data, the processed data can be pre-stored and then directly used. Therefore, another embodiment of the present application provides a data storage method, including:
[0149] All preprocessed texts to be classified are stored in the first preset file.
[0150] The first preset file includes but is not limited to: .pkl file.
[0151] After the preprocessed text to be classified is stored, multiple categories and the category vector corresponding to each category are stored in the second preset file.
[0152] The second preset file includes but is not limited to: .npy file.
[0153] In addition, it is necessary to initialize the text collection and category collection to be classified regularly, and select a form that can be quickly read and written to store the processed data and converted vectors.
[0154] It should be noted that, based on the above process shown in S101-S07, this embodiment can achieve the following beneficial effects:
[0155] 1. Improve the ability to understand the classified information: By deeply analyzing the semantic information of the text data, the embodiments of the present application can more accurately capture the true intention of the text, avoid the limitations of the traditional keyword matching method, reduce the erroneous classification caused by misunderstanding, and improve the intelligence and accuracy of the system.
[0156] 2. Improve the relevance of recommendation results: Optimize the recommendation algorithm: Using word vectors and BM25 weighting technology, the embodiment of the present application can more effectively evaluate and sort candidate content, ensuring that the classification is highly relevant to the text content. Through mathematical vector representation and weight evaluation, the text intent can be accurately matched, significantly improving the relevance and accuracy of text classification.
[0157] 3. Improve computing efficiency: We have designed an efficient similarity calculation mechanism that can complete the similarity evaluation between user queries and a large number of candidate contents in a short time. We have optimized the algorithm structure, reduced the consumption of computing resources, and improved the system's response speed and processing power.
[0158] 4. Improve user experience: Users no longer need to filter information from a large amount of irrelevant content and can obtain data of interest in a shorter period of time, which significantly improves user efficiency and satisfaction.
[0159] 5. Broad application prospects: The technology of the embodiments of this application is not only applicable to the field of information retrieval, but can also be widely used in e-commerce, social media, online education and other fields, and has broad market application prospects. By improving the accuracy and efficiency of the classification function, the embodiments of this application enhance the overall effectiveness of the text classification system and provide strong technical support for various application scenarios.
[0160] In summary, by performing vector conversion on the preprocessed text to be classified to obtain a high-dimensional vector, the word information is fully analyzed during the vector conversion process, the conversion is performed based on the word information, and the cosine similarity between the category vector and the high-dimensional vector is calculated. The categories are divided according to the similarity, which reduces the erroneous division caused by misunderstanding and achieves accurate text classification.
[0161] like Figure 4 As shown, it is a schematic diagram of the architecture of a text classification device provided in an embodiment of the present application, and the text classification device includes: an acquisition unit 100, an input unit 200, a preprocessing unit 300, a conversion unit 400, a calculation unit 500, a screening unit 600 and an adding unit 700.
[0162] The acquisition unit 100 is used to acquire a text set to be classified and a category set; wherein the text set to be classified includes a plurality of texts to be classified, and the category set includes a plurality of categories.
[0163] The input unit 200 is used to input multiple categories into the Chinese word vector model to obtain multiple category vectors.
[0164] The preprocessing unit 300 is used to perform data preprocessing on each text to be classified to obtain the preprocessed text to be classified.
[0165] The preprocessing unit 300 is specifically used for: for each text to be classified, deleting blank characters in the text to be classified to obtain the deleted text to be classified; deleting repeated characters in the deleted text to be classified to obtain the deduplicated text to be classified; deleting non-Chinese characters in the deduplicated text to be classified to obtain the preprocessed text to be classified.
[0166] The conversion unit 400 is used to perform vector conversion on the preprocessed text to be classified to obtain a high-dimensional vector.
[0167] The conversion unit 400 is specifically used for: performing word segmentation on the preprocessed text to be classified to obtain a plurality of words; for each word, calculating the word frequency based on the word frequency calculation formula, the word and the preprocessed text to be classified to which the word belongs; calculating the rarity of the word based on the text set, the word, the preprocessed text data to which the word belongs and the inverse text frequency calculation formula; determining the weight of the word according to the word frequency and the rarity of the word; inputting the word into the Chinese word vector model to obtain a word vector; multiplying the word vector by the word weight to obtain the word product; calculating the sum of the products of all words, and dividing the sum of all products by the number of words to obtain a high-dimensional vector.
[0168] The conversion unit 400 is specifically used for: when the rarity of a word is greater than 0, calculating the text length of the preprocessed text to be classified to which the word belongs; calculating the average length of all texts to be classified in the text set; calculating the weight of the word based on the text length, average length, word frequency, rarity of the word, preset adjustment parameters and BM25 calculation formula of the preprocessed text data to which the word belongs; when the rarity of the word is less than 0, determining 0 as the weight of the word.
[0169] The calculation unit 500 is used to calculate the cosine similarity between the category vector and the high-dimensional vector for each category vector, so as to obtain the similarity between the high-dimensional vector and each category vector.
[0170] The screening unit 600 is used to screen out the category vector with the highest similarity from the similarities between the high-dimensional vector and each category vector, and mark it as the target category vector.
[0171] The adding unit 700 is used to add the to-be-classified text corresponding to the high-dimensional vector to the category corresponding to the target category vector.
[0172] In summary, by converting the preprocessed text to be classified into vectors to obtain high-dimensional vectors, the word information is fully analyzed in the process of vector conversion, the conversion is performed based on the word information, and the cosine similarity between the category vector and the high-dimensional vector is calculated. The categories are divided according to the similarity, which reduces the erroneous division caused by misunderstanding and achieves accurate text classification.
[0173] Preferably, combined Figure 4 The text classification device further includes: a first storage unit and a second storage unit.
[0174] The first storage unit is used to store all pre-processed texts to be classified into a first preset file.
[0175] The second storage unit is used to store the multiple categories and the category vector corresponding to each category in a second preset file after the preprocessed text to be classified is stored.
[0176] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can refer to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can refer to the partial description of the method embodiment. The system and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.
[0177] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0178] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A text classification method, characterized in that: include: Acquire a text set to be classified and a category set; wherein the text set to be classified includes a plurality of texts to be classified, and the category set includes a plurality of categories; Inputting the multiple categories into a Chinese word vector model to obtain multiple category vectors; For each of the to-be-classified texts, data preprocessing is performed on the to-be-classified texts to obtain preprocessed to-be-classified texts; Performing vector conversion on the preprocessed text to be classified to obtain a high-dimensional vector; For each of the category vectors, calculating the cosine similarity between the category vector and the high-dimensional vector to obtain the similarity between the high-dimensional vector and each category vector; From the similarities between the high-dimensional vector and each category vector, select the category vector with the highest similarity and mark it as the target category vector; The text to be classified corresponding to the high-dimensional vector is added to the category corresponding to the target category vector.
2. The method according to claim 1, characterized in that For each of the to-be-classified texts, data preprocessing is performed on the to-be-classified text to obtain the preprocessed to-be-classified text, including: For each text to be classified, delete the blank characters in the text to be classified to obtain the deleted text to be classified; De-duplication processing is performed on the repeated characters in the deleted text to be classified to obtain a de-duplication text to be classified; The non-Chinese characters in the deduplicated text to be classified are deleted to obtain the preprocessed text to be classified.
3. The method according to claim 1, characterized in that The step of converting the preprocessed text to be classified into a vector to obtain a high-dimensional vector includes: Segment the preprocessed text to be classified to obtain multiple words; For each of the words, the word frequency is calculated based on a word frequency calculation formula, the word and the preprocessed text to be classified to which the word belongs; Calculate the rarity of a word based on the text set, the word, the preprocessed text data to which the word belongs, and an inverse text frequency calculation formula; Determining the weight of the word according to the word frequency and the rarity of the word; Input the words into the Chinese word vector model to obtain a word vector; Multiplying the word vector by the weight of the word to obtain the product of the word; The sum of the products of all words is calculated, and the sum of all the products is divided by the number of words to obtain a high-dimensional vector.
4. The method according to claim 3, characterized in that Determining the weight of a word according to the word frequency and the rarity of the word includes: When the rarity of the word is greater than 0, calculating the text length of the preprocessed text to be classified to which the word belongs; Calculate the average length of all the texts to be classified in the text collection; The weight of the word is calculated based on the text length of the preprocessed text data to which the word belongs, the average length, the word frequency of the word, the rarity of the word, the preset adjustment parameters and the BM25 calculation formula; When the rarity of the word is less than 0, 0 is determined as the weight of the word.
5. The method according to claim 1, characterized in that Also includes: Storing all pre-processed texts to be classified into a first preset file; After the preprocessed text to be classified is stored, the plurality of categories and the category vector corresponding to each category are stored in a second preset file.
6. A text classification device, characterized in that: include: An acquisition unit, used to acquire a text set to be classified and a category set; wherein the text set to be classified includes a plurality of texts to be classified, and the category set includes a plurality of categories; An input unit, used for inputting the plurality of categories into a Chinese word vector model to obtain a plurality of category vectors; A preprocessing unit, used for performing data preprocessing on each of the to-be-classified texts to obtain preprocessed to-be-classified texts; A conversion unit, used for performing vector conversion on the preprocessed text to be classified to obtain a high-dimensional vector; A calculation unit, configured to calculate, for each of the category vectors, a cosine similarity between the category vector and the high-dimensional vector, to obtain a similarity between the high-dimensional vector and each category vector; A screening unit, configured to screen out a category vector with the highest similarity from the similarities between the high-dimensional vector and each category vector, and mark it as a target category vector; The adding unit is used to add the to-be-classified text corresponding to the high-dimensional vector to the category corresponding to the target category vector.
7. The device according to claim 6, characterized in that The pre-processing unit is specifically used for: For each text to be classified, delete the blank characters in the text to be classified to obtain the deleted text to be classified; De-duplication processing is performed on the repeated characters in the deleted text to be classified to obtain a de-duplication text to be classified; The non-Chinese characters in the deduplicated text to be classified are deleted to obtain the preprocessed text to be classified.
8. The device according to claim 6, characterized in that The conversion unit is specifically used for: Segment the preprocessed text to be classified to obtain multiple words; For each of the words, the word frequency is calculated based on a word frequency calculation formula, the word and the preprocessed text to be classified to which the word belongs; Calculate the rarity of a word based on the text set, the word, the preprocessed text data to which the word belongs, and an inverse text frequency calculation formula; Determining the weight of the word according to the word frequency and the rarity of the word; Input the words into the Chinese word vector model to obtain a word vector; Multiplying the word vector by the weight of the word to obtain the product of the word; The sum of the products of all words is calculated, and the sum of all the products is divided by the number of words to obtain a high-dimensional vector.
9. The device according to claim 8, characterized in that The conversion unit is specifically used for: When the rarity of the word is greater than 0, calculating the text length of the preprocessed text to be classified to which the word belongs; Calculate the average length of all the texts to be classified in the text collection; The weight of the word is calculated based on the text length of the preprocessed text data to which the word belongs, the average length, the word frequency of the word, the rarity of the word, the preset adjustment parameters and the BM25 calculation formula; When the rarity of the word is less than 0, 0 is determined as the weight of the word.
10. The device according to claim 6, characterized in that Also includes: A first storage unit, used for storing all pre-processed texts to be classified into a first preset file; The second storage unit is used to store the plurality of categories and the category vector corresponding to each category in a second preset file after the preprocessed text to be classified is stored.