A text classification method, device, equipment and storage medium

Abstract

The application discloses a text classification method, which is used to solve the problems that the text classification effect is poor when the prior art uses a text classification model to perform text classification, and a large amount of human and computing resources are consumed in the model training process and the model training period is long. The method comprises the following steps: generating a second query word library according to the sorting results of each query word in a first query word library constructed in advance; inputting a to-be-classified text obtained into the second query word library to determine whether there is a keyword matching the to-be-classified text in the second query word library; when the determination result is no, inputting the to-be-classified text into the first query word library to determine at least one keyword corresponding to the to-be-classified text in the first query word library; and classifying the to-be-classified text according to the keyword.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a text classification method, apparatus, device, and storage medium. Background Technology

[0002] In recent years, with the rapid development of natural language processing technology, functions such as question answering systems, comment analysis, sentiment analysis, customer profiling, and content search based on natural language processing technology have been widely used in various industries.

[0003] Among them, text classification technology is a core key technology in natural language processing algorithms. Through text classification technology, the natural language text to be processed can be classified and matched with keywords, topic words or tags corresponding to the text, so that the natural language text can be processed according to the classification results.

[0004] Currently, commonly used text classification techniques can be mainly divided into two categories: text classification techniques based on classification models built using traditional machine learning algorithms, such as TF-IDF text classification; and text classification techniques based on classification models built using deep learning algorithms, such as Long Short-Term Memory (LSTM) networks.

[0005] Text classification techniques, which rely on trained text classification models, depend entirely on the accuracy of the classification results. The quality of existing text classification models largely depends on the quality of the training dataset. To ensure the effectiveness of the trained model, a large amount of precisely labeled training samples is required during training. This process consumes significant human and computational resources, leading to longer training times and less precise parameter adjustments.

[0006] Therefore, improving the classification speed and accuracy of text classification to better realize natural language processing has become a technical problem that relevant technicians in this field urgently need to solve. Summary of the Invention

[0007] This application provides a text classification method to address the problems of poor text classification performance, high computational and manpower costs, and long training cycles in existing text classification models.

[0008] This application also provides a text classification device to solve the problems of poor text classification effect when using text classification models in the prior art, and the need for a large amount of human and computing resources and a long model training cycle during the model training process.

[0009] This application also provides a text classification device to solve the problems of poor text classification effect when using text classification models in the prior art, and the need for a large amount of human and computing resources and a long model training cycle during the model training process.

[0010] This application also provides a computer-readable storage medium to address the problems of poor text classification performance and the high manpower and computing power required for model training, as well as the long training cycle, when using existing text classification models for text classification.

[0011] The embodiments of this application adopt the following technical solutions:

[0012] A text classification method includes: generating a second query term library based on the sorting results of each query term in a pre-constructed first query term library; inputting the obtained text to be classified into the second query term library, and determining whether there is a keyword in the second query term library that matches the text to be classified; when the determination result is negative, inputting the text to be classified into the first query term library, and determining at least one keyword corresponding to the text to be classified in the first query term library; and classifying the text to be classified based on the keyword.

[0013] A text classification device includes: a query term construction unit, configured to generate a second query term based on the sorting results of query terms in a pre-constructed first query term; a judgment unit, configured to input the obtained text to be classified into the second query term and determine whether there is a keyword in the second query term that matches the text to be classified; a keyword matching unit, configured to input the text to be classified into the first query term and determine at least one keyword corresponding to the text to be classified in the first query term when the judgment result is negative; and a classification unit, configured to classify the text to be classified based on the keyword.

[0014] A text classification device, comprising:

[0015] The processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the following operations: generating a second query term library based on the sorting results of query terms in a pre-constructed first query term library; inputting the obtained text to be classified into the second query term library, and determining whether there is a keyword in the second query term library that matches the text to be classified; when the determination result is negative, inputting the text to be classified into the first query term library, and determining at least one keyword corresponding to the text to be classified in the first query term library; and classifying the text to be classified based on the keyword.

[0016] A computer-readable storage medium stores one or more programs that, when executed by an electronic device including multiple applications, cause the electronic device to perform the following operations: generating a second query terminology based on the sorting results of query terms in a pre-constructed first query terminology; inputting acquired text to be classified into the second query terminology and determining whether there is a keyword in the second query terminology that matches the text to be classified; when the determination result is negative, inputting the text to be classified into the first query terminology and determining at least one keyword corresponding to the text to be classified in the first query terminology; and classifying the text to be classified based on the keyword.

[0017] The above-described technical solutions adopted in the embodiments of this application can achieve the following beneficial effects:

[0018] The text classification method provided in this application mainly relies on a pre-built first query terminology. This terminology stores commonly used keywords corresponding to different types obtained through clustering algorithms. For the text to be classified, a query can be performed directly in this terminology. Based on the query results, keywords corresponding to the text to be classified can be determined. Then, based on the types of these keywords, the type of the text to be classified can be determined, thus completing the classification of the text. The text classification method provided in this application does not rely on a pre-trained classification model. Therefore, compared to existing text classification methods, the text classification method provided in this application has higher classification speed and accuracy. To further improve text classification efficiency, in this embodiment, a second query terminology can be generated based on the ranking of each query term in the first query terminology. Before inputting the text to be classified into the first query terminology for querying, the second query terminology is used to search for matching keywords. Only when no keywords are found in the second query terminology is the text to be classified input into the first query terminology for keyword querying. Since the second query terminology contains fewer keywords and they are more important than the first query terminology, the method provided in this embodiment can further improve the efficiency of text classification. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0020] Figure 1 This is a schematic diagram illustrating a specific process of a text classification method provided in an embodiment of this application;

[0021] Figure 2 A schematic diagram of the specific structure of a text classification device provided in this application embodiment;

[0022] Figure 3 This is a schematic diagram of the specific structure of a text classification device provided in an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0025] This application provides a text classification method to address the problems of poor text classification performance, high computational cost, and long training cycles in existing text classification models.

[0026] The execution subject of the text classification method provided in this application embodiment may be, but is not limited to, at least one of a comment management server, a data recommendation server, a big data analysis server, a music recommendation server, and a social system; in addition, the execution subject of the method may also be the application (APP) itself running on these servers.

[0027] For ease of description, the following description uses a comment management server as the execution subject of this method as an example to introduce its implementation. It should be understood that using a comment management server as the execution subject is merely an illustrative example and should not be construed as a limitation of the method.

[0028] The schematic diagram illustrating the specific implementation process of the text classification method provided in this application is shown below. Figure 1 As shown, the main steps include the following:

[0029] Step 11: Generate a second query term library based on the sorting results of each query term in the pre-built first query term library;

[0030] In this embodiment of the application, the first query term database can be obtained by clustering the collected historical corpus data using a clustering algorithm. Specifically, in one implementation, the comment management server can construct the first query term database using the following method: acquiring historical corpus data, performing clustering processing on the historical corpus data according to a preset clustering algorithm to determine query terms; and constructing the first query term database based on the determined query terms.

[0031] The historical corpus data can be obtained from the database corresponding to the comment management server itself, or it can be collected from the Internet.

[0032] It should be noted that, in order to improve the query accuracy of the pre-built first query lexicon, when building the first query lexicon, the domain corresponding to the corpus data used to build the first database can be determined according to the business usage scenario of the first query lexicon. Then, when collecting corpus data, only data in the corresponding domain can be collected, and a dataset can be built based on the collected corpus data.

[0033] For example, if this first query terminology is subsequently used primarily for a music comment management server, then the music domain can be considered as the relevant domain for this first query terminology. This allows for the subsequent searching of comments from various music platforms on the internet, which can then be used as corpus data. In one implementation, the comment management server can collect corpus data from relevant domains using web crawlers.

[0034] In this embodiment, the comment management server can utilize the improved k-means clustering algorithm k-means++ to construct the first query term. The k-means++ algorithm primarily optimizes the method for selecting initial centroids in the traditional k-means algorithm. The traditional k-means algorithm randomly selects K points from the dataset as cluster centers. K-means++, however, selects K cluster centers using the following sub-steps:

[0035] Sub-step 1101: Randomly select a sample point in the dataset as the first initial cluster center;

[0036] Sub-step 1102: Calculate the distance between each sample point in the dataset and the first cluster center obtained by initialization through sub-step 1101;

[0037] Sub-step 1103: Select the second cluster center based on the distance calculated by performing sub-step 1102;

[0038] According to the K-means++ algorithm, cluster centers can be selected according to the following rules: sample points that are farther away from the first cluster center have a higher probability of being selected as cluster centers.

[0039] Sub-step 1104: Repeat sub-steps 1101 to 1103 until all k cluster centers are determined.

[0040] Additionally, it should be noted that in this embodiment, the purpose of clustering the collected dataset is not to perform a final clustering of the dataset, but rather to accurately and comprehensively identify query terms associated with the dataset through clustering algorithms, and then construct a first query term library based on these query terms for use in subsequent text classification. Therefore, in this embodiment, the number of cluster centers K can be determined using the following method:

[0041] Assuming that K cluster centers have been determined through sub-steps 1101 to 1104, the difference between the sum of squared errors (SSE) of the k-th cluster center and the SSE of the (k-1)-th cluster center is calculated. If this difference is less than a preset threshold, then k is taken as the final number of cluster centers that can be determined for the dataset. That is, the operation terminates after k cluster centers have been determined through sub-steps 1101 to 1104. If the difference is never less than the preset threshold, then sub-steps 1101 to 1104 are repeated until all cluster centers in the dataset are determined.

[0042] By executing the above sub-steps, the comment management server can cluster the obtained dataset into K clusters, and use the words corresponding to the cluster center of each cluster as query terms, and then construct the first query term library based on these query terms.

[0043] In one implementation, to further improve text classification efficiency, in this embodiment, the query terms in the first query terminology are sorted, and the query terms in the first query terminology are filtered according to the sorting results. Then, a second query terminology is constructed based on the filtering results. Before inputting the text to be classified into the first query terminology for querying, the second query terminology is used to search for matching keywords. Only when no keywords are found in the second query terminology is the text to be classified input into the first query terminology for keyword querying. Since the second query terminology contains fewer keywords and they are more important than the first query terminology, the method provided in this embodiment can further improve the efficiency of text classification.

[0044] In one implementation, the comment management server can generate a second query terminology database as follows: determining the frequency of occurrence of each query term in the first query terminology database; determining the word weight of each query term in the first query terminology database; sorting each query term in the first query terminology database according to the frequency of occurrence and the word weight, and obtaining a sorting result; and generating the second query terminology database according to the sorting result.

[0045] In one implementation, the first query term library constructed by the method described in the embodiments of this application stores query terms corresponding to different types. The same query term may correspond to different types. For example, the query term "puppy" may appear in comments related to pets or comments related to appliances, such as "puppy vacuum cleaner". The higher the frequency of a query term in the first query term library, the more types it corresponds to, and the higher the probability of matching the query term with the text to be classified. Therefore, query terms whose frequency of occurrence meets the preset frequency threshold can be filtered out for subsequent construction of the second query term library.

[0046] Step 12: Input the obtained text to be classified into the second query term library, and determine whether there are keywords in the second query term library that match the text to be classified. If the determination result is no, proceed to step 13; if the determination result is yes, proceed to step 14.

[0047] In this embodiment, after obtaining the text to be categorized (e.g., a review text for a specific product), the review management server can first preprocess the obtained text data. In one implementation, this preprocessing process may include:

[0048] Sub-step 1201: Perform Chinese word segmentation on the obtained text data to be classified to obtain the word sequence corresponding to the text to be classified;

[0049] Specifically, the comment management server can input the text to be classified into a pre-built word segmentation module, and then the word segmentation module can process the text to be classified to obtain the word sequence corresponding to the text to be classified.

[0050] For example, suppose the text to be classified is "claims specialist A followed up promptly, I am very satisfied". After the word segmentation module processes the text to be analyzed, the initial words in the resulting word sequence include the following six words: "claims specialist A, follow up, promptly, I, very, satisfied".

[0051] It should be noted that the specific construction method of the word segmentation module belongs to the common technical solutions in the relevant technical field. The specific construction and usage method of the word segmentation module will not be described in detail in the embodiments of this application.

[0052] Sub-step 1202: Filter the word sequence obtained by executing sub-step 1201;

[0053] Filter low-frequency words, stop words, marker information, and synonyms in the word sequence.

[0054] Sub-step 1203 performs text correction on the word sequence after filtering by sub-step 1202. After text correction, the preprocessing of the text to be classified is completed.

[0055] After preprocessing the text to be classified, the comment management server can input the word sequence corresponding to the text to be classified into the second query thesaurus, and then query each word in the word sequence in turn to see if there are matching keywords in the second query thesaurus.

[0056] Step 13: If the judgment result obtained by executing step 12 is negative, then input the word sequence corresponding to the text to be classified into the first query term library to determine at least one keyword corresponding to the text to be classified in the first query term library;

[0057] In this embodiment, the comment management server can query keywords corresponding to the text to be classified in a first query thesaurus based on semantic similarity. It should be noted that, to improve text classification efficiency, in one implementation, before inputting the word sequence corresponding to the text to be classified into the first query thesaurus for keyword matching, the comment management server needs to further filter the word sequence. For example, modal particles or adjectives in the text often have little impact on the semantics of the entire text, while words with higher weights often have a deeper impact on the semantics of the text. Therefore, in this embodiment, the comment management server can further filter the word sequence based on the part of speech and weight of each word, thereby selecting words that are more important to the semantics of the text to be classified. Specifically, in one embodiment… In this implementation, the comment management server can process the word sequence corresponding to the text to be classified according to the following method, and search for corresponding keywords in the first query terminology based on the processed word sequence: filter the word sequence according to the part of speech of each word in the word sequence to obtain a second word sequence; determine the word weight of each word in the second word sequence, filter the second word sequence according to the word weight to obtain a third word sequence; determine the semantic similarity between each word in the third word sequence and each query word in the first query terminology according to the semantic similarity algorithm; and determine the keywords corresponding to the text to be classified in the first query terminology according to the semantic similarity.

[0058] In this embodiment, the comment management server can calculate the word weight of each word in the word sequence based on the frequency of each word in the text to be classified and the frequency of each word in the word sequence in the entire first query thesaurus, using the Term Frequency-Inverse Document Frequency (TF-IDF) algorithm. In one implementation, the comment management server can calculate the word weight of each word in the word sequence according to the following sub-steps:

[0059] Sub-step 1301: Determine the first word frequency of each word in the second word sequence obtained after word order filtering of the word sequence;

[0060] Among them, the first word frequency represents the frequency of the word in the text to be classified.

[0061] Sub-step 1302: Determine the second word frequency of each word in the second word sequence;

[0062] The second word frequency represents the frequency of a word's occurrence in the first query term database.

[0063] Sub-step 1303: Determine the word weight of each word in the second word sequence based on the first word frequency and the second word frequency.

[0064] In one implementation, the comment management server can calculate the word weight of each word in the second word sequence based on the TF-IDF algorithm, as follows:

[0065] a. The comment management server determines the total number of words corresponding to the text to be classified. Based on the first word frequency and the total number of words, the term frequency (TF) is determined. Specifically, the term frequency (TF) of each word can be calculated according to the following formula [1]:

[0066]

[0067] Where, n i,j Indicates the term t i In document d j Number of times it appears in TF i,j Indicates the term t i In document d j The frequency of occurrence.

[0068] b. The comment management server determines the total number of query terms contained in the first query term library. Based on the second term frequency and the total number of query terms contained in the first query term library, it determines the inverse document frequency (IDF). Specifically, the inverse document frequency (IDF) of each word can be calculated according to the following formula [2]:

[0069]

[0070] Where |D| represents the total number of documents, |j:t i ∈d j | indicates that the term t is included. i The number of documents.

[0071] The IDF (Intense Prevalence Factor) represents the prevalence of each word in a word sequence. The fewer documents in the first query term that contain a certain word, the larger the IDF of that word, indicating that the word has a stronger predictive ability for the text type to be classified, that is, the word has a good ability to distinguish categories.

[0072] c. The comment management server calculates and determines the word weight of each word based on the TF and IDF obtained by performing the above steps.

[0073] Specifically, the word weights of each word are calculated according to the following formula [3]:

[0074] TF-IDF = TF·IDF [3]

[0075] d. The comment management server filters each word in the second word sequence according to the preset weight threshold, and discards words whose weight is lower than the preset weight threshold, thereby obtaining the third word sequence.

[0076] In one implementation, the comment management server can calculate the similarity between each word in the third word sequence and each query word in the first query term library based on a pre-trained text similarity recognition model, and then determine at least one keyword corresponding to the text to be classified in the first query term library based on the similarity ranking.

[0077] Step 14: Classify the text to be classified based on the keywords determined by performing Steps 12 and 13 above.

[0078] Specifically, the comment management server can determine the topic or intent corresponding to the keywords, classify the text to be classified, and thus complete the text classification process for the text to be classified.

[0079] Additionally, it should be noted that in this embodiment, in addition to classifying the collected comment texts according to the above method, the comment management server can also perform emotion recognition on the collected comment texts through a sentiment analysis module, thereby obtaining the sentiment tendency of the comments.

[0080] Specifically, the comment management server can build a lexicon from the preprocessed and segmented historical comment texts, and input the segmentation results from the lexicon into the word2vec model for training, using each segmentation as a unit. Through word2vec training, a similarity calculation model for each segmentation is formed.

[0081] The sentiment tendency is divided into four levels: extremely positive, positive, negative, and extremely negative. The order of each word segment in the lexicon is determined based on word frequency, and the top n (e.g., top 500) words are identified as high-frequency words. The sentiment tendency of the high-frequency words is calculated based on the k-Nearest Neighbor (KNN) algorithm. Specifically, "very good", "good", "bad", and "very" can be used as the center words for the first matching of the four sentiment levels, and feature lexicon and sentiment lexicon are classified.

[0082] When performing sentiment recognition on comments, the sentiment tendency of each word segment in the comment can be identified sequentially. Specifically, nearest neighbor matching can be performed on each word segment to find the nearest neighbor word array. Then, the similarity array of the nearest neighbor word array is determined using a trained word2vec model. Each nearest neighbor word array is compared with four sentiment level vocabularies. If there is a word in the vocabulary with a similarity of more than 70%, the word is considered to belong to that sentiment level. Finally, the sentiment word vector is obtained through the nearest neighbor words. If there are feature keywords or words in the synonym forest, the vector value is doubled. The sentiment vector of the sentence is directly obtained by linearly adding the sentiment vectors of each word segment. Finally, the sentiment vector is used to calculate its sentiment tendency through a support vector machine.

[0083] Finally, the determined sentiment tendencies are summarized using the text classification results from steps 11 to 14 above, which leads to a comprehensive analysis of the comments, such as the sentiment tendency and intent of the comment.

[0084] The text classification method provided in this application mainly relies on a pre-built first query terminology. This terminology stores commonly used keywords corresponding to different types obtained through clustering algorithms. For the text to be classified, a query can be performed directly in this terminology. Based on the query results, keywords corresponding to the text to be classified can be determined. Then, based on the types of these keywords, the type of the text to be classified can be determined, thus completing the classification of the text. The text classification method provided in this application does not rely on a pre-trained classification model. Therefore, compared to existing text classification methods, the text classification method provided in this application has higher classification speed and accuracy. To further improve text classification efficiency, in this embodiment, a second query terminology can be generated based on the ranking of each query term in the first query terminology. Before inputting the text to be classified into the first query terminology for querying, the second query terminology is used to search for matching keywords. Only when no keywords are found in the second query terminology is the text to be classified input into the first query terminology for keyword querying. Since the second query terminology contains fewer keywords and they are more important than the first query terminology, the method provided in this embodiment can further improve the efficiency of text classification.

[0085] In one embodiment, this application also provides a text classification device to address the problems of poor text classification performance and the high computational and human resource requirements and long training cycles in existing text classification models. A schematic diagram of the specific structure of this application system's automatic scaling device is shown below. Figure 2 As shown, it includes: a query term building unit 21, a judgment unit 22, a keyword matching unit 23, and a classification unit 24.

[0086] The query term building unit 21 is used to generate a second query term based on the sorting results of each query term in the pre-built first query term.

[0087] The judgment unit 22 is used to input the obtained text to be classified into the second query term library and judge whether there are keywords in the second query term library that match the text to be classified.

[0088] Keyword matching unit 23 is used to input the text to be classified into the first query term library when the judgment result is negative, and to determine at least one keyword corresponding to the text to be classified in the first query term library;

[0089] The classification unit 24 is used to classify the text to be classified based on the keywords.

[0090] In one implementation, the query term construction unit 21 is specifically used for: determining the frequency of occurrence of each query term in the first query term library; sorting each query term in the first query term library according to the frequency of occurrence of each query term in the first query term library to obtain a sorting result; and generating the second query term library according to the sorting result.

[0091] In one implementation, the query term construction unit 21 is specifically used for: acquiring historical corpus data, performing clustering processing on the historical corpus data according to a preset clustering algorithm to determine query terms; and constructing the first query term library based on the determined query terms.

[0092] In one implementation, the keyword matching unit 23 is specifically used for: performing word segmentation on the text to be classified to obtain a word sequence; filtering the word sequence according to the part of speech of each word in the word sequence to obtain a second word sequence; determining the word weight of each word in the second word sequence, filtering the second word sequence according to the word weight to obtain a third word sequence; determining the semantic similarity between each word in the third word sequence and each query word in the first query term library according to a semantic similarity algorithm; and determining the keywords corresponding to the text to be classified in the first query term library according to the semantic similarity.

[0093] In one implementation, the keyword matching unit 23 is specifically configured to: determine the first word frequency of each word in the second word sequence, wherein the first word frequency represents the frequency of occurrence of the word in the text to be classified; determine the second word frequency of each word in the second word sequence, wherein the second word frequency represents the frequency of occurrence of the word in the first query term library; and determine the word weight of each word in the second word sequence based on the first word frequency and the second word frequency.

[0094] The text classification device provided in this application mainly achieves text classification based on a pre-built first query terminology. The first query terminology stores commonly used keywords corresponding to different types obtained through clustering algorithms. For the text to be classified, a query can be performed directly in the first query terminology. Based on the query results, the keywords corresponding to the text to be classified can be determined. Then, based on the type of these keywords, the type of the text to be classified can be determined, thereby completing the classification of the text to be classified. The text classification method provided in this application does not rely on a pre-trained classification model. Therefore, compared with existing text classification methods, the text classification method provided in this application has higher classification speed and higher accuracy. To further improve text classification efficiency, in this embodiment, a second query terminology can be generated based on the ranking of each query term in the first query terminology. Before inputting the text to be classified into the first query terminology for querying, the second query terminology is used to search for matching keywords. Only when no keywords are found in the second query terminology is the text to be classified input into the first query terminology for keyword querying. Since the second query terminology contains fewer keywords and they are more important than the first query terminology, the method provided in this embodiment can further improve the efficiency of text classification.

[0095] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.

[0096] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0097] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0098] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a text classification device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:

[0099] A second query terminology is generated based on the sorting results of each query term in the pre-constructed first query terminology. The obtained text to be classified is input into the second query terminology, and it is determined whether there is a keyword in the second query terminology that matches the text to be classified. If the determination result is negative, the text to be classified is input into the first query terminology, and at least one keyword corresponding to the text to be classified in the first query terminology is determined. The text to be classified is then classified based on the keyword.

[0100] The above is as stated in this application. Figure 3 The text classification electronic device method disclosed in the illustrated embodiments can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0101] Of course, in addition to software implementation, the electronic device of this application does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0102] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by a portable electronic device including multiple applications, enable the portable electronic device to perform... Figure 1 The method of the illustrated embodiment is specifically used to perform the following operations:

[0103] A second query terminology is generated based on the sorting results of each query term in the pre-constructed first query terminology. The obtained text to be classified is input into the second query terminology, and it is determined whether there is a keyword in the second query terminology that matches the text to be classified. If the determination result is negative, the text to be classified is input into the first query terminology, and at least one keyword corresponding to the text to be classified in the first query terminology is determined. The text to be classified is then classified based on the keyword.

[0104] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0108] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0109] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0110] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0111] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0112] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A text classification method, characterized in that, include: A second query terminology is generated based on the sorting results of each query term in the pre-built first query terminology database. The obtained text to be classified is input into the second query term library, and it is determined whether there are keywords in the second query term library that match the text to be classified; When the judgment result is negative, the text to be classified is input into the first query term library to determine at least one keyword corresponding to the text to be classified in the first query term library; The text to be classified is classified based on the keywords. Specifically, generating a second query term based on each query keyword in a pre-constructed first query term database includes: Determine the frequency of occurrence of each query term in the first query term database; Determine the word weight of each query term in the first query term library; Based on the frequency of occurrence and the weight of each query term in the first query term library, the query terms in the first query term library are sorted to obtain a sorting result; Based on the sorting results, the second query terminology is generated; After obtaining the text to be classified, the obtained text data is preprocessed, including: Perform Chinese word segmentation on the obtained text data to be classified to obtain the word sequence corresponding to the text to be classified. The obtained word sequence is filtered. Text correction is performed on the filtered word sequence. After text correction, the preprocessing of the text to be classified is completed. The word weights of each word in the word sequence are calculated based on the frequency of each word in the text to be classified and the frequency of each word in the word sequence in the entire first query thesaurus. The word frequency-inverse document frequency algorithm is used to calculate the word weights of each word in the word sequence.

2. The method according to claim 1, characterized in that, The first query thesaurus is built in advance, specifically including: Obtain historical corpus data, and perform clustering processing on the historical corpus data according to a preset clustering algorithm to determine the query terms; Based on the determined query terms, construct the first query term library.

3. The method according to claim 1, characterized in that, The step of determining at least one keyword corresponding to the text to be classified in the first query thesaurus specifically includes: The text to be classified is segmented to obtain a word sequence; Based on the part of speech of each word in the word sequence, the word sequence is filtered to obtain a second word sequence; Determine the word weight of each word in the second word sequence, and filter the second word sequence according to the word weight to obtain the third word sequence; Based on the semantic similarity algorithm, the semantic similarity between each word in the third word sequence and each query word in the first query thesaurus is determined; Based on the semantic similarity, keywords corresponding to the text to be classified are determined in the first query thesaurus.

4. The method according to claim 3, characterized in that, Determining the word weight of each word in the second word sequence specifically includes: Determine the first word frequency of each word in the second word sequence, wherein the first word frequency represents the frequency of occurrence of the word in the text to be classified; Determine the second word frequency of each word in the second word sequence, wherein the second word frequency represents the frequency of occurrence of the word in the first query thesaurus; The word weights of each word in the second word sequence are determined based on the first word frequency and the second word frequency.

5. A text classification apparatus for performing the method as described in any one of claims 1 to 4, characterized in that, include: The query term building unit is used to generate a second query term based on the sorting results of each query term in the pre-built first query term. The judgment unit is used to input the acquired text to be classified into the second query term library and judge whether there are keywords in the second query term library that match the text to be classified. The keyword matching unit is used to input the text to be classified into the first query term library when the judgment result is negative, and to determine at least one keyword corresponding to the text to be classified in the first query term library; A classification unit is used to classify the text to be classified based on the keywords.

6. The apparatus according to claim 5, characterized in that, The query term construction unit is specifically used for: Determine the frequency of occurrence of each query term in the first query term database; Based on the frequency of occurrence of each query term in the first query term database, the query terms in the first query term database are sorted to obtain a sorting result; Based on the sorting results, the second query term library is generated.

7. The apparatus according to claim 5, characterized in that, The keyword matching unit is specifically used for: The text to be classified is segmented to obtain a word sequence; Based on the part of speech of each word in the word sequence, the word sequence is filtered to obtain a second word sequence; Determine the word weight of each word in the second word sequence, and filter the second word sequence according to the word weight to obtain the third word sequence; Based on the semantic similarity algorithm, the semantic similarity between each word in the third word sequence and each query word in the first query thesaurus is determined; Based on the semantic similarity, keywords corresponding to the text to be classified are determined in the first query thesaurus.

8. A text classification device, comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the method as described in any one of claims 1 to 4.

9. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the method as described in any one of claims 1 to 4.

Images