Text classification method and apparatus

Abstract

This invention discloses a text classification method and apparatus, relating to the field of computer technology. The method includes: segmenting the text to be classified into multiple semantic units, and then constructing a first word order tree based on the multiple semantic units; the first word order tree includes multiple branches, each branch storing a semantic unit combination, wherein the semantic units in the semantic unit combination are arranged according to their order of appearance in the text to be classified; comparing the first word order tree with a second word order tree corresponding to text of a known category, and determining the similarity between the text to be classified and the text of the known category based on the comparison result; and determining the category of the text to be classified based on the similarity between the text to be classified and the text of the known category. Through these steps, the recall rate of categories with smaller corpora can be improved while maintaining the overall accuracy of the classification results, making it particularly suitable for text classification scenarios with small sample sizes or severe sample imbalance problems.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a text classification method and apparatus. Background Technology

[0002] Currently, the industry primarily employs two methods for text classification: machine learning and deep learning. Machine learning methods mainly rely on models such as logistic regression (LR), support vector machines (SVM), or Naive Bayes classifiers. In practice, text classification is achieved by extracting low-level data features, mining features based on term frequency-inverse document frequency (TF-IDF), and incorporating linear transformation and activation functions. Deep learning methods primarily rely on neural network models such as convolutional neural networks (CNN) or recurrent neural networks (RNN). In practice, text classification is performed through operations such as convolution, pooling, fully connected layers, or long short-term memory.

[0003] In realizing this invention, the inventors discovered at least the following problems in the prior art: First, traditional machine learning algorithms are all based on feature mining using TF-IDF for algorithm fitting. Since TF-IDF does not consider word order, each word is recorded as a separate feature during one-hot encoding, causing the text to lose its word order, resulting in often unsatisfactory text classification performance. Second, when the corpus quality is poor and there is a serious imbalance in sample distribution (e.g., the ratio of the number of sentences in the two categories is greater than 20), deep learning algorithms have good recognition performance in categories with large corpus volumes, but in categories with small corpus volumes, they suffer from insufficient feature learning, inability to correctly distinguish features in small corpus volumes, and extremely poor learning performance. This leads to categories with small corpus volumes being almost impossible to correctly identify, resulting in extremely poor classification performance and extremely low recall. Even with methods such as sample augmentation, adding noise, and balancing samples, it is difficult to achieve satisfactory results. Summary of the Invention

[0004] In view of this, the present invention provides a text classification method and apparatus that can improve the recall rate of categories with smaller corpora while ensuring the overall accuracy of classification results, and is particularly suitable for text classification scenarios with small sample sizes or severe sample imbalance problems.

[0005] To achieve the above objectives, according to one aspect of the present invention, a text classification method is provided.

[0006] The text classification method of the present invention includes segmenting the text to be classified into multiple semantic units, and then constructing a first word order tree based on the multiple semantic units; wherein, the first word order tree includes multiple branches, each branch stores a semantic unit combination, and the semantic units in the semantic unit combination are arranged according to their order of appearance in the text to be classified; comparing the first word order tree with a second word order tree corresponding to texts of known categories in a classification corpus, and then determining the similarity between the text to be classified and the texts of known categories based on the comparison result; and determining the category of the text to be classified based on the similarity between the text to be classified and the texts of known categories.

[0007] Optionally, the step of constructing a first word order tree based on the plurality of semantic units includes: constructing a plurality of semantic unit combinations based on the plurality of semantic units, and determining the level number of each semantic unit in the semantic unit combination; the level number is obtained by sorting the semantic units in the same semantic unit combination according to their order of appearance in the text to be classified; storing the semantic units in the plurality of semantic unit combinations into nodes at the corresponding levels in the multi-branch tree according to the level number, and placing each semantic unit in the same semantic unit combination in the same branch of the multi-branch tree to obtain the first word order tree.

[0008] Optionally, the comparison result includes: semantic unit combinations in the second word order tree that match the first word order tree; the step of comparing the first word order tree with the second word order tree corresponding to the text of the known category in the classification corpus includes: obtaining all semantic unit combinations from the first word order tree, and then traversing and comparing the second word order tree corresponding to the text of the known category according to each obtained semantic unit combination, so as to determine the semantic unit combinations in the second word order tree that match the first word order tree.

[0009] Optionally, the step of determining the similarity between the text to be classified and the text of the known category based on the comparison result includes: determining the maximum length of the semantic unit combination that matches the first word order tree in the second word order tree, and then calculating the similarity between the text to be classified and the text of the known category based on the maximum length and the length of the text of the known category.

[0010] Optionally, the step of determining the category of the text to be classified based on the similarity between the text to be classified and the text of the known category includes: determining a first weight value and a second weight value for the text to be classified to belong to the known category based on the similarity between the text to be classified and each text under the same known category; wherein, the first weight value is the maximum value of the similarity between the text to be classified and each text under the same known category, and the second weight value is the average value of the similarity between the text to be classified and each text under the same known category; if the first weight value of a known category is greater than a first threshold, the known category is taken as the category of the text to be classified; otherwise, the second weight values ​​of all known categories are sorted from largest to smallest, and the top N known categories in the sorting result are taken as the category of the text to be classified; wherein, N is an integer greater than or equal to 1.

[0011] To achieve the above objectives, according to another aspect of the present invention, a text classification device is provided.

[0012] The text classification device of the present invention includes: a construction module, configured to segment the text to be classified into multiple semantic units, and then construct a first word order tree based on the multiple semantic units; wherein the first word order tree includes multiple branches, each branch stores a semantic unit combination, and the semantic units in the semantic unit combination are arranged according to their order of appearance in the text to be classified; a determination module, configured to compare the first word order tree with a second word order tree corresponding to texts of known categories in a classification corpus, and then determine the similarity between the text to be classified and the texts of the known categories based on the comparison result; and a classification module, configured to determine the category of the text to be classified based on the similarity between the text to be classified and the texts of the known categories.

[0013] Optionally, the construction module constructs a first word order tree based on the plurality of semantic units, including: the construction module constructs a plurality of semantic unit combinations based on the plurality of semantic units, and determines the level number of each semantic unit in the semantic unit combination; the level number is obtained by sorting the semantic units in the same semantic unit combination according to their order of appearance in the text to be classified; the construction module stores the semantic units in the plurality of semantic unit combinations into nodes at the corresponding levels in the multi-branch tree according to the level number, and makes each semantic unit in the same semantic unit combination located in the same branch of the multi-branch tree, so as to obtain the first word order tree.

[0014] Optionally, the comparison result includes: semantic unit combinations in the second word order tree that match the first word order tree; the comparison of the first word order tree with the second word order tree corresponding to the text of the known category in the classification corpus by the determining module includes: the determining module obtaining all semantic unit combinations from the first word order tree, and then traversing and comparing the second word order tree corresponding to the text of the known category according to each obtained semantic unit combination, so as to determine the semantic unit combinations in the second word order tree that match the first word order tree.

[0015] Optionally, the determining module determines the similarity between the text to be classified and the text of the known category based on the comparison results, including: the determining module determines the maximum length of the semantic unit combination that matches the first word order tree in the second word order tree, and then calculates the similarity between the text to be classified and the text of the known category based on the maximum length and the length of the text of the known category.

[0016] Optionally, the classification module determines the category of the text to be classified based on the similarity between the text to be classified and the text of the known category, including: the classification module determines a first weight value and a second weight value for the text to be classified to belong to the known category based on the similarity between the text to be classified and each text under the same known category; wherein, the first weight value is the maximum similarity between the text to be classified and each text under the same known category, and the second weight value is the average similarity between the text to be classified and each text under the same known category; if the first weight value of a known category is greater than a first threshold, the classification module takes that known category as the category of the text to be classified; otherwise, the classification module sorts the second weight values ​​of all known categories from largest to smallest, and takes the top N known categories in the sorting result as the category of the text to be classified; wherein, N is an integer greater than or equal to.

[0017] To achieve the above objectives, according to another aspect of the present invention, an electronic device is provided.

[0018] The electronic device of the present invention includes: one or more processors; and a storage device for storing one or more programs; wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the text classification method of the present invention.

[0019] To achieve the above objectives, according to another aspect of the present invention, a computer-readable medium is provided.

[0020] The present invention provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the text classification method of the present invention.

[0021] One embodiment of the above invention has the following advantages or beneficial effects: by segmenting the text to be classified into multiple semantic units, and then constructing a first word order tree based on the multiple semantic units, comparing the first word order tree with the second word order tree corresponding to the text of known categories in the classification corpus, and then determining the similarity between the text to be classified and the text of the known categories based on the comparison results, and then determining the category of the text to be classified based on the similarity between the text to be classified and the text of the known categories, it is possible to improve the recall rate of categories with smaller corpora while ensuring the overall accuracy of the classification results, which is especially suitable for text classification scenarios with small sample sizes or serious sample imbalance problems.

[0022] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0023] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0024] Figure 1 This is a schematic diagram of the main flow of a text classification method according to an embodiment of the present invention;

[0025] Figure 2 This is a schematic diagram of the main flow of a text classification method according to another embodiment of the present invention;

[0026] Figure 3 This is a schematic diagram of an optional structure of the word order tree according to an embodiment of the present invention;

[0027] Figure 4 This is a schematic diagram of an optional structure of the word order tree according to an embodiment of the present invention;

[0028] Figure 5 This is a schematic diagram of the main modules of a text classification device according to an embodiment of the present invention;

[0029] Figure 6 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0030] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing the embodiments of the present invention. Detailed Implementation

[0031] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0032] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0033] Figure 1 This is a schematic diagram of the main flow of a text classification method according to an embodiment of the present invention. Figure 1 As shown, the text classification method of this invention includes:

[0034] Step S101: Divide the text to be classified into multiple semantic units, and then construct a first word order tree based on the multiple semantic units.

[0035] The text to be classified can be a sentence entered by the user, such as "I want to go eat," "What types of insurance are there," or "What smartphones are available for around 1000 yuan?" In this step, when segmenting the text, if segmentation is done by word, the semantic unit is the segmented word; if segmentation is done by character, the semantic unit is the segmented character. For example, assuming the text to be classified is "I want to go eat," if segmented using a word segmentation tool like Jieba, the semantic unit is the segmented word, such as "I," "want to," and "eat." If segmented by character, the semantic unit is the segmented character, such as "I," "want," "go," "eat," and "food." Similarly, assuming the text to be classified is "What types of insurance are there?" if segmented using a word segmentation tool like Jieba, the semantic unit is the segmented word, such as "insurance," "have," "which," and "types." If segmented by character, the semantic unit is the segmented character, such as "insurance," "risk," "have," "which," "some," "kind," and "category."

[0036] After segmenting the text into multiple semantic units, a first word order tree can be constructed based on these semantic units. This first word order tree includes multiple branches, each storing a combination of semantic units. The semantic units in each combination are arranged according to their order of appearance in the text to be classified. For example, assuming the text to be classified is "What types of insurance are there?", if a branch of the first word order tree stores the semantic unit combination [insurance, have, types], then the three nodes of that branch from top to bottom can sequentially store "insurance", "have", and "types".

[0037] In this embodiment of the invention, the first word order tree constructed through this step retains the word order features of the text to be classified, which helps to improve the accuracy of subsequent text classification.

[0038] Step S102: Compare the first word order tree with the second word order tree corresponding to the text of the known category in the classification corpus, and then determine the similarity between the text to be classified and the text of the known category based on the comparison result.

[0039] For example, suppose the classification corpus contains two text sets with known categories: Category 1: Pre-purchase insurance consultation; text set under Category 1: [How to buy insurance, what types of insurance are available, how much does insurance cost annually]; Category 2: Post-purchase insurance consultation; text set under Category 2: [How to file a claim, can I get insurance if I'm hit by a car, which number to call for insurance processing]. Then, step S102 can be used to calculate the similarity between the text to be classified and the texts of each known category in the classification corpus, specifically the similarity between the text to be classified and the texts in Category 1 ("How to buy insurance"), "What types of insurance are available?", "How much does insurance cost annually?", "How to file a claim", "Can I get insurance if I'm hit by a car?", and "Which number to call for insurance processing?".

[0040] Step S103: Determine the category of the text to be classified based on the similarity between the text to be classified and the text of the known categories.

[0041] In an optional implementation, step S103 includes: determining a first weight value and a second weight value for the text to be classified to belong to a known category based on the similarity between the text to be classified and each text under the same known category; wherein, the first weight value is the maximum similarity between the text to be classified and each text under the same known category, and the second weight value is the average similarity between the text to be classified and each text under the same known category; if the first weight value of a known category is greater than a first threshold, the known category is taken as the category of the text to be classified; otherwise, the second weight values ​​of all known categories are sorted from largest to smallest, and the top N known categories in the sorting result are taken as the category of the text to be classified; wherein, N is an integer greater than or equal to 1.

[0042] In another optional implementation, step S103 includes: determining a second weight value for a known category based on the similarity between the text to be classified and each text in the same known category; wherein the second weight value is the average of the similarity between the text to be classified and each text in the same known category; sorting the second weight values ​​of all known categories from largest to smallest, and taking the top N known categories in the sorting result as the category of the text to be classified; wherein N is an integer greater than or equal to 1.

[0043] In this embodiment of the invention, the above steps can improve the recall rate of categories with smaller corpora while ensuring the overall accuracy of the classification results. This is especially suitable for text classification scenarios with a small number of samples or severe sample imbalance.

[0044] Figure 2 This is a schematic diagram of the main flow of a text classification method according to another embodiment of the present invention. Figure 2 As shown, the text classification method of this invention includes:

[0045] Step S201: Divide the text to be classified into multiple semantic units.

[0046] The text to be classified can be a sentence entered by the user, such as "I want to go out to eat", "What types of insurance are there?", "What smartphones are available for around 1000 yuan?", etc. In this step, when segmenting the text to be classified, if the segmentation is done by word, the semantic unit is the segmented word; if the segmentation is done by character, the semantic unit is the segmented character.

[0047] For example, suppose the text to be classified is "I want to go eat". If it is segmented using a word segmentation tool such as Jieba, the semantic units are the segmented words, such as "I", "want to", and "eat". If it is segmented by character, the semantic units are the segmented characters, such as "I", "want", "go", "eat", and "rice". Similarly, suppose the text to be classified is "What kinds of insurance are there?". If it is segmented using a word segmentation tool such as Jieba, the semantic units are the segmented words, such as "insurance", "have", "which", and "types". If it is segmented by character, the semantic units are the segmented characters, such as "insurance", "risk", "have", "which", "some", "kind", and "category".

[0048] Step S202: Construct multiple semantic unit combinations based on the multiple semantic units, and determine the hierarchical number of each semantic unit in the semantic unit combination.

[0049] In one optional implementation, the multiple semantic units can be recursively traversed sequentially using a breadth-first search method to obtain combinations of the multiple semantic units. For example, assuming the text to be classified is "I want to go eat", the multiple semantic units obtained through step S201 are [I, want, go, eat, return], and the length of the semantic unit combination is 1 to 5. Then, all semantic unit combinations of each length can be exhaustively enumerated using a breadth-first search method, such as semantic unit combination [I, want, go, eat, rice], semantic unit combination [I, want, eat, rice], semantic unit combination [go, eat, rice], semantic unit combination [eat, rice], etc.

[0050] While constructing multiple semantic unit combinations, the semantic units in the same semantic unit combination can be sorted according to their order of appearance in the text to be classified, so as to obtain the hierarchical number of each semantic unit in the semantic unit combination. For example, in the semantic unit combination [I, want, go, eat, rice], there are 5 semantic units. After sorting them according to the order in which they appear in the text to be classified, we get the following result: [(I, 1), (want, 2), (go, 3), (eat, 4), (rice, 5)] (This result indicates that in this semantic unit combination, the level number of "I" is 1, the level number of "want" is 2, the level number of "go" is 3, the level number of "eat" is 4, and the level number of "rice" is 5). In the semantic unit combination [go, eat, rice], there are 3 semantic units. After sorting them according to the order in which they appear in the text to be classified, we get the following result: [(go, 1), (eat, 2), (rice, 3)] (This result indicates that in this semantic unit combination, the level number of "go" is 1, the level number of "eat" is 2, and the level number of "rice" is 3).

[0051] Step S203: Store the semantic units in the multiple semantic unit combinations into the nodes of the corresponding levels in the multi-branch tree according to the level number, and make each semantic unit in the same semantic unit combination be located in the same branch of the multi-branch tree to obtain the first word order tree.

[0052] For example, assuming the text to be classified is "I want to go eat", a semantic unit combination obtained through step S202 satisfies: [(I, 1), (want, 2), (go, 3), (eat, 4), (food, 5)]. Then, according to the size of the hierarchy number, "I" can be filled into the first-level node (the child node of the root node) of a branch, "want" can be filled into the second-level node of the same branch, "go" can be filled into the third-level node of the same branch, "eat" can be filled into the fourth-level node of the same branch, and "food" can be filled into the fifth-level node of the same branch. Another semantic unit combination obtained through step S202 satisfies: [(go, 1), (eat, 2), (food, 3)]. Then, according to the size of the hierarchy number, "go" can be filled into the first-level node of another branch, "eat" can be filled into the second-level node of the other branch, and "food" can be filled into the third-level node of the other branch. By iteratively performing the above filling steps for each semantic unit combination obtained through step S202, the first word order tree can be obtained.

[0053] In this embodiment of the invention, the first word order tree constructed through this step retains the word order features of the text to be classified, which helps to improve the accuracy of subsequent text classification.

[0054] Step S204: Obtain all semantic unit combinations from the first word order tree, and then traverse and compare the second word order tree corresponding to the known category of text according to each obtained semantic unit combination to determine the semantic unit combination in the second word order tree that matches the first word order tree.

[0055] In an optional implementation, step S204 specifically includes: A. For each semantic unit combination obtained from the first word order tree, based on the level number L of the current semantic unit, find all nodes in the second word order tree corresponding to the known category of text in the L level; B. If the current semantic unit exists in a node in the L level of the second word order tree, it is considered to match a node, and then based on the level number L+1 of the next semantic unit, find all child nodes of the matching node in the L level; if the current semantic unit does not exist in a node in the L level of the second word order tree, the semantic unit combination fails to match; C. If the next semantic unit exists in a child node of the matching node in the L level, it is considered to match another node; D. If all semantic units in the semantic unit combination have matching nodes in the second word order tree, it is considered that there is a semantic unit combination that matches it in the second word order tree; E. After traversing and comparing all semantic unit combinations in the first word order tree according to steps A to D, all semantic unit combinations in the second word order tree that match the first word order tree can be obtained.

[0056] In an embodiment of the present invention, semantic unit combinations of the text to be classified are stored based on the first word order tree, and the matching semantic unit combinations are determined based on the first word order tree and the second word order tree, which can reduce the space utilization rate and speed up the matching query speed.

[0057] In specific implementation, before step S204, the method of the embodiment of the present invention may further include the following steps: constructing corresponding second word order trees for texts of each known category in the classification corpus. For example, there are two text sets of known categories in the classification corpus, namely: Category 1: Consultation before insurance purchase; the text set under Category 1 [How to buy insurance, what types of insurance are there, how much insurance needs to be purchased annually]; Category 2: Consultation after insurance purchase; the text set under Category 2 [How to claim compensation, can I be insured if I am hit by a car, which phone number to call for insurance processing]. Then, corresponding second word order trees can be constructed for these six texts of known categories respectively. The second word order tree includes multiple branches, each branch stores a semantic unit combination, and each semantic unit in the semantic unit combination is arranged in the order in which it appears in the text of the known category;

[0058] Step S205: Determine the maximum length of the semantic unit combination that exists in the second word order tree and matches the first word order tree, and then calculate the similarity between the text to be classified and the text of the known category according to the maximum length and the length of the text of the known category.

[0059] Among them, the length of the semantic unit combination can be defined as the total number of characters included in the semantic unit combination. For example, assume that the semantic unit combinations that exist in the second word order tree and match the first word order tree are [I, want, to, eat], [want, to, eat], and [to, eat], then the lengths of these three matching semantic unit combinations are 4, 3, and 2 respectively, and the maximum length among these three matching semantic unit combinations is 4.

[0060] Further, the similarity between the text to be classified and the text of the known category can be calculated according to the following formula:

[0061]

[0062] Among them, C j-i represents the similarity between the text to be classified and the text of the known category, L i-i,max represents the maximum length of the semantic unit combination that exists in the second word order tree and matches the first word order tree, L i represents the length of the text of the known category.

[0063] For example, suppose the classification corpus contains two text sets with known categories: Category 1: Pre-purchase insurance consultation; text set under Category 1: [How to buy insurance, what types of insurance are available, how much does insurance cost annually]; Category 2: Post-purchase insurance consultation; text set under Category 2: [How to file a claim, can I get insurance if I'm hit by a car, which number to call for insurance processing]. Then, step S205 can calculate the similarity between the text to be classified and the texts of each known category in the classification corpus, specifically the similarity between the text to be classified and the texts in Category 1 ("How to buy insurance"), "What types of insurance are available?", "How much does insurance cost annually?", "How to file a claim", "Can I get insurance if I'm hit by a car?", and "Which number to call for insurance processing?".

[0064] Step S206: Determine the category of the text to be classified based on the similarity between the text to be classified and the text of the known categories.

[0065] In an optional implementation, step S206 includes: determining a first weight value and a second weight value for the text to be classified to belong to a known category based on the similarity between the text to be classified and each text under the same known category; wherein, the first weight value is the maximum value of the similarity between the text to be classified and each text under the same known category, and the second weight value is the average value of the similarity between the text to be classified and each text under the same known category; if the first weight value of a known category is greater than a first threshold, the known category is taken as the category of the text to be classified; otherwise, the second weight values ​​of all known categories are sorted from largest to smallest, and the top N known categories in the sorting result are taken as the category of the text to be classified; wherein, N is an integer greater than or equal to 1.

[0066] For example, assuming the first threshold is set to 1 and N is set to 1, the similarity between the text to be processed and each text in category 1, calculated through step S205, is 0.8, 0.4, and 0.6, respectively. The similarity between the text to be processed and each text in category 2 is 0.6, 0.4, 0.6, and 0.2, respectively. Then, the first weight value of category 1 is 0.8, the second weight value of category 1 is 0.6, the first weight value of category 2 is 0.6, and the second weight value of category 2 is 0.45. Since the first weight values ​​of category 1 and category 2 are both less than the first threshold, and the second weight value of category 1 is greater than the second weight value of category 2, the second category is chosen as the category of the text to be processed.

[0067] In this embodiment of the invention, the above steps determine the similarity between the text to be classified and texts under each known category, and calculate the weight value of the text to be classified belonging to each known category based on the similarity, thereby accurately determining the category to which the text to be classified belongs. Compared with the prior art, this invention, through the above steps, can improve the recall rate of categories with smaller corpora while ensuring the overall accuracy of the classification results, and is particularly suitable for text classification scenarios with small sample sizes or severe sample imbalance problems.

[0068] Figure 3 This is a schematic diagram of an optional structure of the word order tree according to an embodiment of the present invention. Figure 3 The word order tree is constructed using the text "I want to go eat" as an example. If the text "I want to go eat" is the text to be classified, then... Figure 3 The word order tree shown is the first word order tree; if the text "I want to go eat" is a text of a known category, then... Figure 3 The word order tree shown is a second word order tree. For example... Figure 3 As shown, it includes multiple branches from the root node (i.e., the root node in the diagram) to the leaf nodes. Each branch stores a semantic unit combination, and the semantic units in this combination are filled into the nodes of the corresponding level of that branch according to their level numbers. For example, in Figure 3 In the semantic unit combination [(I, 1), (Think, 2), (Go, 3), (Eat, 4), (Food, 5)] stored in the leftmost branch, the semantic unit "I" is filled in the first-level node, the semantic unit "Think" is filled in the second-level node in this branch, the semantic unit "Go" is filled in the third-level node in this branch, the semantic unit "Eat" is filled in the fourth-level node in this branch, and the semantic unit "Food" is filled in the fifth-level node in this branch.

[0069] Figure 4 This is a schematic diagram of an optional structure of the word order tree according to an embodiment of the present invention. Figure 4 The word order tree is constructed using the text "Let's go to Lanzhou to eat Lanzhou noodles" as an example. If the text "Let's go to Lanzhou to eat Lanzhou noodles" is the text to be classified, then... Figure 4 The word order tree shown is the first word order tree; if the text "Let's go to Lanzhou to eat Lanzhou noodles" is a text of a known category, then... Figure 4 The word order tree shown is a second word order tree. For example... Figure 4 As shown, it includes multiple branches from the root node (i.e., the root node in the diagram) to the leaf nodes. Each branch stores a semantic unit combination, and the semantic units in this combination are filled into the nodes of the corresponding level of that branch according to their level numbers. For example, in Figure 4In the semantic unit combination [(Together, 1), (Go, 2), (Lanzhou, 3), (Eat, 4), (Ramen, 5)] stored in the leftmost branch, the semantic unit "Together" fills the nodes in the first layer, the semantic unit "Go" fills the nodes in the second layer of the branch, the semantic unit "Lanzhou" fills the nodes in the third layer of the branch, the semantic unit "Eat" fills the nodes in the fourth layer of the branch, and the semantic unit "Ramen" fills the nodes in the fifth layer of the branch.

[0070] Figure 5 This is a schematic diagram of the main modules of a text classification device according to an embodiment of the present invention. Figure 5 As shown, the text classification device 500 of this embodiment includes: a construction module 501, a determination module 502, and a classification module 503.

[0071] The construction module 501 is used to segment the text to be classified into multiple semantic units, and then construct a first word order tree based on the multiple semantic units. The first word order tree includes multiple branches, each branch storing a semantic unit combination, and the semantic units in the semantic unit combination are arranged according to their order of appearance in the text to be classified.

[0072] The text to be classified can be a sentence entered by the user, such as "I want to go eat," "What types of insurance are there," or "What smartphones are available for around 1000 yuan?" Specifically, when the construction module 501 segments the text to be classified, if it segments by word, the semantic unit is the segmented word; if it segments by character, the semantic unit is the segmented character. For example, assuming the text to be classified is "I want to go eat," if it is segmented using a word segmentation tool like Jieba, the semantic unit is the segmented word, such as "I," "want to," and "eat." If it is segmented by character, the semantic unit is the segmented character, such as "I," "want," "go," "eat," and "rice." Similarly, assuming the text to be classified is "What types of insurance are there?" if it is segmented using a word segmentation tool like Jieba, the semantic unit is the segmented word, such as "insurance," "have," "which," and "types." If it is segmented by character, the semantic unit is the segmented character, such as "insurance," "risk," "have," "which," "some," "kind," and "category."

[0073] Furthermore, after segmenting to obtain multiple semantic units, the construction module 501 can construct a first word order tree based on the following method: the construction module 501 constructs multiple semantic unit combinations based on the multiple semantic units, and determines the level number of each semantic unit in the semantic unit combination; the level number is obtained by sorting the semantic units in the same semantic unit combination according to their order of appearance in the text to be classified; the construction module 501 stores the semantic units in the multiple semantic unit combinations into nodes of the corresponding level in the multi-branch tree according to the level number, and makes each semantic unit in the same semantic unit combination located in the same branch of the multi-branch tree to obtain the first word order tree.

[0074] In this embodiment of the invention, the first word order tree constructed by the construction module retains the word order features of the text to be classified, which helps to improve the accuracy of subsequent text classification.

[0075] The determination module 502 is used to compare the first word order tree with the second word order tree corresponding to the text of the known category in the classification corpus, and then determine the similarity between the text to be classified and the text of the known category based on the comparison result.

[0076] For example, the comparison result includes: semantic unit combinations in the second word order tree that match the first word order tree. In this example, the comparison of the first word order tree and the second word order tree corresponding to the text of the known category in the classification corpus by the determining module 502 may specifically include: the determining module 502 obtaining all semantic unit combinations from the first word order tree, and then traversing and comparing the second word order tree corresponding to the text of the known category according to each obtained semantic unit combination, in order to determine the semantic unit combinations in the second word order tree that match the first word order tree.

[0077] In this embodiment of the invention, the semantic unit combination of the text to be classified is stored based on the first word order tree, and the matching semantic unit combination is determined based on the first word order tree and the second word order tree, which can reduce the space utilization and speed up the matching query.

[0078] Furthermore, in this example, the determination module 502's determination of the similarity between the text to be classified and the text of the known category based on the comparison results may specifically include: the determination module 502 determining the maximum length of the semantic unit combination that matches the first word order tree in the second word order tree, and then calculating the similarity between the text to be classified and the text of the known category based on the maximum length and the length of the text of the known category.

[0079] Among them, the length of the semantic unit combination can be defined as the total number of characters included in the semantic unit combination. For example, assume that the semantic unit combinations existing in the second word order tree and matching the first word order tree are [I, want, eat, rice], [want, eat, rice], and [eat, rice]. Then the lengths of these three matching semantic unit combinations are 4, 3, and 2 respectively, and the maximum length among these three matching semantic unit combinations is 4. Further, the determination module 502 can calculate the similarity between the text to be classified and the text of the known category according to the following formula:

[0080]

[0081] where C j-i represents the similarity between the text to be classified and the text of the known category, L i-i,max represents the maximum length of the semantic unit combinations existing in the second word order tree and matching the first word order tree, and L i represents the length of the text of the known category.

[0082] For example, assume that there are two text sets of known categories in the classification corpus, which are: Category 1: Consultation before insurance purchase; the text set under Category 1 [How to buy insurance, What types of insurance are there, How much insurance needs to be purchased annually]; Category 2: Consultation after insurance purchase; the text set under Category 2 [How to make a claim, Can I be insured if I am hit by a car, Which phone number to call for insurance handling]. Then the similarity between the text to be classified and each text of the known category in the classification corpus can be calculated through step S205, that is, the similarity between the text to be classified and the text "How to buy insurance" of Category 1, the similarity between the text to be classified and the text "What types of insurance are there" of Category 1, the similarity between the text to be classified and the text "How much insurance needs to be purchased annually" of Category 1, the similarity between the text to be classified and the text "How to make a claim" of Category 2, the similarity between the text to be classified and the text "Can I be insured if I am hit by a car" of Category 2, and the similarity between the text to be classified and the text "Which phone number to call for insurance handling" of Category 2.

[0083] The classification module 503 is used to determine the category of the text to be classified according to the similarity between the text to be classified and the text of the known category.

[0084] In an optional implementation, the classification module 503 determines the category of the text to be classified based on the similarity between the text to be classified and the text of the known category, specifically including: the classification module 503 determines a first weight value and a second weight value for the text to be classified to belong to the known category based on the similarity between the text to be classified and each text under the same known category; wherein, the first weight value is the maximum value of the similarity between the text to be classified and each text under the same known category, and the second weight value is the average value of the similarity between the text to be classified and each text under the same known category; if the first weight value of a known category is greater than a first threshold, the classification module 503 takes that known category as the category of the text to be classified; otherwise, the classification module 503 sorts the second weight values ​​of all known categories from largest to smallest, and takes the top N known categories in the sorting result as the category of the text to be classified; wherein, N is an integer greater than or equal to 1.

[0085] In another optional implementation, the classification module 503 determines the category of the text to be classified based on the similarity between the text to be classified and the text of the known category, specifically including: the classification module 503 determines a second weight value of the known category based on the similarity between the text to be classified and each text under the same known category; wherein, the second weight value is the average of the similarity between the text to be classified and each text under the same known category; the classification module 503 sorts the second weight values ​​of all known categories from largest to smallest, and takes the top N known categories in the sorting result as the category of the text to be classified; wherein, N is an integer greater than or equal to 1.

[0086] In this embodiment of the invention, the above device can improve the recall rate of categories with smaller corpora while ensuring the overall accuracy of the classification results. It is especially suitable for text classification scenarios with a small number of samples or serious sample imbalance problems.

[0087] Figure 6 An exemplary system architecture 600 is shown that can be applied to the text classification method or text classification apparatus of the present invention.

[0088] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, and 603, a network 604, and a server 605. Network 604 serves as the medium for providing communication links between terminal devices 601, 602, and 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0089] Users can use terminal devices 601, 602, and 603 to interact with server 605 via network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0090] Terminal devices 601, 602, and 603 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0091] Server 605 can be a server providing various services, such as a backend management server supporting shopping websites browsed by users using terminal devices 601, 602, and 603. The backend management server can perform intent recognition processing such as text classification on the received input questions, then generate a matching response, and feed back the processing result (such as the generated matching response) to the terminal device.

[0092] It should be noted that the text classification method provided in this embodiment of the invention is generally executed by server 605, and correspondingly, the text classification device is generally set in server 605.

[0093] It should be understood that Figure 6 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0094] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing an electronic device according to embodiments of the present invention. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0095] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0096] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0097] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined above in the system of this invention.

[0098] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0099] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0100] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including a construction module, a determination module, and a classification module. The names of these modules do not necessarily limit the module itself; for example, the construction module can also be described as "a module that constructs a first word order tree corresponding to the text to be classified."

[0101] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to perform the following process: segmenting the text to be classified into multiple semantic units, and then constructing a first word order tree based on the multiple semantic units; wherein the first word order tree includes multiple branches, each branch storing a semantic unit combination, and the semantic units in the semantic unit combination are arranged according to their order of appearance in the text to be classified; comparing the first word order tree with a second word order tree corresponding to texts of known categories in a classification corpus, and then determining the similarity between the text to be classified and the texts of the known categories based on the comparison result; and determining the category of the text to be classified based on the similarity between the text to be classified and the texts of the known categories.

[0102] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A text classification method, characterized in that, The method includes: The text to be classified is divided into multiple semantic units based on words or characters, and then a first word order tree is constructed based on the multiple semantic units; wherein, the first word order tree includes multiple branches, each branch stores a semantic unit combination, and the semantic units in the semantic unit combination are arranged according to the order in which they appear in the text to be classified; The first word order tree is compared with the second word order tree corresponding to the text of the known category in the classification corpus, and then the similarity between the text to be classified and the text of the known category is determined based on the comparison results. The category of the text to be classified is determined based on the similarity between the text to be classified and the text of the known categories.

2. The method according to claim 1, characterized in that, The step of constructing the first word order tree based on the plurality of semantic units includes: Multiple semantic unit combinations are constructed based on the multiple semantic units, and the level number of each semantic unit in the semantic unit combination is determined. The level number is obtained by sorting the semantic units in the same semantic unit combination according to their order of appearance in the text to be classified. The semantic units in the multiple semantic unit combinations are stored in the corresponding level nodes in the multi-branch tree according to the level number, and each semantic unit in the same semantic unit combination is located in the same branch of the multi-branch tree to obtain the first word order tree.

3. The method according to claim 1, characterized in that, The comparison results include: combinations of semantic units in the second word order tree that match the first word order tree; The step of comparing the first word order tree with the second word order tree corresponding to the text of the known category in the classification corpus includes: obtaining all semantic unit combinations from the first word order tree, and then traversing and comparing the second word order tree corresponding to the text of the known category according to each obtained semantic unit combination, so as to determine the semantic unit combination in the second word order tree that matches the first word order tree.

4. The method according to claim 3, characterized in that, The step of determining the similarity between the text to be classified and the text of the known category based on the comparison results includes: determining the maximum length of the semantic unit combination that matches the first word order tree in the second word order tree, and then calculating the similarity between the text to be classified and the text of the known category based on the maximum length and the length of the text of the known category.

5. The method according to claim 1, characterized in that, The step of determining the category of the text to be classified based on its similarity to texts of known categories includes: The first weight value and the second weight value of the text to be classified are determined based on the similarity between the text to be classified and each text in the same known category. The first weight value is the maximum similarity between the text to be classified and each text in the same known category, and the second weight value is the average similarity between the text to be classified and each text in the same known category. If the first weight value of a known category is greater than a first threshold, the known category is taken as the category of the text to be classified. Otherwise, the second weight values ​​of all known categories are sorted from largest to smallest, and the top N known categories in the sorting result are taken as the category of the text to be classified. Here, N is an integer greater than or equal to 1.

6. A text classification device, characterized in that, The device includes: The construction module is used to segment the text to be classified into multiple semantic units based on words or characters, and then construct a first word order tree based on the multiple semantic units; wherein, the first word order tree includes multiple branches, each branch stores a semantic unit combination, and the semantic units in the semantic unit combination are arranged according to the order in which they appear in the text to be classified; The determination module is used to compare the first word order tree with the second word order tree corresponding to the text of known categories in the classification corpus, and then determine the similarity between the text to be classified and the text of known categories based on the comparison results; The classification module is used to determine the category of the text to be classified based on the similarity between the text to be classified and the text of the known categories.

7. The apparatus according to claim 6, characterized in that, The construction module constructs a first word order tree based on the plurality of semantic units, including: The construction module constructs multiple semantic unit combinations based on the multiple semantic units, and determines the level number of each semantic unit in the semantic unit combination; the level number is obtained by sorting the semantic units in the same semantic unit combination according to their order of appearance in the text to be classified; the construction module stores the semantic units in the multiple semantic unit combinations into nodes of the corresponding level in the multi-branch tree according to the level number, and makes each semantic unit in the same semantic unit combination located in the same branch of the multi-branch tree to obtain the first word order tree.

8. The apparatus according to claim 6, characterized in that, The comparison results include: combinations of semantic units in the second word order tree that match the first word order tree; The determination module compares the first word order tree with the second word order tree corresponding to the text of the known category in the classification corpus, including: the determination module obtains all semantic unit combinations from the first word order tree, and then traverses and compares the second word order tree corresponding to the text of the known category according to each obtained semantic unit combination, so as to determine the semantic unit combination in the second word order tree that matches the first word order tree.

9. The apparatus according to claim 8, characterized in that, The determination module determines the similarity between the text to be classified and the text of the known category based on the comparison results, including: the determination module determines the maximum length of the semantic unit combination that matches the first word order tree in the second word order tree, and then calculates the similarity between the text to be classified and the text of the known category based on the maximum length and the length of the text of the known category.

10. The apparatus according to claim 6, characterized in that, The classification module determines the category of the text to be classified based on the similarity between the text to be classified and the text of known categories, including: The classification module determines a first weight value and a second weight value for the text to be classified to belong to a known category based on the similarity between the text to be classified and each text in the same known category. The first weight value is the maximum similarity between the text to be classified and each text in the same known category, and the second weight value is the average similarity between the text to be classified and each text in the same known category. If the first weight value of a known category is greater than a first threshold, the classification module assigns that known category to the text to be classified. Otherwise, the classification module sorts all known categories by their second weight values ​​from largest to smallest and assigns the top N known categories from the sorting results to the text to be classified. Here, N is an integer greater than or equal to N.

11. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1 to 5.

12. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 5.

Images