Text classification method, device, electronic device and storage medium

Abstract

The present application relates to the technical field of machine learning, and discloses a text classification method, device, electronic device and storage medium. The text classification method comprises: obtaining a target feature vector of a text to be processed and at least two text sets, wherein each text set corresponds to a category, and each text set includes feature vectors of text data belonging to the same category; for each text set, obtaining K nearest neighbor feature vectors of the target feature vector from each text set, obtaining a degree of aggregation between the target feature vector and each text set based on the K nearest neighbor feature vectors and the target feature vector, and obtaining a comparison result of the degree of aggregation and a category aggregation degree of each text set; determining a target text set from at least two text sets based on the comparison result corresponding to each text set; determining the category of the target text set as the category of the text to be processed, thereby improving the accuracy of text classification.

Description

Technical Field

[0001] The present application relates to the technical field of machine learning, and in particular to a text classification method, device, electronic device and storage medium. Background Art

[0002] Text classification model is one of the important applications in the field of artificial intelligence. Text classification model can identify the category to which the text belongs. Text classification model has a wide range of applications in intelligent customer service, news recommendation, intent recognition system, etc., that is, text classification model is the basic component of these complex systems.

[0003] At present, in many text classification tasks, the KNN (k-Nearest Neighbor) algorithm is usually selected to quickly build a text classification model based on existing text data. The basic idea is to compare the feature vector of the text to be processed with the feature vector of the text data in the training set when the classification of the text data in the training set is known, and find the top K text data in the training set that are most similar to the text to be processed, and take the classification that appears most times in these K text data as the category to which the text to be processed belongs.

[0004] However, the classification results of the text classification model based on the KNN algorithm are extremely sensitive to the selection of the K value and the distribution of text data in the training set. In actual applications, for some application scenarios with fewer samples, it is easy to have an unbalanced distribution of text data in each category, which seriously reduces the accuracy of text classification. Summary of the invention

[0005] The embodiments of the present application provide a text classification method, device, electronic device and storage medium, which can reduce the impact of the value of K on the classification results, improve the problem of inaccurate classification results due to different data distribution or data imbalance, and improve the accuracy of text classification.

[0006] In a first aspect, an embodiment of the present application provides a text classification method, including:

[0007] Obtaining a target feature vector of a text to be processed and at least two text sets, wherein each text set corresponds to a category, and each text set includes feature vectors of text data belonging to the same category;

[0008] For each text set, K nearest neighbor feature vectors of the target feature vector are obtained from each text set, and based on the K nearest neighbor feature vectors and the target feature vector, a degree of aggregation between the target feature vector and each text set is obtained, and a comparison result of the degree of aggregation and the class aggregation degree of each text set is obtained; wherein the degree of aggregation represents the density between the target feature vector and the K nearest neighbor feature vectors, and the class aggregation represents the density of feature vector distribution in the same text set;

[0009] Determining a target text set from the at least two text sets based on the comparison results corresponding to each text set;

[0010] The category of the target text set is determined as the category of the text to be processed.

[0011] Optionally, obtaining K nearest neighbor feature vectors of the target feature vector from each text set specifically includes:

[0012] Obtaining the similarity between each feature vector in each text set and the target feature vector;

[0013] According to the order from large to small similarity, the top K feature vectors are determined as the K nearest neighbor feature vectors of the target feature vector.

[0014] Optionally, obtaining the degree of aggregation between the target feature vector and each text set based on the k nearest neighbor feature vectors and the target feature vector specifically includes:

[0015] Obtaining the similarity between each feature vector of the K nearest neighbor feature vectors and the target feature vector;

[0016] The average value of the similarities corresponding to the K nearest feature vectors is determined as the degree of aggregation between the target feature vector and each text set.

[0017] Optionally, the comparison result is a ratio of the aggregation degree to the category aggregation degree, and the step of determining a target text set from the at least two text sets based on the comparison result corresponding to each text set specifically includes:

[0018] Based on the comparison results corresponding to each text set, determining the classification probability that the text to be processed belongs to each text set;

[0019] The text set with the highest classification probability is determined as the target text set.

[0020] Optionally, the category aggregation degree of each text set is obtained in the following manner:

[0021] For each feature vector in each text set, obtain the similarity between each feature vector and other feature vectors in each text set, sort the obtained similarities from large to small, and determine the degree of aggregation between each feature vector and each text set based on the top K similarities;

[0022] Based on the aggregation degree corresponding to each feature vector in each text set, the category aggregation degree corresponding to each text set is determined.

[0023] Optionally, determining the degree of aggregation between each feature vector and each text set based on the top K similarities in the ranking specifically includes:

[0024] The average value of the top K similarities is determined as the degree of aggregation between each feature vector and each text set.

[0025] Optionally, determining the category aggregation degree corresponding to each text set based on the aggregation degree corresponding to each feature vector in each text set specifically includes:

[0026] The average value of the aggregation degrees corresponding to the feature vectors in each text set is determined as the category aggregation degree corresponding to each text set.

[0027] Optionally, each category corresponds to a user intent.

[0028] In a second aspect, an embodiment of the present application provides a text classification device, including:

[0029] An acquisition module, used for obtaining a target feature vector of a text to be processed and at least two text sets, wherein each text set corresponds to a category, and each text set includes feature vectors of text data belonging to the same category;

[0030] A degree of aggregation calculation module is used to obtain, for each text set, K nearest neighbor feature vectors of the target feature vector from each text set, obtain the degree of aggregation between the target feature vector and each text set based on the K nearest neighbor feature vectors and the target feature vector, and obtain a comparison result of the degree of aggregation and the category aggregation degree of each text set; wherein the degree of aggregation represents the density between the target feature vector and the K nearest neighbor feature vectors, and the category aggregation represents the density of feature vector distribution in the same text set;

[0031] The classification module is used to determine a target text set from the at least two text sets based on the comparison result corresponding to each text set, and determine the category of the target text set as the category of the text to be processed.

[0032] Optionally, the aggregation degree calculation module is specifically used to: obtain the similarity between each feature vector in each text set and the target feature vector; and sort the feature vectors with the highest similarity from large to small, and determine the top K feature vectors as the K nearest neighbor feature vectors of the target feature vector.

[0033] Optionally, the aggregation degree calculation module is specifically used to: obtain the similarity between each feature vector of the K nearest neighbor feature vectors and the target feature vector; and determine the average value of the similarities corresponding to the K nearest neighbor feature vectors as the aggregation degree between the target feature vector and each text set.

[0034] Optionally, the comparison result is the ratio of the aggregation degree to the category set degree, and the classification module is specifically used to: determine the classification probability that the text to be processed belongs to each text set based on the comparison result corresponding to each text set; and determine the text set with the largest classification probability as the target text set.

[0035] Optionally, the text classification device further includes a training module, which is used to obtain the category aggregation degree of each text set in the following manner:

[0036] For each feature vector in each text set, obtain the similarity between each feature vector and other feature vectors in each text set, sort the obtained similarities from large to small, and determine the degree of aggregation between each feature vector and each text set based on the top K similarities;

[0037] Based on the aggregation degree corresponding to each feature vector in each text set, the category aggregation degree corresponding to each text set is determined.

[0038] Optionally, the training module is specifically used to: determine the average value of the top K similarities as the degree of aggregation between each feature vector and each text set.

[0039] Optionally, the training module is specifically used to: determine the average value of the aggregation degrees corresponding to the feature vectors in each text set as the category aggregation degree corresponding to each text set.

[0040] Optionally, each category corresponds to a user intent.

[0041] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0042] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having computer program instructions stored thereon, which implement the steps of any of the above methods when executed by a processor.

[0043] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a processor, the steps of any of the above methods are implemented.

[0044] The text classification method, device, electronic device and storage medium provided by the embodiment of the present application can automatically learn the distribution of text data of the same category in the feature space from the text collection, and obtain the category aggregation degree that characterizes the density of the feature vector distribution in the text collection; when performing text classification, the text data of different categories in the training set are divided into the text collection corresponding to each category, and the K text data that are most similar to the text to be processed are searched in the text collection of each category respectively, and then the aggregation degree of the text to be processed for each category is calculated respectively, and the most suitable category of the text to be processed is determined by comparing the aggregation degree and category aggregation degree of the text to be processed for the category. On the one hand, searching for K nearest neighbor text data and subsequent processing in the text collections of different categories can reduce the influence of the value of K on the classification result, and on the other hand, by comparing the aggregation degree and category aggregation degree of the text to be processed for each category, the problem of inaccurate classification results caused by different data distribution or data imbalance can be improved, and the accuracy of text classification can be improved through the optimization of the above two aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A schematic diagram of text classification based on the KNN algorithm when K=3 provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of text classification based on the KNN algorithm when K=5 provided in an embodiment of the present application;

[0048] Figure 3 An example of a distribution diagram of text data in a feature space provided in an embodiment of the present application;

[0049] Figure 4 A flowchart of a text classification method provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of a process for obtaining the category aggregation degree of a text collection provided in an embodiment of the present application;

[0051] Figure 6 An example of a distribution diagram of text data in a feature space provided in an embodiment of the present application;

[0052] Figure 7 A schematic diagram of the structure of a text classification device provided in an embodiment of the present application;

[0053] Figure 8 It is a schematic diagram of the structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0055] It should be noted that the following embodiments and features in the embodiments may be combined with each other in the absence of conflict; and, based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in the field without making any creative work are within the scope of protection of the present disclosure.

[0056] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein may be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present disclosure, it should be understood by those skilled in the art that an aspect described herein may be implemented independently of any other aspect, and two or more of these aspects may be combined in various ways. For example, any number of aspects described herein may be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein may be used to implement this device and / or practice this method.

[0057] For ease of understanding, the terms involved in the embodiments of the present application are explained below:

[0058] Stop words refer to certain characters or words that are automatically filtered out before or after processing natural language data (or text) in order to save storage space and improve search efficiency in information retrieval. Stop words are manually input and not automatically generated. The generated stop words will form a stop word list.

[0059] Chinese word segmentation is the process of dividing a sequence of Chinese characters into several separate words, that is, recombining continuous character sequences into word sequences according to certain specifications. Commonly used Chinese word segmentation tools include jieba.

[0060] Word2vec is a group of related models used to generate word vectors. These models are shallow, two-layer neural networks that are trained to reconstruct linguistic word texts. The network is represented by words and needs to guess the input words in adjacent positions. Under the word bag model assumption in word2vec, the order of words is not important. After training, the word2vec model can be used to map each word to a vector, which can be used to represent the relationship between words. The vector is the hidden layer of the neural network.

[0061] The basic idea of ​​the KNN algorithm is: when the classification of text data in the training set is known, the feature vector of the text to be processed is compared with the feature vector of the text data in the training set, and the first K text data in the training set that are most similar to the text to be processed are found, and the classification with the most occurrences in these K text data is taken as the category to which the text to be processed belongs. Figure 1 For example, Figure 1 Each point in represents the feature vector of the text data, where the triangular points belong to category 1, the circular points belong to category 2, and the square points represent the feature vector of the text to be processed. Assuming K = 3, the KNN algorithm will find the three points closest to the square points (i.e. Figure 1 The points in the dotted circle in the middle) are used to determine which category the three points belong to. If more of the three points belong to category one, the text to be processed (i.e., the points in the square) is classified into category one.

[0062] However, the classification results of the text classification model based on the KNN algorithm are extremely sensitive to the selection of the K value and the distribution of the text data in the training set.

[0063] In practical applications, different K values ​​may lead to different classification results. Figure 1 When K=3, the KNN algorithm finds the three points closest to the feature vector of the text to be processed (i.e., the points of the square), and then classifies the text to be processed into category 1 (i.e., the category to which the triangle points belong). Figure 2For example, when K=5, the KNN algorithm finds the five points closest to the feature vector of the text to be processed (i.e., the square points), and there are 3 circular points among these five points, so the feature vector of the text to be processed (i.e., the square points) is classified into category two.

[0064] In addition, the KNN algorithm does not consider that the distribution of text data of different categories may be different, or the number and distribution density of data of different categories may be different due to the imbalance of sample numbers. Figure 3 For example, each point represents a text data. The black points belong to category 1, and the white points belong to category 2. The distribution area of ​​black points is narrow and dense, while the distribution area of ​​white points is wide and sparse. For the gray points to be classified, the nearest neighbor points found by KNN are all black points, so the gray points will be classified into category 1. However, the analysis Figure 3 From the distribution of the midpoints, we can find that the black points are close to each other, while the gray points are obviously at a certain distance from the nearest black points. On the contrary, the interval between the gray points and the white points is more in line with the distribution of the white points. Therefore, the gray points should be classified as category 2.

[0065] Therefore, for some application scenarios with fewer samples, it is easy to have an uneven distribution of text data in each category, which seriously reduces the accuracy of text classification.

[0066] Based on this, the present application embodiment provides a text classification method, referring to Figure 4 , specifically including the following steps:

[0067] S401, obtaining a target feature vector of a text to be processed and at least two text sets, wherein each text set corresponds to a category, and each text set includes feature vectors of text data belonging to the same category.

[0068] In specific implementation, text data used or appearing in the application scenario can be obtained, the text data can be divided into multiple categories, and the text data of the same category can be put into the same text set. The categories of text data can be set according to application requirements. For example, taking the intelligent customer service of a shopping website as an example, the text input by the user can be divided into product consultation, price consultation, payment operation consultation, activity consultation, after-sales consultation, etc., which is not limited in the embodiments of this application.

[0069] In specific implementation, the text data in each text set can be preprocessed in advance. The purpose of preprocessing is to remove noise data in the text data. The preprocessing may specifically include: first removing special characters (other characters except Chinese and English) and stop words in the text data, and then performing word segmentation on the text data to obtain a number of word segments. Then, the several word segments are converted into feature vectors, i.e., feature vectors of the text data to be processed, and the feature vectors of each text data are stored in the corresponding text set for easy use in text classification.

[0070] Specifically, the FastText word vector model and SIF weighted combination can be used to convert text into feature vectors, and the formula is: v = ∑α*FastText(w). First, use the FastText word vector model to convert all the word segments in the text data into word vectors, and then multiply each word vector by the SIF weight (the SIF weight calculation formula is: α = a / (a+P(w)), where a is 0.01, and P(w) is the probability of the word segment appearing) and then sum them all up, and finally get the vector representation of the text data. Similarly, the above-mentioned FastText word vector model and SIF weighted combination can be used to convert the text to be processed into a feature vector, that is, to obtain the target feature vector.

[0071] Of course, other methods may also be used to convert text into feature vectors, such as the word2vec model, which is not limited in the embodiments of the present application.

[0072] S402. For each text set, obtain K nearest neighbor feature vectors of the target feature vector from the text set, obtain the aggregation degree between the target feature vector and the text set based on the K nearest neighbor feature vectors and the target feature vector, and obtain a comparison result between the aggregation degree and the category aggregation degree of the text set.

[0073] The aggregation degree represents the density between the target feature vector and the K nearest feature vectors, and this density is used to represent the aggregation degree between the text to be processed and the text set. The category aggregation degree represents the density of the feature vector distribution in the same text set.

[0074] Among them, K nearest feature vectors refer to K feature vectors that are closest to the target feature vector. In specific implementation, obtaining the K nearest feature vectors of the target feature vector from the text set in step S402 specifically includes: obtaining the similarity between each feature vector in the text set and the target feature vector, sorting the similarities from large to small, and determining the top K feature vectors as the K nearest feature vectors of the target feature vector. Alternatively, the kd tree can also be used to search for the K nearest feature vectors, which is a prior art and will not be described in detail.

[0075] In specific implementation, the step S402 obtains the degree of aggregation between the target feature vector and the text set based on the k nearest neighbor feature vectors and the target feature vector, specifically including: obtaining the similarity between each feature vector of the K nearest neighbor feature vectors and the target feature vector, and determining the average value of the similarities corresponding to the K nearest neighbor feature vectors as the degree of aggregation between the target feature vector and the text set.

[0076] Take a text set V of category c c For example, first calculate the target feature vector J and the text set V c The similarity of each feature vector in , where the target feature vector J and the text set V c The i-th eigenvector v in c,i The similarity is denoted as S c,i =Sim(v c,i ,J), all S c,i =Sim(v c,i ,J) is composed of a set S c ; then from S c Take the most similar K feature vectors, and obtain the set of k nearest feature vectors as k s =Max(S c ,k); finally, the average similarity of the k nearest feature vectors is calculated as the target feature vector J and the text set V c The degree of polymerization h J,c =Mean(k s ).

[0077] S403: Determine a target text set from at least two text sets based on the comparison result corresponding to each text set.

[0078] In specific implementation, the ratio of the aggregation degree to the category set degree can be used as the comparison result. To this end, based on the comparison result corresponding to each text set, the classification probability of the processed text belonging to each text set can be determined, and the text set with the largest classification probability can be determined as the target text set.

[0079] For example, the text set V c The category aggregation degree is a c , target feature vector J and text set V c The degree of polymerization between c , then the degree of polymerization h c and category aggregation degree a c The comparison result is h c / a c Then, based on the Softmax function P = Softmax (h c / a c), obtain the classification probability that the target feature vector J belongs to the category c, and determine the text set with the largest classification probability as the target text set.

[0080] In specific implementation, h c / a c The closer it is to 1, the more likely it is that the target feature vector J is in the text set V. c The closer the distribution in the text set V is, c Therefore, the comparison results h corresponding to each text set can also be compared. c / a c , select the h closest to 1 c / a c , the h c / a c The corresponding text set is determined as the target text set.

[0081] The target feature vector J and the text set V c The degree of aggregation between them is divided by the text set V c The category set degree is equivalent to the target feature vector J and the text set V c The absolute quantity of the degree of aggregation between the two is relatively converted, and classification is performed based on the ratio of the two. This fully considers the distribution of text data in each category and helps to improve the accuracy of text classification.

[0082] S404: Determine the category of the target text set as the category of the text to be processed.

[0083] When implementing it, refer to Figure 5 , the category aggregation degree of each text collection can be obtained as follows:

[0084] S501. For each feature vector in the text set, obtain the similarity between the feature vector and other feature vectors in the text set, sort the obtained similarities from large to small, and determine the degree of aggregation between the feature vector and the text set based on the top K similarities.

[0085] Specifically, the average value of the top K similarities can be determined as the degree of aggregation between the feature vector and the text set, or the median value of the top K similarities can be taken as the degree of aggregation between the feature vector and the text set.

[0086] S502: Determine the category aggregation degree corresponding to the text set based on the aggregation degree corresponding to each feature vector in the text set.

[0087] Specifically, the average value of the aggregation degree corresponding to the feature vectors in the text set can be determined as the category aggregation degree corresponding to the text set. Alternatively, the median value of the aggregation degree corresponding to the feature vectors in the text set can be taken as the aggregation degree between the feature vector and the text set.

[0088] For example, assuming K = 3, the text set V c There are 20 feature vectors of text data in the text set V. c Take one of the feature vectors as an example, calculate the similarity between the feature vector and the other 19 feature vectors, take the 3 largest similarities, and take the average of the 3 similarities as the aggregation degree between the feature vector and the text set. The above method can obtain the aggregation degree corresponding to each of the 20 feature vectors, and then take the average of the aggregation degree corresponding to each of the 20 feature vectors as the category aggregation degree corresponding to the text set.

[0089] by Figure 6 For example, each point represents a text data, black points belong to category 1, white points belong to category 2, and the distribution of text data of the same category in the feature space is similar. They are aggregated together to form a category area. The distribution area of ​​black points is narrow and dense, while the distribution area of ​​white points is wide and sparse. Take K = 3 as an example, refer to Figure 5 The steps shown in the figure are used to calculate the class aggregation degree of class one and class two. Figure 6 The line segments in the figure show that the line segment length is negatively correlated with the category aggregation degree, that is, the longer the line segment, the smaller the category aggregation degree, indicating that the text data of this category is more sparsely distributed. For example, the black points are densely distributed, and the category aggregation degree is high, while the white points are sparsely distributed, and the category aggregation degree is low. Then, refer to Figure 4 In the steps shown, the aggregation degree h1 of the gray point and category one and the aggregation degree h2 of the gray point and category two are calculated respectively, and then the aggregation degree h1 is compared with the analog aggregation degree of category one, and the aggregation degree h2 is compared with the category aggregation degree of category two. Based on the comparison results of the two, it is found that the aggregation degree h2 is closer to the category aggregation degree of category two, that is, the gray point is more consistent with the distribution of the white points, so the gray point is assigned to category two.

[0090] The text classification method of the embodiment of the present application can automatically learn the distribution of text data of the same category in the feature space from the text collection, and obtain the category aggregation degree that characterizes the density of the feature vector distribution in the text collection; when performing text classification, the text data of different categories in the training set are divided into the text collection corresponding to each category, and the K text data that are most similar to the text to be processed are searched in the text collection of each category, and then the aggregation degree of the text to be processed for each category is calculated respectively, and the most suitable category of the text to be processed is determined by comparing the aggregation degree and category aggregation degree of the text to be processed for the category. On the one hand, searching for K nearest neighbor text data and subsequent processing in text collections of different categories can reduce the influence of the value of K on the classification result, and on the other hand, by comparing the aggregation degree and category aggregation degree of the text to be processed for each category, the problem of inaccurate classification results caused by different data distribution or data imbalance can be improved, and the accuracy of text classification can be improved through the optimization of the above two aspects.

[0091] The text classification method of the embodiment of the present application can be applied in application scenarios such as intelligent customer service, news recommendation, and intent recognition. Based on the text data in these application scenarios, a text classification model with high classification accuracy can be quickly constructed, and text classification can also be performed well for scenarios with fewer samples.

[0092] The following uses banking services as an example to illustrate the application of the classification method of the embodiment of the present application in intent recognition.

[0093] In intent recognition, each category in the above classification method corresponds to a user intent, and the text set of each corresponding category includes text data indicating the same intent.

[0094] For example, if the specific business is the unmanned banking business of a bank, it is necessary to identify the user's intention based on a sentence input by the user through voice or text, and then provide the user with corresponding services or guide the user to perform corresponding operations. At this time, the intent category can be determined according to the application requirements, for example, it can include the six categories of transfer, deposit, withdrawal, confirmation, modification, and cancellation, and then collect the text data corresponding to various intents and store them in the corresponding text collections respectively. Among them, the text data of the three categories of transfer, deposit, and withdrawal are generally long sentences, with a large amount of data and dense and sparse distribution, while the text data of the three categories of confirmation, modification, and cancellation are generally long phrases or words, with less data and dense distribution.

[0095] First, training is performed based on the text sets corresponding to various intents to obtain the category aggregation degree corresponding to each intent.

[0096] The first step is to preprocess the text data in the text collection.

[0097] The purpose of preprocessing is to remove noise data in text data, wherein the preprocessing may specifically include: first removing special characters (other characters except Chinese and English) and stop words in the text data, and then performing word segmentation on the text data to obtain a number of word segments.

[0098] The second step is to convert the preprocessed text data into feature vectors.

[0099] The word segmentation of the text data is converted into a feature vector, that is, the feature vector of the text data to be processed, and the feature vector of each text data is stored in the corresponding text set for easy use in text classification.

[0100] The third step is to calculate the category aggregation degree.

[0101] Calculate the aggregation degree of each text data in the text collection for this type of intent. Take a category c as an example, V c Represents a text collection of category c, reference Figure 5 The text set V is calculated by c Each text data in the text set V c Then calculate the average of these aggregation degrees as the category aggregation degree of category c. Through the above method, the category aggregation degrees of all categories are obtained.

[0102] Next, based on the category aggregation degree obtained in the training phase, the intent of the text to be processed input by the user is identified.

[0103] The first step is to preprocess the text to be processed input by the user.

[0104] The preprocessing method refers to the preprocessing method in the training phase.

[0105] The second step is to convert the preprocessed text into a feature vector.

[0106] The same method as in the training phase is used to vectorize the text to be processed and generate a vector v of the text to be processed.

[0107] The third step is to calculate the aggregation degree of the text to be processed for each intent category.

[0108] Specific reference Figure 4 Step S403 in .

[0109] Step 4: Determine the intent recognition results.

[0110] Based on the Softmax function P = Softmax (h c / a c), obtain the classification probability of the text to be processed belonging to each intent category, and determine the text set with the largest classification probability as the target text set. The intent recognition result returned by the Softmax function can be expressed as: {"class_0":p0, "class_1":p1,...}, where class_0 and class_1 represent intent categories, and p0 and p1 represent classification probabilities.

[0111] For example, the text input by the user is: "I want to transfer money", and after data preprocessing, the word segmentation result is ["I", "want", "want", "transfer"]. The word segmentation result is vectorized to obtain the target feature vector v. Calculate the aggregation degree of all categories to obtain the category aggregation degree vector [a 0 ,a 1 ,a 2 ,a 3 ,a 4 ,a 5 ], and the corresponding intent categories are ["transfer", "deposit", "draw", "cancel", "modify", "confirm"]. Comparing the category aggregation degrees, we get the classification probability vector [0.91, 0.04, 0.02, 0.01, 0.01, 0.01], and the final intent recognition result is: {"transfer": 0.91, "deposit": 0.04, "draw": 0.02, "cancel": 0.01, "modify": 0.01, "confirm": 0.01}, which means that the user's intention is "transfer".

[0112] like Figure 7 As shown, based on the same inventive concept as the above-mentioned text classification method, the embodiment of the present application further provides a text classification device 70, which specifically includes:

[0113] An acquisition module 701 is used to obtain a target feature vector of a text to be processed and at least two text sets, wherein each text set corresponds to a category, and each text set includes feature vectors of text data belonging to the same category;

[0114] The aggregation degree calculation module 702 is used to obtain, for each text set, the K nearest neighbor feature vectors of the target feature vector from the text set, obtain the aggregation degree between the target feature vector and the text set based on the K nearest neighbor feature vectors and the target feature vector, and obtain the comparison result of the aggregation degree and the category aggregation degree of the text set; wherein the aggregation degree represents the density between the target feature vector and the K nearest neighbor feature vectors, and the category aggregation degree represents the density of the feature vector distribution in the same text set;

[0115] The classification module 703 is used to determine a target text set from the at least two text sets based on the comparison result corresponding to each text set, and determine the category of the target text set as the category of the text to be processed.

[0116] Optionally, the aggregation degree calculation module 702 is specifically used to: obtain the similarity between each feature vector in each text set and the target feature vector; sort the feature vectors with the highest similarity from large to small, and determine the top K feature vectors as the K nearest neighbor feature vectors of the target feature vector.

[0117] Optionally, the aggregation degree calculation module 702 is specifically used to: obtain the similarity between each feature vector of the K nearest neighbor feature vectors and the target feature vector; and determine the average value of the similarities corresponding to the K nearest neighbor feature vectors as the aggregation degree between the target feature vector and each text set.

[0118] Optionally, the comparison result is the ratio of the aggregation degree to the category set degree, and the classification module 703 is specifically used to: determine the classification probability that the text to be processed belongs to each text set based on the comparison result corresponding to each text set; and determine the text set with the largest classification probability as the target text set.

[0119] Optionally, the text classification device 70 further includes a training module, which is used to obtain the category aggregation degree of each text set in the following manner:

[0120] For each feature vector in each text set, obtain the similarity between each feature vector and other feature vectors in each text set, sort the obtained similarities from large to small, and determine the degree of aggregation between each feature vector and each text set based on the top K similarities;

[0121] Based on the aggregation degree corresponding to each feature vector in each text set, the category aggregation degree corresponding to each text set is determined.

[0122] Optionally, the training module is specifically used to: determine the average value of the top K similarities as the degree of aggregation between each feature vector and each text set.

[0123] Optionally, the training module is specifically used to: determine the average value of the aggregation degrees corresponding to the feature vectors in each text set as the category aggregation degree corresponding to each text set.

[0124] Optionally, each category corresponds to a user intent.

[0125] The text classification device proposed in the embodiment of the present application adopts the same inventive concept as the above-mentioned text classification method and can achieve the same beneficial effects, which will not be repeated here.

[0126] Based on the same inventive concept as the above-mentioned text classification method, the embodiment of the present application also provides an electronic device, which can be a desktop computer, a portable computer, a smart phone, a tablet computer, a personal digital assistant (PDA), a server, etc. Figure 8 As shown, the electronic device 80 may include a processor 801 and a memory 802 .

[0127] Processor 801 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the text classification method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0128] Memory 802, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules. Memory can include at least one type of storage medium, for example, can include flash memory, hard disk, multimedia card, card-type memory, random access memory (Random Access Memory, RAM), static random access memory (Static Random Access Memory, SRAM), programmable read-only memory (Programmable Read Only Memory, PROM), read-only memory (Read Only Memory, ROM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), magnetic memory, disk, optical disk, etc. Memory is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. Memory 902 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0129] An embodiment of the present application provides a computer-readable storage medium for storing computer program instructions used by the above electronic device, which includes a program for executing the above text classification method.

[0130] The above-mentioned computer storage media can be any available media or data storage devices that can be accessed by a computer, including but not limited to magnetic storage (such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc.), optical storage (such as CD, DVD, BD, HVD, etc.), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)), etc.

[0131] The above embodiments are only used to introduce the technical solutions of the present application in detail, but the description of the above embodiments is only used to help understand the methods of the embodiments of the present application and should not be understood as limiting the embodiments of the present application. Any changes or substitutions that can be easily thought of by technicians in this technical field should be included in the protection scope of the embodiments of the present application.

Claims

1. A text classification method, characterized in that: include: Obtaining a target feature vector of a text to be processed and at least two text sets, wherein each text set corresponds to a category, and each text set includes feature vectors of text data belonging to the same category; For each text set, K nearest neighbor feature vectors of the target feature vector are obtained from each text set, and based on the K nearest neighbor feature vectors and the target feature vector, a degree of aggregation between the target feature vector and each text set is obtained, and a comparison result of the degree of aggregation and the category degree of aggregation of each text set is obtained; wherein the degree of aggregation represents the density between the target feature vector and the K nearest neighbor feature vectors, and the category degree of aggregation represents the density of feature vector distribution in the same text set; Determining a target text set from the at least two text sets based on the comparison results corresponding to each text set; Determining the category of the target text set as the category of the text to be processed; The step of determining a target text set from the at least two text sets based on the comparison result corresponding to each text set specifically includes: Based on the comparison result corresponding to each text set, determine the classification probability that the text to be processed belongs to each text set; wherein the comparison result is / , is the aggregation degree, and is the category aggregation degree; The text set with the highest classification probability is determined as the target text set.

2. The method according to claim 1, characterized in that: The step of obtaining K nearest neighbor feature vectors of the target feature vector from each text set specifically includes: Obtaining the similarity between each feature vector in each text set and the target feature vector; According to the order from large to small similarity, the top K feature vectors are determined as the K nearest neighbor feature vectors of the target feature vector.

3. The method according to claim 1, characterized in that The step of obtaining the degree of aggregation between the target feature vector and each text set based on the k nearest neighbor feature vectors and the target feature vector specifically includes: Obtaining the similarity between each feature vector of the K nearest neighbor feature vectors and the target feature vector; The average value of the similarities corresponding to the K nearest feature vectors is determined as the degree of aggregation between the target feature vector and each text set.

4. The method according to any one of claims 1 to 3, characterized in that: The category aggregation degree of each text collection is obtained by: For each feature vector in each text set, obtain the similarity between each feature vector and other feature vectors in each text set, sort the obtained similarities from large to small, and determine the degree of aggregation between each feature vector and each text set based on the top K similarities; Based on the aggregation degree corresponding to each feature vector in each text set, the category aggregation degree corresponding to each text set is determined.

5. The method according to claim 4, characterized in that The step of determining the degree of aggregation between each feature vector and each text set based on the top K similarities in the ranking specifically includes: The average value of the top K similarities is determined as the degree of aggregation between each feature vector and each text set.

6. The method according to claim 4, characterized in that Determining the category aggregation degree corresponding to each text set based on the aggregation degree corresponding to each feature vector in each text set specifically includes: The average value of the aggregation degrees corresponding to the feature vectors in each text set is determined as the category aggregation degree corresponding to each text set.

7. A text classification device, characterized in that: An acquisition module, used for obtaining a target feature vector of a text to be processed and at least two text sets, wherein each text set corresponds to a category, and each text set includes feature vectors of text data belonging to the same category; A degree of aggregation calculation module is used to obtain, for each text set, K nearest neighbor feature vectors of the target feature vector from each text set, obtain the degree of aggregation between the target feature vector and each text set based on the K nearest neighbor feature vectors and the target feature vector, and obtain a comparison result between the degree of aggregation and the category degree of aggregation of each text set; wherein the degree of aggregation represents the density between the target feature vector and the K nearest neighbor feature vectors, and the category degree of aggregation represents the density of feature vector distribution in the same text set; A classification module, configured to determine a target text set from the at least two text sets based on the comparison result corresponding to each text set, and determine the category of the target text set as the category of the text to be processed; The classification module is specifically used to: determine the classification probability that the text to be processed belongs to each text set based on the comparison result corresponding to each text set; wherein the comparison result is / , is the aggregation degree, and is the category aggregation degree; and determine the text set with the largest classification probability as the target text set.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images