Text classification method and device, electronic equipment and computer readable storage medium

Abstract

The application provides a text classification method and device, electronic equipment and a computer readable storage medium. The method comprises: clustering each text data in a text data set according to each category number in a plurality of pre-set category numbers, to obtain a category to which each text data belongs under different category numbers; wherein each text data corresponds to one belonging category under one category number; dividing the plurality of text data into a plurality of groups of text data based on the category to which each text data belongs under different category numbers, and the same group of text data belongs to the same category under each category number; and determining a category division result of each group of text data according to the category to which each group of text data belongs under different category numbers. According to the embodiments of the application, the accuracy of the text classification result can be improved.

Description

Technical Field

[0001] This application relates to the field of text classification technology, and in particular to a text classification method, apparatus, electronic device and computer-readable storage medium. Background Technology

[0002] Clustering is the process of dividing a dataset into different clusters (groups) according to specific rules. Clustering aims to maximize the similarity of data within the same cluster and minimize the similarity between different clusters. The results of clustering can be used for data analysis and data mining in relevant fields. Clustering can be applied to various technical fields. For example, in business, clustering can help market analysts discover different customer groups from a database of basic customer information; in biology, clustering can be used to deduce the classification of plants and animals, classify genes, and gain insights into the inherent structure of populations; in geographic information systems, clustering can be used to identify similar regions in Earth observation databases; and with the expansion of clustering applications, it is also used for text classification, thus requiring improved accuracy of text classification results. Summary of the Invention

[0003] This application provides a text classification method, apparatus, electronic device, and computer-readable storage medium that can improve the accuracy of text classification results.

[0004] Firstly, this application provides a text classification method, which includes: clustering multiple text data in a text data set according to each of a pre-set number of categories to obtain the category to which each text data belongs under different numbers of categories; wherein, each text data corresponds to one category under one number of categories; based on the category to which each text data belongs under different numbers of categories, dividing the multiple text data into multiple groups of text data, with the same group of text data belonging to the same category under each number of categories; and determining the category division result of each group of text data according to the category to which each group of text data belongs under different numbers of categories.

[0005] Secondly, this application provides a text classification device, which includes: a clustering module, used to cluster multiple text data in a text data set according to each of a pre-set number of categories, to obtain the category to which each text data belongs under different number of categories; wherein each text data corresponds to one category under one number of categories; a partitioning module, used to partition the multiple text data into multiple groups of text data based on the category to which each text data belongs under different number of categories, wherein the text data in the same group belongs to the same category under each number of categories; and a determination module, used to determine the category partitioning result of each group of text data according to the category to which each group of text data belongs under different number of categories.

[0006] Thirdly, this application provides an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the above-described text classification method.

[0007] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-described text classification method when executed by a processor / processing core.

[0008] The embodiments provided in this application, when classifying multiple text data in a text data set, can first perform multiple clustering operations according to a pre-set number of clusters to obtain multiple clustering results; then, based on the multiple clustering results, at least one set of related text data is determined, and the multiple text data in each set of related text data belong to the same category in each clustering result; thus, each set of related text data can be classified according to the category to which each set of related text data belongs in each clustering result.

[0009] In the method of this application embodiment, multiple text data in the text data set are clustered according to each of the pre-set multiple number of categories to obtain the category to which each text data belongs under different number of categories. Each text data corresponds to a category under one number of categories. Then, the text data is grouped according to the category to which each text data belongs under different number of categories. The text data in the same group belongs to the same category under each number of categories. According to this text classification method, multiple text data are first classified by different number of categories (there are multiple number of categories to indicate how many categories there are). Each group of text data that is subsequently selected belongs to the same category under the corresponding number of categories, regardless of which number of categories is used for clustering. The category division result obtained in this way can ensure that multiple groups of text data in the text data set are of the same type of text data, regardless of how many categories they are divided into, thereby improving the reliability of the category division result obtained by text classification and improving the accuracy of the text classification result.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0011] The accompanying drawings are provided to further illustrate the present application and form part of the specification. They are used together with the embodiments of the present application to explain the application and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed example embodiments described with reference to the accompanying drawings, in which:

[0012] Figure 1 A flowchart illustrating a text classification method provided in this application embodiment;

[0013] Figure 2 A flowchart of a text classification method provided for an exemplary embodiment of this application.

[0014] Figure 3 A block diagram of a text classification device provided in an embodiment of this application;

[0015] Figure 4 This is a block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0016] To enable those skilled in the art to better understand the technical solutions of this application, exemplary embodiments of this application are described below in conjunction with the accompanying drawings, including various details of the embodiments of this application to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0017] Where there is no conflict, the various embodiments of this application and the features thereof may be combined with each other.

[0018] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0019] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, the presence of the stated feature, integral, step, operation, element, and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof is not excluded. Terms such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0020] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0021] Clustering is a technique for finding inherent structures among data. It organizes sample data in a dataset into similar groups, called clusters. Data within the same cluster are highly similar, while data in different clusters are highly dissimilar. In terms of data, individuals with similar characteristics are more likely to cluster together, and vice versa.

[0022] Clustering has applications in a variety of technical fields. For example: in business, clustering helps market analysts identify different customer groups from a database of basic customer data, allowing them to tailor different purchasing patterns for each group; in biology, clustering can be used to deduce the classification of plants and animals, classify genes, and gain insights into the inherent structure of populations; in geographic information systems, clustering can identify similar regions in Earth observation databases; in real estate, clustering can be used to group houses in a city based on type, value, and geographical location; and in document management, clustering can be used to classify documents and is an important tool for mining textual information, significantly contributing to simplifying textual data and accelerating text retrieval.

[0023] K-means clustering is a classic partition-based clustering method. The execution process of K-means clustering may include the following steps: S01, randomly select k elements from the dataset as the initial cluster centers (also called cluster centers or centroids) for each of the k clusters; S02, calculate the distances from all other elements in the dataset (excluding the k elements) to the k cluster centers, using the distance as the basis to measure the similarity between the other elements and the k cluster centers, and cluster these elements into the clusters with the highest similarity (the smaller the distance, the more similar); S03, based on the clustering results of these elements, recalculate the cluster centers of the k clusters. For example, for each cluster, calculate the mean of all elements in the cluster as the new cluster center; S04, re-cluster all elements in the dataset according to the new cluster centers; S05, iterate through steps S03 and S04 until the execution process converges. The meaning of convergence is that the output conditions of the clustering results can be reached through a finite number of iterations (e.g., the clustering results no longer change); S06, output the clustering results.

[0024] The execution process of K-means clustering shows that the first step is to initialize k cluster centers. Then, new cluster centers need to be determined based on the results of each clustering, so that the similarity of data in the same cluster is as high as possible, and the similarity between different clusters is as low as possible.

[0025] However, K-means clustering has the following problems: It requires a pre-defined value for k, which is difficult to determine without sufficient knowledge of the data; it is sensitive to the initial cluster centers, as different random cluster centers yield completely different results for the same dataset, significantly impacting the overall outcome; it is also sensitive to noise, with even a small amount of noise greatly affecting the mean; and its iterative approach may only yield a local optimum, not a global optimum. Therefore, the convergence of K-means clustering largely depends on the initialization of the cluster centers. Inappropriate initial cluster center selection can lead to errors in the clustering results, significantly affecting their accuracy.

[0026] In related technologies, K-means++ improves the method of initializing cluster centers. K-means++ is an algorithm for selecting initial cluster centers for the k-means clustering algorithm. The execution process of k-means++ can include the following steps: First, randomly (uniformly distributed) a sample point (sample data) from the dataset as the first initial cluster center; Second, calculate the shortest distance between each sample and the existing cluster centers, then calculate the probability of each sample point being selected as the next cluster center, and finally select the sample point corresponding to the highest probability value as the next cluster center. It can be seen that the farther the sample point is from the existing cluster center, the more likely it is to be selected as the next cluster center; Third, repeat the second step until k cluster centers are selected; Fourth, calculate and compare the distances of the remaining sample points to the k cluster centers, and assign them to different cluster centers accordingly, completing the clustering. This method performs poorly in cases of imbalanced data; and due to the inherent orderliness in the cluster center selection process, the selection of the kth cluster center depends on the values ​​of the first k-1 cluster centers, resulting in relatively high time and space complexity in the computation process.

[0027] The text classification method according to embodiments of this application can be executed by electronic devices such as terminal devices or servers. Terminal devices can be in-vehicle devices, user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistants (PDAs), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. This text classification method can be implemented by a processor calling computer-readable program instructions stored in memory. Servers can include independent physical servers, server clusters consisting of multiple servers, or cloud servers capable of cloud computing.

[0028] Figure 1A flowchart illustrating a text classification method provided in an embodiment of this application. (Refer to...) Figure 1 The text classification method may include the following steps.

[0029] S110, For multiple text data in the text data set, cluster them according to each of the pre-set number of categories to obtain the category to which each text data belongs under different number of categories; wherein, each text data corresponds to one category under one number of categories.

[0030] In some embodiments, the text data set can be a collection of text data from different application scenarios. For example, the text data may include: text summaries (e.g., news text summaries, legal text summaries, etc.), dialogue text from historical call data, ancient poems, and modern poems; the text data set can also be, for example, a large-scale plain text dataset (e.g., encyclopedia, news, Q&A, etc.). It should be understood that the embodiments of this application do not specifically limit the application scenarios and specific content of the text data set.

[0031] In step S110, the number of categories is used to characterize the number of cluster centers when clustering multiple text data in the text dataset. Each category corresponds to one clustering operation. For example, the number of categories (also called the number of clusters or the number of cluster centers) is represented by the value k. Multiple k values ​​can be, for example, 20, 25, and 30. When k = 20, the value k is 20, indicating that multiple text data in the text dataset are classified under 20 cluster centers, and each text data belongs to one of the 20 categories. When k = 25, the value k is 25, indicating that multiple text data in the text dataset are classified under 25 cluster centers, and each text data belongs to one of the 25 categories. When k = 30, the value k is 30, indicating that multiple text data in the text dataset are classified under 30 cluster centers, and each text data belongs to one of the 30 categories.

[0032] S120, based on the category to which each text data belongs under different number of categories, divide multiple text data into multiple groups of text data, with the same group of text data belonging to the same category under each number of categories.

[0033] As an example, suppose the text dataset contains multiple text data including Test1, Test2, ..., TestM. After the above steps, the category to which each text data belongs under different number of categories is obtained. Based on the category to which each text data belongs under different number of categories, Test1, Test2, ..., TestM are grouped to obtain multiple groups of text data. In the multiple groups of text data, the text data in the same group belongs to the same category under each number of categories.

[0034] For example, if a set of text data includes three texts: Test1, Test2, and Test3, and assume multiple values ​​of k: 20, 25, and 30; when k is 20, under category number 20 ([A1,A2,...,A20]), Test1, Test2, and Test3 all belong to category A1; when k is 25, under category number 25 ([B1,B2,...,B25]), Test1, Test2, and Test3 all belong to category B5; when k is 30, under category number 30 ([C1,C2,...,C30]), Test1, Test2, and Test3 all belong to category C6. Through this step, multiple text data groups can be obtained from the multiple text data in the text dataset. Regardless of how many categories the multiple text data are divided into, the same set of text data belongs to the same category under each category number.

[0035] S130, determine the category classification result of each group of text data based on the category to which each group of text data belongs under different number of categories.

[0036] As an example, for the text data set Test1, Test2, and Test3, this set of text data belongs to the same category (A1) with a category count of 20, to the same category (B5) with a category count of 25, and to the same category (C6) with a category count of 30. Therefore, the classification result for this set of text data is that it belongs to the same category. In other words, the text data set Test1, Test2, and Test3 can be classified into the category corresponding to the category name determined by categories A1, B5, and C6.

[0037] According to the text classification method of this application embodiment, multiple text data in a text data set are clustered according to each of a pre-set number of categories, to obtain the category to which each text data belongs under different numbers of categories. Each text data corresponds to a category under one number of categories. Then, the text data is grouped according to the category to which each text data belongs under different numbers of categories. The text data in the same group belongs to the same category under each number of categories. According to this text classification method, multiple text data are first classified by different numbers of categories (to indicate how many categories there are). Each group of text data selected subsequently belongs to the same category under the corresponding number of categories, regardless of which number of categories is used for clustering. The category division result obtained in this way can ensure that multiple groups of text data in the text data set are of the same type, regardless of how many categories they are divided into, thereby improving the reliability of the category division result obtained by text classification and improving the accuracy of the text classification result.

[0038] In some embodiments, before clustering according to each of the preset multiple number of categories in step S110 to obtain the category to which each text data belongs under different number of categories, the text classification method may further include: S11, setting a category base value for multiple number of categories; S12, incrementing and / or decrementing the category base value by a predetermined number of times with a predetermined step size to obtain multiple different number of categories; S13, using the category base value and the multiple different number of categories as the preset multiple number of categories.

[0039] In step S11, the category base value is a set reference value for the number of categories. The number of categories is expanded based on the category base value to obtain multiple categories expanded around the category base value. For example, the category base value can be the average of multiple categories.

[0040] In some embodiments, the number of multiple categories can be expressed as the following formula (1):

[0041] k∈∑ i＝0 n±d*i (1)

[0042] In the above formula (1), k represents the number of categories, n is the category base value of the number of categories, d represents the step size between different k values, and for example, the value range of d is [2,5]; i represents the predetermined number of increments and / or decrements with a step size of d. For example, i can be the number of step sizes between the maximum and minimum k values ​​and the category base value of k. For example, the value range of i is [3,5]. Assuming n = 40, d = 5, i = 4, then the value of k is [20,25,30,35,40,45,50,55,60]. Among them, the category base value is 40, and the values ​​of k 45, 50, 55 and 60 are the category addition values ​​obtained by each increment when the category base value of 40 is increased by 5; the values ​​of k 35, 30, 25 and 20 are the category addition values ​​obtained by each decrement when the category base value of 40 is decreased by 5.

[0043] In this embodiment, by using the class base value as a basis and incrementing / decrementing by a predetermined step size, a large number of usable classes can be obtained. This helps to solve the problem of difficulty in determining the k value in traditional methods and provides data preparation for clustering according to the number of classes.

[0044] In some embodiments, the category base value is an empirical value determined based on the predetermined number of basic categories of the text data set in the business scenario; or, the category base value is the number of categories obtained by manually classifying the extracted text data after randomly extracting data from the text data set without replacement.

[0045] In this embodiment, the base value for the number of clusters can be determined based on prior knowledge of the data. For example, if the business scenario from which the data comes is dialogue mining, and assuming there are n categories of basic dialogue in this scenario, then the base value of k is set to n. In other scenarios, assuming the total number of texts in the text dataset is T, where T is an integer greater than or equal to 1; if T is less than a preset first value (the total number of texts is relatively small), then the base value of k can be set to n. The calculation result can be rounded up or down, or the base value of k can be set to the calculation result of log(T) and rounded up or down. If the number of text data entries T in the text data set is greater than or equal to this first value (the total number of texts is large), then the base value of k can be set to the first value. The first value can be any value set by the user; there are no restrictions here.

[0046] Optionally, random, non-replacement data can be extracted from multiple text data points in the dataset, and the extracted data can be manually classified. The resulting number of categories can then be set as the category base value. In this embodiment, the selection of the category base value is not randomly specified by the user, but rather based on prior knowledge or the results of manual classification, providing a reliable basis for setting the number of multiple categories.

[0047] In some embodiments, step S110 may specifically include: S21, performing feature mapping on each piece of text data in the text data set; S22, obtaining a set of text feature vectors based on the feature vectors of each piece of text data obtained from the feature mapping; S23, classifying each text feature vector in the set of text feature vectors according to the number of each category, and obtaining the category to which each text feature vector belongs under different number of categories; S24, taking the category to which each text feature vector belongs under different number of categories as the category to which each piece of text data belongs under different number of categories.

[0048] In step S21, text data can be feature-mapped using text feature algorithms. Feature mapping, also known as text representation or encoding representation, can be implemented using any of the following methods: deep text representation (Word to Vector, word2vec) model, general semantic representation model (Bidirectional Encoder Representations from Transformers, BERT), and Simple Contrastive Learning of Sentence Embeddings (SimCSE) technique. Specifically, the BERT model is a bidirectional Transformer encoder that takes a natural language sentence as input and, after training, obtains the encoded representation of each component unit in the sentence; the word2vec model maps all words in the corpus to a high-dimensional space and obtains the high-dimensional space quantity corresponding to each word; SimCSE is a method for sentence embedding using a contrastive learning framework. The SimCSE algorithm is used for unsupervised training of the language model, thereby enhancing the model's semantic representation ability and generalization ability.

[0049] In step S23, the K-means or K-means++ algorithm can be used to classify multiple text feature vectors in the text feature vector set multiple times according to multiple clustering numbers, so as to obtain multiple classification results.

[0050] In this embodiment, multiple text data in a text dataset can be classified according to their inherent similarity. The feature vectors of the text data are used to characterize the inherent features of the text data. Therefore, based on the feature vectors of each text data obtained from feature mapping, and according to the number of each of the multiple categories, multiple text feature vectors in the text feature vector set can be classified. The category to which each text feature vector belongs under different categories is taken as the category to which the corresponding text data belongs under different categories, providing a data foundation for subsequent grouping of multiple text data in the text dataset based on the category to which each text data belongs under different categories.

[0051] In some embodiments, step S120 may specifically include:

[0052] S31, obtain the first text data from the text data set as the current text data, and assign the current text data to the category under each category number as the current category. The first text data is any text data in the text data set.

[0053] As an example, suppose the text data obtained from the text data set is Text1, that is, the current text data is Text1, and the current categories are the categories to which Text1 belongs under different number of categories.

[0054] S32, among the multiple text data included in each category, obtain the text data commonly included in each category as the first set of text data.

[0055] As an example, for the current text data Text1, when the number of categories is 20, among the 20 categories [A1, A2, ..., A20], there is exactly one category A1 that contains Text1, so we can obtain multiple text data included in category A1; when the number of categories is 25, among the 25 categories [B1, B2, ..., B25], there is exactly one category B5 that contains Text1, so we can obtain multiple text data included in category B5; ...; and so on, we can obtain multiple text data included in each of the current categories to which Text1 belongs; by taking the intersection of the multiple text data included in each of the current categories, the text data in the intersection is taken as the text data commonly included in each category, that is, the first set of text data.

[0056] S33, obtain the second text data from the text data set as the new current text data. The second text data is any text data after removing the first group of grouped text data from the text data set.

[0057] S34, the new current text data is assigned to the category under each category number as the new current category.

[0058] S35, in the multiple text data included in the new current categories, obtain the text data that is commonly included in each category as the second group of text data, until each text data in the text data set is grouped, resulting in multiple groups of divided text data.

[0059] In this embodiment, multiple text data in the text dataset are traversed sequentially to find text data belonging to the same category under different number of categories, regardless of which number of categories is used for classification. The resulting category division result can ensure that multiple groups of text data in the text dataset are of the same type, regardless of which number of categories is used for clustering. This can improve the reliability of the category division result and the accuracy of the text classification result.

[0060] In some embodiments, step S130 may specifically include: S41, assigning the category to which the i-th group of text data belongs under different number of categories as each category of the i-th group of text data, where i is an integer from 1 to n, and n is the total number of groups into which the multiple text data are divided; S42, setting a category name corresponding to each category of the i-th group of text data; S43, counting the number of texts in the i-th group of text data; S44, if the number of texts is greater than a predetermined threshold, classifying the i-th group of text data into the text category corresponding to the category name, as the category classification result of the i-th group of text data.

[0061] After step S43, the method further includes: S45, where if the number of texts is less than or equal to a predetermined threshold, the i-th group of text data is treated as noise text data.

[0062] In steps S44 and S45, the predetermined quantity threshold is a value obtained by calculating the ratio of the total number of text data in the text dataset to the category base value of the preset category number. For example, if the total number of texts in the text dataset is T and the preset category base value is t, then the predetermined quantity threshold is the ratio of T to t, i.e., T / t. Optionally, the predetermined quantity threshold can also be any value set by the user, without any limitation here.

[0063] In this embodiment, the multiple text data in the text dataset for clustering are unordered or unlabeled. The purpose of text classification is to find data of the same category while ensuring the highest possible accuracy. Therefore, it is not necessary to retain every single text data. The multiple text datasets are filtered by determining whether the number of text data in each group is greater than a predetermined threshold. T / t is the average number of text data, representing the average number of text data in each category under the category base value. In this embodiment, each retained text data group corresponds to a category name, and each category name corresponds to a category in the category division result. Text data groups with too few text data (less than or equal to T / t) are deleted, and only text data groups that meet the requirements (greater than T / t) are retained. That is, the final text classification result obtained from multiple text data in the text dataset is multiple groups of text data, each group of text data including multiple text data with a number greater than the preset threshold. These multiple groups of text data may include every text data in the text dataset, or only some of the text data. As can be seen, this example can remove noisy text data (i.e., text data with low similarity to other text data in the text data set) while maintaining a certain number of categories. This improves the reliability and accuracy of the category classification results, reduces the amount of data processing required for later use of the text data, and improves data processing efficiency.

[0064] According to the embodiments of this application, multiple text data are classified using different numbers of categories (there are multiple numbers of categories to indicate how many classes there are). Each group of text data subsequently selected belongs to the same category under the corresponding number of categories, regardless of which category it is clustered into. This ensures that multiple groups of text data in the text data set, regardless of how many classes they are divided into, belong to the same type of text data within the same group. This improves the reliability and accuracy of the text classification results. The text classification method of this application is applicable to any text classification with a specified category base value; therefore, the method of this application has good versatility and transferability.

[0065] Figure 2This is a flowchart illustrating a text classification method as an exemplary embodiment of this application. In a business scenario involving dialogue script mining, a dialogue script knowledge base is established using the mined dialogue script texts. An intelligent training robot can then provide dialogue script training to new agents based on the dialogue script texts in the knowledge base, facilitating their rapid familiarization with the business and mastery of effective communication skills. Currently, establishing the dialogue script knowledge base typically requires manual extraction and classification of dialogue script texts from a large amount of historical call data, which is labor-intensive. The text classification method of this application allows for the processing of historical call data to obtain classified dialogue script texts.

[0066] In some embodiments, such as Figure 2 As shown, this text classification method includes the following steps.

[0067] S201, Input multiple text data.

[0068] In this step, the data to be clustered includes multiple text data, specifically, multiple historical call text data.

[0069] S202, cluster multiple text data according to each of the multiple number of categories to obtain the category to which each text data belongs under different number of categories.

[0070] In this step, the base value of the category can be set based on prior knowledge. For example, if the number of categories of basic conversations in the agent conversation knowledge base is 30, then the base value of k is set to 30; if the number of categories of basic conversations in the agent conversation knowledge base is 40, then the base value of k is set to 40; if the number of categories of basic conversations in the agent conversation knowledge base is other values, then the base value of k is set to the same value as the other values.

[0071] Optionally, a portion of the data can be randomly extracted without replacement from multiple historical call text data and manually classified to obtain the number of categories, which can be set as the category base value. Based on the category base value and the above formula (1), multiple categories can be obtained.

[0072] S203: Extract one piece of text data from multiple pieces of text data, and obtain the category to which the text data belongs under different number of categories.

[0073] In this step, one historical call text data is extracted from multiple historical call text data, and the category to which the historical call text data belongs under different number of categories is obtained.

[0074] S204, take the intersection of the text data in each category to which the text data belongs under different number of categories.

[0075] In this step, the text data of the historical call can be intersected in the various categories to which the text data belongs under different number of categories. The historical call text data in the intersection is grouped as a text data group, namely the first group of text data.

[0076] S205: Iterate through each subsequent text data in multiple text data sets and perform loop processing.

[0077] In this step, each subsequent traversed text data is any historical call text data after removing the first group of grouped text data from multiple historical call text data. The cyclic processing includes obtaining the category to which the text data belongs under different number of categories in step S203 and step S204, resulting in multiple text data groups.

[0078] S206, count the number of text data in each text data group and remove noisy data.

[0079] In this step, each text data group corresponds to a category name. A category name represents one category in the category division result for that text data group. Multiple categories can include, for example, category γ_1, category γ_2, category γ_3, ..., γ_s, where s is an integer greater than or equal to 1. The number of text data contained in each category (γ_1, γ_2, γ_3, ..., γ_s) is counted, that is, the number of historical call text data contained in each category.

[0080] In this step, if the number of text data in a text data group is greater than a predetermined threshold, the text data group is retained; otherwise, the text data group is treated as noise data and the noise data is removed.

[0081] S207, Output the category classification results.

[0082] In this step, text data with a quantity greater than a predetermined threshold are grouped and assigned to text categories corresponding to category names. For each text category corresponding to a category name, the historical call text data in the text data group corresponding to that text type is used as the agent script text of a mined text type.

[0083] In this embodiment, multiple text data are classified using different numbers of categories (there are multiple numbers of categories to indicate how many classes there are). Each group of text data selected subsequently belongs to the same category under the corresponding number of categories, regardless of which number of categories is used for clustering. This classification result can ensure that multiple groups of text data in the text data set are of the same type, regardless of how many classes they are divided into, thereby improving the reliability and accuracy of the classification results obtained from text classification.

[0084] It is understood that the various method embodiments mentioned above in this application can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this application will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0085] In addition, this application also provides a text classification device, an electronic device, and a computer-readable storage medium, all of which can be used to implement any of the text classification methods provided in this application. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section, and will not be repeated here.

[0086] Figure 3 This is a block diagram of a text classification device provided in an embodiment of this application. (Refer to...) Figure 3 This application provides a text classification device 300, which may include the following modules.

[0087] Clustering module 310 is used to cluster multiple text data in a text data set according to each of the pre-set number of categories, so as to obtain the category to which each text data belongs under different number of categories; wherein, each text data corresponds to one category under one number of categories;

[0088] The segmentation module 320 is used to divide multiple text data into multiple groups of text data based on the category to which each text data belongs under different number of categories, with the same group of text data belonging to the same category under each number of categories.

[0089] The determination module 330 is used to determine the category classification result of each group of text data based on the category to which each group of text data belongs under different number of categories.

[0090] In some embodiments, the text classification device 300 further includes a category number generation module, configured to set a category base value for a plurality of category numbers before clustering according to each of the pre-set plurality of category numbers to obtain the category to which each text data belongs under different category numbers; to obtain a plurality of different category numbers by incrementing and / or decrementing the category base values ​​by a predetermined number of times with a predetermined step size; and to use the category base values ​​and the plurality of different category numbers as the pre-set plurality of category numbers.

[0091] In some embodiments, the category base value is an empirical value determined based on the predetermined number of basic categories of the text data set in the business scenario; or, the category base value is the number of categories obtained by manually classifying the extracted text data after randomly extracting data from the text data set without replacement.

[0092] In some embodiments, the clustering module 310 is specifically used for: performing feature mapping on each piece of text data in the text data set; obtaining a set of text feature vectors based on the feature vectors of each piece of text data obtained from the feature mapping; classifying each text feature vector in the set of text feature vectors according to the number of each category, and obtaining the category to which each text feature vector belongs under different number of categories; and taking the category to which each text feature vector belongs under different number of categories as the category to which each piece of text data belongs under different number of categories.

[0093] In some embodiments, the segmentation module 320 is specifically configured to: obtain first text data from the text data set as current text data; assign the current text data to the category to which it belongs under each category number as current categories; the first text data is any text data in the text data set; among the multiple text data included in the current categories, obtain the text data commonly contained in each category as a first group of text data; obtain second text data from the text data set as current text data; the second text data is any text data after removing the grouped first group of text data from the text data set; assign the current text data to the category to which it belongs under each category number as current categories; among the multiple text data included in the current categories, obtain the text data commonly contained in each category as a second group of text data, until each text data in the text data set is grouped, resulting in multiple groups of segmented text data.

[0094] In some embodiments, the determining module 330 is specifically used to: assign the category to which the i-th group of text data belongs under different number of categories as each category of the i-th group of text data, where i is an integer from 1 to n, and n is the total number of groups into which the multiple text data are divided; set a category name corresponding to each category of the i-th group of text data; count the number of texts in the i-th group of text data; and if the number of texts is greater than a predetermined threshold, classify the i-th group of text data into the text category corresponding to the category name, as the category classification result of the i-th group of text data.

[0095] In some embodiments, the determining module 330, after specifically used to count the number of texts in the i-th group of text data, is further specifically used to: treat the i-th group of text data as noise text data if the number of texts is less than or equal to a predetermined number threshold.

[0096] In some embodiments, the predetermined quantity threshold is a value obtained by calculating the ratio of the total number of text data in the text data set to the category base value of a preset number of categories.

[0097] According to the text classification apparatus of this application embodiment, multiple text data in a text data set are clustered according to each of a pre-set number of categories, to obtain the category to which each text data belongs under different numbers of categories. Each text data corresponds to a category under one number of categories. Then, the text data is grouped according to the category to which each text data belongs under different numbers of categories. The same group of text data belongs to the same category under each number of categories. According to this text classification method, multiple text data are first classified by different numbers of categories (there are multiple numbers of categories to indicate how many categories there are). Each group of text data that is subsequently selected belongs to the same category under the corresponding number of categories, regardless of which number of categories is used for clustering. The category division result obtained in this way can ensure that multiple groups of text data in the text data set are of the same type, regardless of how many categories they are divided into, thereby improving the reliability of the category division result obtained by text classification and improving the accuracy of the text classification result.

[0098] It should be clarified that the present invention is not limited to the specific configurations and processes described in the above embodiments and shown in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here, and the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0099] Figure 4 This is a block diagram of an electronic device provided in an embodiment of this application.

[0100] Reference Figure 4This application provides an electronic device, which includes: at least one processor 401; at least one memory 402; and one or more I / O interfaces 403 connected between the processor 401 and the memory 402; wherein the memory 402 stores one or more computer programs that can be executed by the at least one processor 401, and the one or more computer programs are executed by the at least one processor 401 to enable the at least one processor 401 to perform the above-described text classification method.

[0101] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program implements the above-described text classification method when executed by a processor / processor core. The computer-readable storage medium may be volatile or non-volatile.

[0102] This application also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described text classification method.

[0103] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0104] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0105] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0106] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing the status information of the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0107] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0108] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0109] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0110] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0111] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0112] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this application as set forth by the appended claims.

Claims

1. A text classification method, characterized in that, include: For multiple text data in a text dataset, clustering is performed according to each of the pre-set number of categories to obtain the category to which each text data belongs under different number of categories; wherein, each text data corresponds to one category under one number of categories; Based on the category to which each text data belongs under different number of categories, the multiple text data are divided into multiple groups of text data, and the text data in the same group belong to the same category under each number of categories. The category classification result of each group of text data is determined based on the category to which each group of text data belongs under different number of categories.

2. The method according to claim 1, characterized in that, Before clustering according to each of the pre-set multiple number of categories to obtain the category to which each text data belongs under different number of categories, the method further includes: Set the category base value for the plurality of categories; Based on the category base value, increment and / or decrement a predetermined number of times with a predetermined step size to obtain multiple different number of categories; The category base value and the number of different categories are used as the preset number of categories.

3. The method according to claim 2, characterized in that, The category base value is an empirical value determined based on the predetermined number of basic categories in the text data set in the business scenario; Alternatively, the category base value is the number of categories obtained by manually classifying the extracted text data after randomly extracting it without replacement from the text data set.

4. The method according to claim 1, characterized in that, The method involves clustering multiple text data points within a text dataset according to a pre-defined number of categories, resulting in the category to which each text data point belongs under different categories, including: Perform feature mapping on each piece of text data in the text dataset; Based on the feature vectors of each text data obtained from feature mapping, a set of text feature vectors is obtained; According to the number of each category, each text feature vector in the text feature vector set is classified to obtain the category to which each text feature vector belongs under different numbers of categories; The category to which each text feature vector belongs under different number of categories is taken as the category to which each text data belongs under different number of categories.

5. The method according to claim 1, characterized in that, The process of dividing the multiple text data into multiple groups based on the category to which each text data belongs under different number of categories includes: The first text data is obtained from the text data set as the current text data, and the category to which the current text data belongs under each category number is taken as the current category. The first text data is any text data in the text data set. Among the multiple text data included in each of the current categories, the text data commonly contained in each category is selected as the first group of text data; Obtain second text data from the text data set as new current text data, wherein the second text data is any text data after removing the first group of grouped text data from the text data set; The category to which the new current text data belongs under each category number is taken as the new current category; Among the multiple text data included in the new current categories, the text data commonly contained in the categories is obtained as the second group of text data, until each text data in the text data set is grouped, resulting in multiple groups of divided text data.

6. The method according to claim 1, characterized in that, The step of determining the category classification result of each group of text data based on the category to which each group of text data belongs under different number of categories includes: The category to which the i-th group of text data belongs under different number of categories is taken as the category of the i-th group of text data, where i is an integer from 1 to n, and n is the total number of groups into which the multiple text data are divided; Set a category name corresponding to each category of the i-th group of text data; Count the number of texts in the i-th group of text data; If the number of texts exceeds a predetermined threshold, the i-th group of text data is assigned to the text category corresponding to the category name, which is used as the category classification result of the i-th group of text data.

7. The method according to claim 6, characterized in that, After counting the number of texts in the i-th group of text data, the method further includes: If the number of texts is less than or equal to the predetermined threshold, the i-th group of text data is treated as noise text data.

8. A text classification device, characterized in that, include: The clustering module is used to cluster multiple text data in a text data set according to each of the pre-set number of categories, so as to obtain the category to which each text data belongs under different number of categories; wherein, each text data corresponds to one category under one number of categories; The segmentation module is used to divide the multiple text data into multiple groups of text data based on the category to which each text data belongs under different number of categories, with the same group of text data belonging to the same category under each number of categories. The determination module is used to determine the category classification result of each group of text data based on the category to which each group of text data belongs under different number of categories.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, the one or more computer programs being executed by the at least one processor to enable the at least one processor to perform the text classification method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the text classification method as described in any one of claims 1-7.

Images