Unsupervised non-parametric text classification method and device and computer equipment

Abstract

The invention relates to an unsupervised non-parametric text classification method and device and computer equipment. The method comprises the following steps: acquiring a training set and a test set; determining a to-be-classified text in a plurality of test texts included in the test set; retrieving a candidate text list in a training set based on the to-be-classified text and the target ES index; respectively inputting the to-be-classified text and the candidate text list into a first classification module, a second classification module and a third classification module to respectively obtain a first number of first tags, a first number of second tags and a first number of third tags; the first classification module, the second classification module and the third classification module respectively correspond to different unsupervised non-parameterized text classification algorithms; and calculating a first number of classification labels of the to-be-classified text based on the first number of first labels, the first number of second labels and the first number of third labels. By adopting the method, the text classification efficiency and accuracy can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of text classification, and in particular to an unsupervised non-parametric text classification method, device and computer equipment. Background Art

[0002] With the rapid development of computer technology, we have entered the era of big data. Correspondingly, the explosive growth of text data has put forward higher requirements for text classification technology. Text classification is an important task in natural language processing. Text classification is widely used in information retrieval, public opinion analysis, content recommendation and other fields.

[0003] At present, traditional text classification technology usually relies on supervised learning. Therefore, in specific implementation, traditional text classification technology often consumes a lot of labeled data acquisition costs and computing resources. Obviously, the technical limitations of traditional text classification technology make it difficult to improve the efficiency and accuracy of text classification in situations where computer resources are limited. Summary of the invention

[0004] Based on this, it is necessary to provide an unsupervised non-parametric text classification method, apparatus, computer device, computer-readable storage medium and computer program product that can improve the efficiency and accuracy of text classification in order to address the above technical problems.

[0005] In a first aspect, the present application provides an unsupervised non-parametric text classification method. The method comprises:

[0006] Obtain a training set and a test set; the training set includes multiple training texts each corresponding to a training label;

[0007] Determine a text to be classified from a plurality of test texts included in the test set;

[0008] Based on the text to be classified and the target ES index, a list of candidate texts is retrieved in the training set; the target ES index corresponds to the training set;

[0009] Inputting the text to be classified and the candidate text list into the first classification module, the second classification module and the third classification module respectively, and obtaining a first number of first labels, a first number of second labels and a first number of third labels respectively; the first classification module, the second classification module and the third classification module respectively correspond to different unsupervised non-parametric text classification algorithms;

[0010] Based on a first number of first labels, a first number of second labels, and a first number of third labels, a first number of classification labels for the text to be classified is calculated.

[0011] In a second aspect, the present application also provides an unsupervised non-parametric text classification device. The device comprises:

[0012] An acquisition module is used to acquire a training set and a test set; the training set includes a plurality of training texts respectively corresponding to training labels; and a text to be classified is determined from a plurality of test texts included in the test set;

[0013] ES retrieval module, used to retrieve a candidate text list in the training set based on the text to be classified and the target ES index; the target ES index corresponds to the training set;

[0014] A classification module, used for inputting the to-be-classified text and the candidate text list into a first classification module, a second classification module and a third classification module respectively, to obtain a first number of first labels, a first number of second labels and a first number of third labels respectively; the first classification module, the second classification module and the third classification module respectively correspond to different unsupervised non-parametric text classification algorithms;

[0015] The election voting module is used to calculate a first number of classification labels for the text to be classified based on a first number of first labels, a first number of second labels, and a first number of third labels.

[0016] In a third aspect, the present application further provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements some or all of the steps described in any method of the first aspect of the present application.

[0017] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps described in any method of the first aspect of the present application.

[0018] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements part or all of the steps described in any method of the first aspect of the embodiment of the present application.

[0019] The above-mentioned unsupervised non-parametric text classification method, device, computer equipment, storage medium and computer program product obtain a training set and a test set; the training set includes multiple training texts corresponding to training labels respectively; the text to be classified is determined from the multiple test texts included in the test set; based on the text to be classified and the target ES index, a candidate text list is retrieved in the training set; the target ES index corresponds to the training set; the text to be classified and the candidate text list are respectively input into the first classification module, the second classification module and the third classification module, and a first number of first labels, a first number of second labels and a first number of third labels are obtained respectively; the first classification module, the second classification module and the third classification module correspond to different unsupervised non-parametric text classification algorithms respectively; based on the first number of first labels, the first number of second labels and the first number of third labels, a first number of classification labels for the text to be classified are calculated. By adopting the unsupervised non-parametric text classification method provided by the present application, by avoiding supervised learning training of the training texts and labels in the training set, the efficiency of text classification can be further improved while the effectiveness of text classification (i.e., time performance) can be improved. Moreover, the target ES index can be used to improve the rapid positioning and retrieval of the text to be classified. At the same time, since the classification label of the text to be classified is determined based on the first label, the second label and the third label obtained by different unsupervised non-parametric text classification algorithms, the accuracy of text classification can be improved while the efficiency of text classification can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 An application environment diagram of an unsupervised non-parametric text classification method in one embodiment;

[0021] Figure 2 A schematic diagram of a flow chart of an unsupervised non-parametric text classification method in one embodiment;

[0022] Figure 3 A schematic diagram of a flow chart of an unsupervised non-parametric text classification method in another embodiment;

[0023] Figure 4 is a structural block diagram of an unsupervised non-parametric text classification device in one embodiment;

[0024] Figure 5 is an internal structure diagram of a computer device in one embodiment;

[0025] Figure 6 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0027] It should be noted that in the following description, the terms "first, second and third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first, second and third" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0028] The unsupervised non-parametric text classification method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 can be a smart phone, a tablet computer, a laptop computer, and a desktop computer.

[0029] The server 104 may be an independent physical server or a service node in a blockchain system. A peer-to-peer (P2P) network is formed between the service nodes in the blockchain system. The P2P protocol is an application layer protocol running on top of the Transmission Control Protocol (TCP).

[0030] In addition, server 104 can also be a server cluster composed of multiple physical servers, and can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), as well as big data and artificial intelligence platforms.

[0031] The terminal 102 and the server 104 may be connected via Bluetooth, USB (Universal Serial Bus) or a network or other communication connection methods, which are not limited in this application.

[0032] In one embodiment, Figure 2 As shown, an unsupervised non-parametric text classification method is provided, which can be Figure 1 The method is performed by a server or a terminal in the embodiment, or by a server and a terminal in collaboration. Figure 1The following steps are used as an example to illustrate the terminal execution in:

[0033] Step 202, obtaining a training set and a test set; the training set includes a plurality of training texts each corresponding to a training label.

[0034] The training set includes multiple training texts, and each training text has a corresponding training label; the training set is used to provide a data basis for text classification in the text classification process of the test set.

[0035] The training label represents the classification result of the training text in terms of the text content attribute. Optionally, the text content attribute may include the subject category, sentiment category or other attributes of the text content. Exemplarily, when the training label is used to represent the classification result of the training text in terms of the subject category of the text content, the training label may include "technology", "finance", "sports" or other labels; when the training label is used to represent the classification result of the training text in terms of the sentiment category of the text content, the training label may include "positive", "neutral" and "negative".

[0036] The test set includes multiple test texts, each of which corresponds to a test label. However, in the embodiment of the present application, the test labels carried by each test text itself are not adopted.

[0037] There is a corresponding relationship between the training set and the test set, and the training set and the test set may be included in the same data set. Optionally, the multiple training texts included in the training set and the multiple test texts included in the test set may have the same text structure, or may be able to be classified into corresponding classification results in terms of text content attributes.

[0038] Specifically, the terminal obtains multiple data sets, and determines a training set and a test set in a target data set; other data sets in the multiple data sets except the target data set include other training sets and other test sets.

[0039] Step 204: determine the text to be classified from the multiple test texts included in the test set.

[0040] The text to be classified is a test text among multiple test texts for which a classification label needs to be determined.

[0041] It is easy to understand that since the test tags carried by each test text itself are not adopted in the embodiment of the present application, the test tags carried by the text to be classified itself are not adopted in the embodiment of the present application.

[0042] Optionally, the text to be classified may be selected from a plurality of test texts through manual setting to classify the labels, or may be obtained by the terminal traversing at least one test text in the test set that has not been classified.

[0043] Step 206, based on the text to be classified and the target ES index, a candidate text list is retrieved in the training set; the target ES index corresponds to the training set.

[0044] The full name of ES is Elasticsearch, which is a search engine. In the embodiment of the present application, the training set corresponds to the target ES index, and other training sets correspond to other ES indexes.

[0045] The target ES index is used to quickly search the text to be classified so as to retrieve a second number of candidate texts having a certain similarity with the text to be classified from a plurality of training texts stored in the target ES index.

[0046] It is easy to understand that since the text to be classified is a test text in the test set, and the test set and the training set are contained in the same data set, based on the data set identifier of the text to be classified, the terminal can determine the data set to which the text to be classified belongs, and thus the terminal can determine the corresponding training set that belongs to the same data set as the text to be classified based on the data set to which the text to be classified belongs, so as to realize the retrieval of a list of candidate texts similar to the text to be classified in the training set based on the target ES search.

[0047] Specifically, based on the dataset identifier of the text to be classified, a target ES index of the dataset represented by the dataset identifier is determined.

[0048] In an exemplary embodiment, the above-mentioned retrieval of a candidate text list in a training set based on the text to be classified and the target ES index includes: obtaining search parameters; the search parameters include a data set identifier and a second number; determining a target ES index based on the data set identifier of the text to be classified; the target ES index corresponds to the training set; and retrieving the training text in the target ES index based on the text to be classified to retrieve a candidate text list including the second number of candidate texts in the training set.

[0049] The candidate text list is part or all of the training texts included in the training set, and the candidate text list includes multiple candidate texts that have certain similarities with the text to be classified in terms of text content attributes. The candidate text list is used to provide a second number of candidate texts that have certain similarities with the text to be classified, so as to assist in determining the first number of classification labels of the text to be classified based on the second number of candidate texts. It is easy to understand that the first number is at least one, the second number is at least one, and the first number is much smaller than the second number.

[0050] Optionally, the second number of candidate texts included in the candidate text list can be determined by manual setting; the second number of candidate texts included in the candidate text list can also be determined based on the number of training texts in the training set. Exemplarily, the number of training texts and the second number of candidate texts can be positively correlated.

[0051] Step 208, input the text to be classified and the candidate text list into the first classification module, the second classification module and the third classification module respectively, and obtain a first number of first labels, a first number of second labels and a first number of third labels respectively; the first classification module, the second classification module and the third classification module correspond to different unsupervised non-parametric text classification algorithms respectively.

[0052] Among them, the first classification module can be a classification module that performs text classification based on a text compression algorithm; the second classification module can be a classification module that performs text classification based on text similarity; and the third classification module can be a classification module that performs text classification based on text difference.

[0053] Optionally, when the first classification module is a classification module that performs text classification based on a text compression algorithm, the text compression algorithm may be gzip, bz2, lzma (Lempel-Ziv-Markov chain Algorithm), zstd (Zstandard) or other unsupervised non-parametric algorithms for compressing text; when the second classification module is a classification module that performs text classification based on text similarity, the text similarity may be calculated by TF-IDF (Term Frequency-Inverse Document Frequency), BM25 (Best Matching 25) or other unsupervised non-parametric algorithms for calculating text similarity; when the third classification module is a classification module that performs classification based on text difference, the text difference may be calculated by Jaccard, Euclidean distance or other unsupervised non-parametric algorithms for calculating text difference.

[0054] The first number of first labels are possible labels of the text to be classified determined by the first classification module based on the text to be classified and the candidate text list; the first number of second labels and the first number of third labels are the same, and will not be repeated here. That is to say, the first number of first labels, the first number of second labels, and the first number of third labels are determined based on different unsupervised non-parametric text classification algorithms. It is easy to understand that since different unsupervised non-parametric text classification algorithms have differences in algorithm properties, the first number of first labels, the first number of second labels, and the first number of third labels may have some of the same labels.

[0055] Unsupervised non-parametric text classification algorithms do not require supervised learning of training sets to train models, saving computing resources and improving classification efficiency. They also refer to text classification algorithms that do not rely on the assumption that data follows a specific distribution or pre-defined training parameters. Unsupervised non-parametric text classification algorithms can adapt to the actual distribution of text data rather than forcing text data to conform to a certain theoretical distribution. Therefore, unsupervised non-parametric text classification algorithms have higher flexibility and efficiency when performing text classification.

[0056] Step 210: Calculate a first number of classification labels for the text to be classified based on a first number of first labels, a first number of second labels, and a first number of third labels.

[0057] The first number of classification labels represent classification results of the text to be classified after text classification based on text content attributes.

[0058] In an exemplary embodiment, the above-mentioned calculation of the first number of classification labels for the text to be classified based on the first number of first labels, the first number of second labels and the first number of third labels includes: determining the first weight of the first classification module, the second weight of the second classification module and the third weight of the third classification module; calculating the first number of classification labels for the text to be classified based on the weighted sum of the first number of first labels, the first weight, the second weight of the first number of second labels, the first number of third labels and the third weight; the first weight corresponds to the first label, the second weight corresponds to the second label, and the third weight corresponds to the third label.

[0059] Furthermore, after calculating the first number of classification labels for the text to be classified, a text-label correspondence relationship can be established between the text to be classified and the first number of classification labels and stored. Optionally, the stored text-label correspondence relationship can be used in a scenario where text corresponding to the user's favorite label is recommended to the user or in other scenarios.

[0060] It is easy to understand that since the test set in the embodiment of the present application does not need to perform supervised learning training on multiple training texts in the training set, the text classification process in the embodiment of the present application is an unsupervised learning method, thereby avoiding the need for supervised learning training on a large amount of training data, so as to save a large amount of computer resources. Furthermore, the embodiment of the present application is suitable for situations where labeled data is difficult to obtain or the cost of obtaining is high, and is also suitable for scenarios where a large amount of unlabeled text data needs to be processed.

[0061] In the above-mentioned unsupervised non-parametric text classification method, a training set and a test set are obtained; the training set includes multiple training texts corresponding to training labels respectively; the text to be classified is determined from the multiple test texts included in the test set; based on the text to be classified and the target ES index, a candidate text list is retrieved in the training set; the target ES index corresponds to the training set; the text to be classified and the candidate text list are respectively input into the first classification module, the second classification module and the third classification module, and a first number of first labels, a first number of second labels and a first number of third labels are respectively obtained; the first classification module, the second classification module and the third classification module correspond to different unsupervised non-parametric text classification algorithms respectively; based on the first number of first labels, the first number of second labels and the first number of third labels, a first number of classification labels for the text to be classified are calculated. By adopting the unsupervised non-parametric text classification method provided by the present application, by avoiding supervised learning training of the training texts and labels in the training set, the efficiency of text classification can be further improved while the effectiveness of text classification (i.e., time performance) can be improved. Moreover, the target ES index can be used to improve the rapid positioning and retrieval of the text to be classified. At the same time, since the classification label of the text to be classified is determined based on the first label, the second label and the third label obtained by different unsupervised non-parametric text classification algorithms, the accuracy of text classification can be improved.

[0062] In an exemplary embodiment, the above-mentioned searching for a candidate text list in a training set based on the text to be classified and the target ES index includes:

[0063] Based on the dataset identifier of the text to be classified, determine the target ES index corresponding to the dataset identifier;

[0064] The training text in the target ES index is retrieved based on the text to be classified to retrieve a candidate text list in the training set; the candidate text list includes a second number of candidate texts.

[0065] The data set identifier is used to represent the unique data set corresponding to the text to be classified. Optionally, the data set identifier can be the name of the data set or other identifier that can represent the unique identity of the data set.

[0066] It is easy to understand that the target index corresponds to the dataset identifier, and the dataset includes a training set, so the target ES index corresponds to the training set.

[0067] In this embodiment, based on the dataset identifier of the text to be classified, the target ES index corresponding to the training set is determined; based on the text to be classified, the training text in the target ES index is retrieved to retrieve a candidate text list including the second number of candidate texts in the training set. Thus, through ES, a search engine that can quickly search for texts, the retrieval efficiency and retrieval accuracy of the candidate text list are improved, and further, it can be ensured that the first number of classification labels of the text to be classified obtained later are also accurate and credible.

[0068] In an exemplary embodiment, the method of calculating a first number of classification labels for the text to be classified based on a first number of first labels, a first number of second labels, and a first number of third labels includes:

[0069] Summarize a first number of first labels, a first number of second labels, and a first number of third labels to obtain a candidate label list;

[0070] Traverse multiple candidate tags included in the candidate tag list and count the number of occurrences of each candidate tag;

[0071] A first number of candidate tags whose occurrence times satisfy a first preset condition are calculated to obtain a first number of classification tags for the text to be classified.

[0072] The candidate label list is used to provide a candidate range for the classification label of the text to be classified, so as to further calculate a first number of classification labels of the text to be classified based on a plurality of candidate labels in the candidate range.

[0073] Specifically, multiple candidate tags included in the candidate tag list are traversed, and in the process of counting the number of occurrences of each candidate tag, if a candidate tag appears multiple times, the corresponding number of times is counted.

[0074] Optionally, the first preset condition is satisfied when the number of occurrences of the candidate tag is greater than a preset number, or when the number of occurrences of the candidate tag is ranked first in descending order.

[0075] Exemplarily, assuming that the first number = 3, the first preset condition is satisfied that the number of occurrences of the candidate label is ranked in the first number, the first label includes A, B and C, the second label includes C, D and E, and the third label includes A, C and E. Then the first number of first labels, the first number of second labels and the first number of third labels are summarized, and the candidate labels included in the obtained candidate label list are A, B, C, C, D, E, A, C and E. Further, the multiple candidate labels included in the candidate label list are traversed, and the number of occurrences of each candidate label is counted. The statistical result is: candidate label A appears 2 times, candidate label B appears 1 time, candidate label C appears 3 times, candidate label D appears 1 time, and candidate label E appears 2 times. It can be seen that the number of occurrences of candidate label C corresponds to the first ranking, the number of occurrences of candidate label A and candidate label E corresponds to the second ranking, and the number of occurrences of candidate label B and candidate label D corresponds to the fourth ranking. At this time, the candidate labels A, C and E meet the first preset condition. Therefore, the classification labels of the text to be classified are A, C and E.

[0076] Optionally, the above search parameters also include a second quantity.

[0077] In an exemplary embodiment, the above-mentioned aggregation of a first number of first tags, a first number of second tags, and a first number of third tags to obtain a candidate tag list includes: based on the first weight, the second weight, and the third weight, the first number of first tags, the first number of second tags, and the first number of third tags are aggregated to obtain a candidate tag list.

[0078] In this embodiment, first, a first number of first labels, a first number of second labels, and a first number of third labels are summarized to obtain a candidate label list, then, multiple candidate labels included in the candidate label list are traversed, and the number of occurrences of each candidate label is counted, and finally, the first number of candidate labels whose number of occurrences meets the first preset condition are calculated to obtain a first number of classification labels for the text to be classified. Thus, by fusing the number of occurrences of the first labels, second labels, and third labels obtained by different text classification algorithms, the accuracy and credibility of the first number of classification labels for the text to be classified can be significantly improved, overcoming the defect that the text classification results of a single unsupervised non-parametric text classification algorithm may not be objective due to its algorithmic nature.

[0079] In an exemplary embodiment, the method includes:

[0080] Define the data fields of the training set and define the corresponding data types based on the data fields;

[0081] Creating multiple formatted training texts based on data fields, data types, and each training text;

[0082] Based on each formatted training text, create a target ES index corresponding to the dataset identifier.

[0083] The data fields of the training set may be text content and the corresponding label category.

[0084] Optionally, the data type may be a string, a number or other types. Exemplarily, when the data fields of the training set are training labels and text content, the data type of the training labels may be a string, and the data type of the text content of the training text may also be a string.

[0085] Specifically, different training sets correspond to their own data fields and data types, and each training text in the same training set corresponds to the same data field and the same data type.

[0086] The formatted training text is used to represent the content of the data field of the training text and the data type corresponding to the data field. Exemplarily, when the data field is a training label and text content, and the data types of the training label and the text content are both strings, the formatted training text is used to represent the training label presented as a string type and the text content presented as a string type in the training text.

[0087] Optionally, the formatted training text may be training text in JSON format.

[0088] Specifically, based on each formatted training text, a target ES index corresponding to a data set identifier is created, an inverted index is established based on each formatted training text, and each formatted training text is saved in the target ES index. The inverted index is used to associate a certain word in a certain formatted training text with all other formatted training texts containing the word. Thus, when a user searches for a certain word, the target ES index can quickly find the training text containing the word without spending a lot of time on traversal search of all training texts.

[0089] Specifically, after the target ES index is created and before the training text in the target ES index is retrieved based on the text to be classified, the target ES index needs to be refreshed to ensure that each formatted training text can be searched.

[0090] It is easy to understand that, since the target ES index corresponds to a data set identifier, the target ES index has a corresponding relationship with the test set and the training set that belong to the same data set.

[0091] In this embodiment, the data fields of each training text are defined, and the corresponding data types are defined based on the data fields; multiple formatted training texts are created based on the data fields, data types, and each training text; and a target ES index corresponding to a data set identifier is created based on each formatted training text. Thus, through the target ES index corresponding to the data set identifier and storing multiple formatted training texts, the training texts in the target ES index can be efficiently retrieved based on the text to be classified, thereby achieving the retrieval efficiency and retrieval accuracy of the candidate text list. Furthermore, based on the target ES index, it can be ensured that the first number of classification labels of the text to be classified obtained subsequently are also accurate and credible.

[0092] In an exemplary embodiment, the above-mentioned searching the training text in the target ES index based on the text to be classified to retrieve the candidate text list in the training set includes:

[0093] The text to be classified is input into the target ES index, so that the target ES index performs word segmentation processing on the text to be classified to obtain multiple words included in the text to be classified, and retrieves each word in the text to be classified based on the inverted index to obtain at least one initial candidate text, and obtains the text similarity between the text to be classified and each initial candidate text based on the text similarity algorithm, and, based on the text similarity between the text to be classified and each initial candidate text, sorts each initial candidate text in descending order of similarity, so as to determine the second largest number of initial candidate texts in the sorting as candidate texts, and obtain a candidate text list.

[0094] The text similarity algorithm may be a TF-IDF algorithm or a BM25 algorithm.

[0095] The second number of candidate texts are, that is, the second number of candidate texts in the at least one initial candidate text that have the highest similarity with the text to be classified.

[0096] It is easy to understand that, since the initial candidate texts are obtained based on the inverted index, there is at least one common word between each initial candidate text and the text to be classified.

[0097] Specifically, when the TF-IDF algorithm is used to calculate the text similarity between the text to be classified and the initial candidate texts, the frequency of each word in the text to be classified and the rarity of each word in all candidate texts are comprehensively considered. The TF value (term frequency) of a certain word indicates the number of times the word appears in the text to be classified, and the IDF value (inverse document frequency) of a certain word indicates the rarity of the word in all candidate texts. After calculating the TF value and IDF value of the word, the weight of the word in the text to be classified can be obtained by multiplying the TF value and the IDF value, thereby calculating the text similarity between the text to be classified and the initial candidate texts.

[0098] In an exemplary embodiment, the text to be classified and the candidate text list are respectively input into the first classification module, the second classification module and the third classification module to obtain a first number of first labels, a first number of second labels and a first number of third labels, respectively, including:

[0099] Inputting the text to be classified and each candidate text into a first classification module, so that the first classification module compresses the text to be classified and each candidate text respectively, obtains the length of the compressed text to be classified and the length of each compressed candidate text respectively, and calculates the distance metric between the text to be classified and each candidate text based on the length of the compressed text to be classified and the length of each compressed candidate text, and obtains a first number of first labels based on the candidate labels corresponding to the first number of candidate texts that meet the second preset condition based on the distance metric;

[0100] The text to be classified and the candidate text list are respectively input into the second classification module and the third classification module to obtain a first number of second labels and a first number of third labels respectively.

[0101] Among them, the first classification module can be composed of a compressor and a classifier. Optionally, the first classification module can be composed of a gzip text compressor and a K-Nearest Neighbor (KNN) classifier, so that the gzip text compressor in the first classification module compresses the text to be classified and each candidate text respectively, and obtains the length of the compressed text to be classified and the length of each compressed candidate text respectively, and calculates the distance metric between the text to be classified and each candidate text based on the length of the compressed text to be classified and the length of each compressed candidate text; then, the KNN classifier of the first classification module determines the first number of candidate texts whose distance metric meets the second preset condition, so that the first number of first labels can be finally obtained based on the candidate labels of the first number of candidate texts.

[0102] Optionally, meeting the second preset condition may be that the distance metric is less than a preset distance, or it may be that the sorting corresponding to the distance metric is among the top first quantity. That is to say, the first quantity of first labels that meet the second preset condition are the first quantity of candidate labels with the smallest distance metric among multiple candidate labels.

[0103] Specifically, the first classification module aims to assign shorter codes to the words that appear more frequently in the text content, so as to be able to represent the text content with as few bits as possible. The idea of using the first classification module to perform text classification on the text to be classified is as follows: (1) The compressor used for text compression in the first classification module is good at capturing regularities; (2) The text content of the same category has more regularities than the text content of different categories. Exemplarily, assume that there is compressed text content t1 = Applications to bioinformatics, compressed text content t2 = Applications to mobile computing, and compressed text content t3 = Inpainting-basedcompression, perceptual coding. Obviously, t1 and t2 are of the same category, but t1 and t3 belong to different categories. Assume that C(·) represents the compression length of the compressed text content. Then at this time, there are four relational expressions: C(t1·t2)-C(t1)<C(t1·t3)-C(t1), C(t1·t2)<C(t1·t3), C(t1)<C(t1·t2), and C(t2)<C(t1·t2). Among them, C(t1·t2) represents the compression length after connecting t1 and t2. In other words, [C(t1·t2)-C(t1)] represents the meaning: based on the information in t1, how many more bytes are needed to encode t2, and [C(t1·t3)-C(t1)] represents the meaning: based on the information in t1, how many more bytes are needed to encode t3. This intuition can be formalized as a distance metric (Compression Length Metric, CLM). In this embodiment, the text to be classified is represented as x, a certain candidate text is represented as y, the length of the compressed text to be classified x is represented as C(x), the length of the compressed candidate text y is represented as C(y), and the compression length obtained after concatenating the text to be classified x and the candidate text y and then compressing is represented as C(xy). Then the formula for the distance metric CLM(x, y) between the text to be classified and the candidate text is as follows:

[0104]

[0105] Based on this, a first number of first labels are obtained based on the candidate labels corresponding to the first number of candidate texts that meet the second preset condition based on the distance metric. In a specific implementation, the candidate labels corresponding to the first number of candidate texts that are sorted from small to large according to the distance metric can be determined to obtain the first number of first labels.

[0106] In this embodiment, through a first classification module corresponding to a text compression algorithm, a first number of candidate texts having a closer distance measurement to the text to be classified are determined from a second number of candidate texts, and the candidate labels of the first number of candidate texts are determined as a first number of first labels. Thus, based on the first classification module, accurate classification of the text to be classified can be achieved in combination with the text compression algorithm.

[0107] In an exemplary embodiment, the text to be classified and the candidate text list are respectively input into the first classification module, the second classification module and the third classification module to obtain a first number of first labels, a first number of second labels and a first number of third labels, respectively, including:

[0108] Input the text to be classified and each candidate text into the second classification module, so that the second classification module constructs a vocabulary of the text to be classified and a vocabulary of each candidate text, determines the TF-IDF vector of the text to be classified based on the vocabulary of the text to be classified, determines the TF-IDF matrix corresponding to the multiple candidate texts based on the vocabulary of each candidate text, and calculates the cosine similarity between the text to be classified and each candidate text based on the TF-IDF vector of the text to be classified and the TF-IDF matrix, and obtains a first number of second labels based on the candidate labels corresponding to the first number of candidate texts whose cosine similarities meet the third preset condition;

[0109] The text to be classified and each candidate text are input into the first classification module and the third classification module respectively, and a first number of first labels and a first number of third labels are obtained respectively.

[0110] The second classification module lists all unique words in the text to be classified and all unique words in each candidate text, respectively, so as to construct a vocabulary list of the text to be classified and a vocabulary list of each candidate text, respectively.

[0111] The second classification module can be composed of a text similarity algorithm and a classifier. Optionally, the second classification module can be composed of a TF-IDF algorithm and a KNN classifier, so that the TF value and IDF value of the text to be classified and each candidate text are calculated respectively by the TF-IDF algorithm in the second classification module, and then the TF-IDF vector of the text to be classified is determined based on the vocabulary of the text to be classified, and the TF-IDF matrix corresponding to the multiple candidate texts is determined based on the vocabulary of each candidate text, and then the cosine similarity between the text to be classified and each candidate text is calculated based on the TF-IDF vector and TF-IDF matrix of the text to be classified; then the first number of candidate texts whose cosine similarity meets the third preset condition is determined by the KNN classifier of the second classification module, so that the first number of second labels can be finally obtained based on the candidate labels of the first number of candidate texts.

[0112] Specifically, in the TF-IDF matrix, each row represents a candidate text, and each column represents a word in the vocabulary of the corresponding candidate text.

[0113] Optionally, the third preset condition is satisfied when the cosine similarity is greater than the first preset similarity or when the cosine similarity is ranked first in descending order. That is, the first number of second tags that satisfy the third preset condition are the first number of candidate tags with the largest cosine similarity among the multiple candidate tags.

[0114] Specifically, the TF-IDF value can reflect the importance of each word in the corresponding text content. The term frequency (TF) indicates the frequency of a word appearing in the corresponding candidate text, while the inverse document frequency (IDF) measures the commonness of the word in all candidate texts. The number of times a word t appears in a candidate text d is represented as f t,d , the total number of occurrences of all words in the candidate text d is expressed as ∑ t'∈d f t',d , the TF value of word t in candidate text d is expressed as TF(t, d), then the calculation formula of TF(t, d) is as follows:

[0115]

[0116] The total number of candidate texts is represented as N, the number of candidate texts containing the word is represented as |{d∈D:t∈d}|, and the IDF value of word t is represented as IDF(t). The calculation formula of IDF(t) is as follows:

[0117]

[0118] The TF-IDF value of a word t in a candidate text d is expressed as TFIDF(t, d), and the calculation formula of TF-IDF(t, d) is as follows:

[0119] TFIDF(t,d)=TF(t,d)×IDF(t);

[0120] Based on the TF-IDF value of each word in each candidate text, a TF-IDF matrix corresponding to multiple candidate texts is formed, and the TF-IDF vector of each candidate text is calculated based on the TF-IDF matrix, and the TF-IDF value of the text to be classified is obtained by referring to the TF-IDF value determination method based on the same principle as above to determine the TF-IDF vector of the text to be classified. The TF-IDF vector of the text to be classified is represented as A, the TF-IDF vector of a candidate text is represented as B, and the cosine similarity between the text to be classified and a candidate text is represented as cosine_similarity(A, B), then the calculation formula of cosine_similarity(A, B) is as follows:

[0121]

[0122] Among them, for easy understanding, the cosine similarity cosine_similarity(A, B) between the text to be classified and a candidate text represents the similarity between the TF-IDF vector of the text to be classified and the TF-IDF vector of a candidate text. Based on this, the first number of second labels are obtained based on the candidate labels corresponding to the first number of candidate texts whose cosine similarity meets the third preset condition. In a specific implementation, the candidate labels corresponding to the first number of candidate texts whose cosine similarity corresponds to the order from large to small are determined to obtain the first number of second labels.

[0123] In this embodiment, through a second classification module corresponding to a text similarity algorithm, a first number of candidate texts having a higher cosine similarity with the text to be classified are determined from a second number of candidate texts, and the candidate labels of the first number of candidate texts are determined as a first number of second labels. Thus, based on the second classification module, accurate classification of the text to be classified can be achieved in combination with text similarity.

[0124] In an exemplary embodiment, the text to be classified and the candidate text list are respectively input into the first classification module, the second classification module and the third classification module to obtain a first number of first labels, a first number of second labels and a first number of third labels, respectively, including:

[0125] Input the text to be classified and each candidate text into the third classification module, so that the third classification module constructs a vocabulary list of the text to be classified and a vocabulary list of each candidate text, determines the number of intersection words between the text to be classified and each candidate text and the number of union words between the text to be classified and each candidate text based on the vocabulary list of the text to be classified and the vocabulary list of each candidate text, calculates the Jaccard similarity coefficient between the text to be classified and each candidate text based on the number of each intersection word and the number of each union word, and obtains a first number of third labels based on the candidate labels corresponding to the first number of candidate texts whose Jaccard similarity coefficients meet the fourth preset condition;

[0126] The text to be classified and each candidate text are input into the first classification module and the second classification module respectively, and a first number of first labels and a first number of second labels are obtained respectively.

[0127] The third classification module lists all unique words in the text to be classified and all unique words in each candidate text, thereby constructing a vocabulary list of the text to be classified and a vocabulary list of each candidate text.

[0128] The third classification module can be composed of a text difference algorithm and a classifier. Optionally, the third classification module can be composed of a Jaccard similarity coefficient and a KNN classifier, so that the number of intersection words between the text to be classified and each candidate text and the number of union words between the text to be classified and each candidate text are determined in the third classification module, and then the Jaccard similarity coefficient between the text to be classified and each candidate text is calculated based on the number of intersection words and the number of union words; then the first number of candidate texts whose Jaccard similarity coefficients meet the fourth preset condition are determined by the KNN classifier of the third classification module, so that the first number of third labels can be finally obtained based on the candidate labels of the first number of candidate texts.

[0129] Optionally, the fourth preset condition is satisfied when the Jaccard similarity coefficient is greater than the second preset similarity, or when the Jaccard similarity coefficient is ranked first in descending order. That is, the first number of third tags that satisfy the fourth preset condition are the first number of candidate tags with the largest Jaccard similarity coefficient among the multiple candidate tags.

[0130] Specifically, the Jaccard similarity coefficient measures the degree of overlap between the text classification to be classified and a candidate text by calculating the ratio of the number of intersection words to the number of union words between the text classification to be classified and a candidate text. It is easy to understand that the higher the Jaccard similarity coefficient, the more common words there are between the text classification to be classified and the candidate text. Let the vocabulary of the text to be classified be C, the vocabulary of a candidate text be D, the number of intersection words between the text to be classified and the candidate text be |C∩D|, the number of union words between the text to be classified and each candidate text be |C∪D|, and the Jaccard similarity coefficient between the text classification to be classified and the candidate text be J(C, D). The calculation formula of J(C, D) is as follows:

[0131]

[0132] Among them, for the sake of understanding, the Jaccard similarity coefficient J(C, D) between the text classification to be classified and the candidate text represents the degree of overlap between the text classification to be classified and the candidate text. Based on this, the first number of third labels are obtained based on the candidate labels corresponding to the first number of candidate texts whose Jaccard similarity coefficients meet the fourth preset condition. In a specific implementation, the candidate labels corresponding to the first number of candidate texts whose Jaccard similarity coefficients are ranked from large to small can be determined to obtain the first number of third labels.

[0133] In this embodiment, through the third classification module corresponding to the text difference algorithm, a first number of candidate texts having a higher Jaccard similarity coefficient with the text to be classified are determined from the second number of candidate texts, and the candidate labels of the first number of candidate texts are determined as the first number of third labels. Thus, based on the third classification module, it is possible to accurately classify the text to be classified in combination with the text difference.

[0134] Optionally, the second classification module and the third classification module may not construct a vocabulary list for each candidate text, but only construct a vocabulary list for the text to be classified.

[0135] In an exemplary embodiment, the text to be classified and the candidate text list are respectively input into the first classification module, the second classification module and the third classification module to obtain a first number of first labels, a first number of second labels and a first number of third labels, respectively, including: first, the text to be classified and the candidate text list are input into the first classification module to obtain a first number of first labels; after obtaining the first number of first labels, the text to be classified and the candidate text list are input into the second classification module to obtain a second number of second labels; after obtaining the first number of second labels, the text to be classified and the candidate text list are input into the third classification module to obtain a first number of third labels.

[0136] It should be noted that the above exemplary embodiment is only a feasible input order, but in a specific implementation, the text to be classified and the candidate text list are respectively input into the first classification module, the second classification module and the third classification module, which can be input into the first classification module, the second classification module and the third classification module at the same time, or the first classification module, the second classification module and the third classification module can be input in a certain order. The present application does not specifically limit the input order of the first classification module, the second classification module and the third classification module.

[0137] In an exemplary embodiment, when the second classification module and the third classification module do not construct a vocabulary for each candidate text, but only construct a vocabulary for the text to be classified, the above method also includes: acquiring multiple initial data sets; performing data cleaning on each initial data set to obtain multiple cleaned data sets; and constructing a vocabulary for each training text included in the training set in each data set.

[0138] The initial data set may be obtained through channels including social media, news websites, forums or other channels including text content.

[0139] Specifically, each initial data set is cleaned, including removing HTML tags, special characters, stop words, etc. in each initial data set, so as to improve the quality of each text content in each initial data set.

[0140] It is easy to understand that if the vocabulary of each training text is constructed in advance, the second classification module and the third classification module can save the time of word segmentation processing on each candidate text, which can improve the classification efficiency of the second classification module and the third classification module. However, constructing the vocabulary of each training text in advance will consume a large amount of system resources in the data preparation stage; on the contrary, if the second classification module and the third classification module construct the vocabulary of each candidate text, it is only necessary to perform targeted word segmentation processing on each determined candidate text. It can be seen that although the second classification module and the third classification module need to perform more word segmentation processing, they can significantly reduce the time spent in the data preparation stage and save a lot of system resources.

[0141] The application process of the above unsupervised non-parametric text classification method is described below in conjunction with a detailed embodiment. Assume that the first number is k, the second number is m, the first classification module is a gzip+KNN module, the second classification module is a TF-IDF+KNN module, and the third classification module is a Jaccard+KNN module. Figure 3 As shown, the details are as follows:

[0142] Get data sets A, B, ..., H; data set A includes training set A and test set A, data set B includes training set B and test set B, data set H includes training set H and test set H; training set A, B, ..., H respectively include multiple training texts corresponding to training labels.

[0143] Define the data fields of each training set as label lab and text content text, and define the data types of label lab and text content text as strings; create multiple training texts in JSON format based on the data fields, data types and each training text; based on each training text in JSON format, create ES indexes corresponding to their respective data set identifiers for training set A, training set B...training set H, namely ES index A, ES index B...ES index H; specifically, ES index A is: [{textA1, labA1}, {textA2, lab A2}...], ES index B is: [{text B1, lab B1}, {text B2, lab B2}...]...ES index H is: [{text H1, lab H1}, {text H2, lab H2}...].

[0144] Among the multiple test texts included in the test set A, the test set B, ..., the test set H, a text to be classified that has not yet been classified is determined.

[0145] Based on the data set identifier of the text to be classified, determine the ES index i of the data set represented by the data set identifier; the target ES index i corresponds to the training set i; i = [A, B, ...H]; the ES index i is: [{text i1, lab i1}, {text i2, labi2} ...]; based on the text to be classified, retrieve the training text in the ES index i to retrieve a candidate text list including m candidate texts in the training set i; specifically, the candidate text list is: {text i1, lab i1}, {text i2, labi2} ... {text im, lab im}.

[0146] The text to be classified and the candidate text list are respectively input into the gzip+KNN module, the TF-IDF+KNN module and the Jaccard+KNN module, and k first labels labg, k second labels labt and k third labels labj are obtained based on distance measurement, cosine similarity and Jaccard similarity coefficient respectively; specifically, the k first labels labg are: {labg1, labg2…labgk}, the k second labels labt are: {labt1, labt2…labtk}…the k third labels labj are: {labj1, labj2…labjk}.

[0147] Summarize k first labels labg, k second labels labt and k third labels labj to obtain a candidate label list; traverse multiple candidate labels included in the candidate label list, and count the number of occurrences count of each candidate label; specifically, the number of occurrences of each candidate label is: {lab i1: count1, lab i2: count2...}; determine the top k candidate labels whose number of occurrences count is sorted from large to small, and obtain k classification labels for the text to be classified; specifically, the k classification labels are: {lab 1, lab 2...lab k}.

[0148] At this point, the text classification of the text to be classified is completed, and the next text to be classified that has not yet been classified is determined from the multiple test texts included in the test set A, test set B...test set H, and the above unsupervised non-parametric text classification method is repeated until the multiple test texts included in the test set A, test set B...test set H have all completed text classification and correspond to k classification labels respectively.

[0149] In this embodiment, by integrating multiple unsupervised non-parametric technologies (gzip, TF-IDF and Jaccard), the need for supervised learning training on a large amount of training data is avoided. While improving the text classification effect by combining multiple unsupervised non-parametric technologies, it also ensures that the unsupervised non-parametric text classification method provided by this embodiment can also run efficiently on terminals with limited computing resources; the ES search engine is used to index and retrieve the text, and the combination of multiple unsupervised non-parametric text classification algorithms and KNN classifiers is used to achieve efficient classification of the text; by counting the number of occurrences of each candidate label obtained by different classification modules and fusing the classification results, the stability and accuracy of the first number of classification labels of the text to be classified are significantly improved.

[0150] The implementation results of the above unsupervised non-parametric text classification method are described below in conjunction with a detailed embodiment, as follows:

[0151] (1) Dataset

[0152] In the embodiment of the present invention, five data sets are selected to study the impact of the number of training texts, the number of topic categories and the length of text on the accuracy. Traditional text classification research has two non-overlapping preferences when selecting evaluation data sets: methods based on Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) prefer large-scale data sets (such as AGNews), while conduction methods such as graph convolutional neural networks focus on smaller data sets (such as 20News, Ohsumed, R8, R52). Therefore, the five selected data sets are AG News, 20News, Ohsumed, R8 and R52. Among them, the AG News dataset is a news classification dataset that contains news article titles and descriptions from four major categories (world, sports, business, and technology). The AGNews dataset is often used for text classification tasks, for example, to determine which news category an article belongs to; the 20News dataset is a widely used text dataset that contains approximately 20,000 newsgroup articles distributed in 20 different subject categories. The 20News dataset is often used for text classification, topic modeling, and other natural language processing (NLP) tasks; the Ohsumed dataset contains medical abstracts from the MEDLINE database for text classification and information retrieval tasks. The Ohsumed dataset contains abstracts of medical literature related to cardiovascular disease; the R8 dataset is often used to evaluate the performance of text classification algorithms. Since the R8 dataset contains a limited number of categories, the R8 dataset has become a useful resource for researchers and developers to test and compare different models. The R8 dataset is often used for long short-term memory networks (LSTMs). The R52 dataset is a text classification model based on Deep Memory (LSTM) to demonstrate how to use modern deep learning methods to solve text classification problems. It contains 52 categories of news articles. The number of subject categories in the R52 dataset is larger than that of the R8 dataset. The R52 dataset is often used to test the performance of text classification models when dealing with more fine-grained classification. Similarly, the R52 dataset is also often used in academic research and industrial applications to evaluate and improve text classification technology.

[0153] Exemplarily, Table 1 is a detailed information table of the data sets used for evaluation, and the detailed information of each data set is listed as shown in Table 1, wherein SN represents the abbreviation of the data set, NEtrain represents the number of training texts, NEtest represents the number of test texts, NC represents the number of topic categories, NW represents the average number of words in each text (including training texts and test texts), NL represents the average number of characters in each text, and VS represents the vocabulary size of the data set.

[0154] Dataset SN NEtrain NEtest NC NW NL VS AGNews AGN 120K 7.6K 4 44 236 128K 20News 20N 11K 7.5K 20 406 1902 277K Ohsumed Oh 3.4K 4K 23 212 1273 55K R8 R8 5.5K 2.2K 8 102 587 24K R52 R52 6.5K 2.6K 52 110 631 26K

[0155] Table 1

[0156] (2) Baseline Model

[0157] The embodiments of the present invention compare the implementation results with 1) neural network methods that require training and 2) non-parametric methods that directly use KNN classifiers, which may or may not use external data for pre-training. Specifically, the baseline model selects mainstream architectures for text classification, such as logistic regression, fastText, RNN with or without attention (conventional LSTM, bidirectional LSTM with attention, hierarchical attention network), CNN, and BERT models. In addition, the baseline model also includes four other non-parametric methods: word2vec (W2V), pre-trained sentence BERT (SentBERT) and gzip, and these four non-parametric methods all use KNN classifiers.

[0158] (3) Implementation details

[0159] For all baseline models, the hyperparameters of the embodiment of the present invention for the full data set are set as follows: Exemplarily, for LSTM, Bidirectional Long Short-Term Memory with Attention (Bi-LSTM+Attn) and fastText, an embedding size of embedding size = 256 and a dropout rate of 0.3 are used for the proportion of neurons randomly discarded at each iteration. For the full data set setting, the parameters are as follows: learning rate is 0.001, decay rate is 0.9 (for Adam optimizer), the number of full iteration epoch training rounds is 20, and batch size batchsize = 64. For LSTM and Bi-LSTM+Attn, the number of RNN layers is set to 1, and the number of hidden layer neurons is set to 64. A hidden layer of size 10 is used for fastText. For the Hierarchical Attention Network (HAN), a single layer is used for both sentence-level and word-level RNNs, the hidden size of the two attention layers is set to 100, and the hidden size of each is set to 50. Training was performed using batch size = 256 and decay rate = 0.5 for six epochs. For implementation, the public hedwig repository was also used, using the shallowest 9-layer version of the Very Deep Convolutional Networks for Text Classification (VDCNN), with batch size set to 64, learning rate set to 1e-4, batch size = 1, and epoch training rounds = 60. The embedding size was set to 16. For the Region-based Convolutional Neural Network (RCNN) with CNN and region nomination, the same batch size and epoch settings as VDCNN were used, as well as embedding size = 256, RNN hidden size = 256, and learning rate = 1e-3.

[0160] In the preprocessing process, the embodiments of the present invention do not use pre-trained word embeddings for any word-based models. The distinction between "training" and "pre-training" is strict, so using pre-trained embeddings will lead to ambiguity in the DNN category. Data enhancement is also not used during training. The tokenization process of word and character-level data and batch padding are inevitable.

[0161] For the method proposed in the embodiment of the present invention, the second number m=50 and the first number k=2 are set on all data sets, and the maximum possible accuracy obtained from the experiment is reported. The method of the present invention only requires a CPU, and an 8-core CPU is used to take advantage of a single processor.

[0162] (4) Comparison of accuracy results

[0163] For example, Table 2 is a comparison table of the test accuracy of all data sets. It can be clearly seen from Table 2 that different models show different performances on the five data sets of AGNews, 20News, Ohsumed, R8 and R52. A careful look at the various performance indicators shows that the embodiment of the present invention has excellent accuracy relative to other models. In terms of pre-training and training, most of the models listed require training to achieve their reported accuracy, and some of them also require pre-training. It is worth noting that neither the embodiment of the present invention nor the gzip model requires any training. This property greatly reduces the threshold for deploying advanced text classification models because it eliminates the need for a large amount of computing resources and complex data preparation processes. In terms of accuracy, with an accuracy of 0.966 in the AGNews data set, the embodiment of the present invention not only surpasses other models in this particular data set, but also shows the highest overall performance in all data sets. On the 20News and R52 data sets, fine-tuning BERT achieved the highest performance among all methods, while the embodiment of the present invention achieved competitive results without any pre-training and training, only 0.001 and 0.009 lower than BERT.

[0164]

[0165] Table 2

[0166] In general, the embodiment of the present invention not only has high-precision performance on multiple data sets, but also has high efficiency and good usability because it does not require pre-training or training, that is, the embodiment of the present invention is an unsupervised non-parametric text classification method, which highlights the powerful generalization ability of the embodiment of the present invention and its potential to simplify and enhance text classification tasks in various fields. Compared with traditional pre-trained models, the text classification performance achieved by the embodiment of the present invention makes the embodiment of the present invention an efficient text classification tool that is both practical and accurate.

[0167] (5) Time performance comparison

[0168] In order to verify the time performance efficiency of the embodiment of the present invention, the embodiment of the present invention is compared with the gzip baseline model that does not require pre-training and training on all data sets. The results are shown in Table 3, which shows the test time performance (unit: seconds) on all data sets.

[0169] Model\Dataset AGNews 20News Ohsumed R8 R52 gzip 65305 15347 1280 637 955 Embodiments of the present invention 376 2583 541.6 193 246

[0170] Table 3

[0171] It can be seen that across all datasets, the embodiments of the present invention consistently outperform the gzip baseline model in terms of time efficiency. The most significant improvements were observed on large datasets (e.g., AGNews, 20News), where processing time was reduced by several orders of magnitude. However, even on small and medium-sized datasets such as Ohsumed, R8, and R52, the embodiments of the present invention show clear advantages and can complete the task in a shorter time. By significantly reducing the processing time, the embodiments of the present invention not only provide an effective solution for text classification tasks, but also open up new possibilities for processing large-scale data analysis in resource-constrained environments. This efficiency, coupled with its scalability and versatility, makes the embodiments of the present invention a strong choice for a wide range of applications.

[0172] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0173] Based on the same inventive concept, the embodiment of the present application also provides an unsupervised non-parametric text classification device for implementing the unsupervised non-parametric text classification method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more unsupervised non-parametric text classification device embodiments provided below can refer to the limitations of the unsupervised non-parametric text classification method above, and will not be repeated here.

[0174] In one embodiment, Figure 4 As shown, an unsupervised non-parametric text classification device is provided, including: an acquisition module 402, an ES retrieval module 404, a classification module 406 and an election voting module 408, wherein:

[0175] The acquisition module 402 is used to acquire a training set and a test set; the training set includes a plurality of training texts respectively corresponding to training labels; and the text to be classified is determined from the plurality of test texts included in the test set.

[0176] The ES retrieval module 404 is used to retrieve a candidate text list in the training set based on the text to be classified and the target ES index; the target ES index corresponds to the training set.

[0177] Classification module 406 is used to input the text to be classified and the candidate text list into the first classification module, the second classification module and the third classification module respectively, to obtain a first number of first labels, a first number of second labels and a first number of third labels respectively; the first classification module, the second classification module and the third classification module correspond to different unsupervised non-parametric text classification algorithms respectively.

[0178] The voting module 408 is used to calculate a first number of classification labels for the text to be classified based on a first number of first labels, a first number of second labels, and a first number of third labels.

[0179] In an exemplary embodiment, the election voting module 408 is also used to determine the first weight of the first classification module, the second weight of the second classification module, and the third weight of the third classification module; based on the weighted sum of the first number of first labels, the first weight, the first number of second labels second weight, the first number of third labels and the third weight, calculate the first number of classification labels for the text to be classified; the first weight corresponds to the first label, the second weight corresponds to the second label, and the third weight corresponds to the third label.

[0180] In an exemplary embodiment, the ES retrieval module 404 is also used to determine a target ES index corresponding to a data set identifier based on the data set identifier of the text to be classified; retrieve the training text in the target ES index based on the text to be classified to retrieve a candidate text list in the training set; the candidate text list includes a second number of candidate texts.

[0181] In an exemplary embodiment, the ES retrieval module 404 is also used to input the text to be classified into the target ES index, so that the target ES index performs word segmentation on the text to be classified to obtain multiple words included in the text to be classified, and retrieves each word in the text to be classified based on the inverted index to obtain at least one initial candidate text, and obtains the text similarity between the text to be classified and each initial candidate text based on the text similarity algorithm, and, based on the text similarity between the text to be classified and each initial candidate text, sorts each initial candidate text in order of similarity from large to small, so as to determine the second largest number of initial candidate texts in the sorting as candidate texts, and obtain a candidate text list.

[0182] In an exemplary embodiment, the election voting module 408 is also used to summarize a first number of first labels, a first number of second labels, and a first number of third labels to obtain a candidate label list; traverse the multiple candidate labels included in the candidate label list to count the number of occurrences of each candidate label; calculate the first number of candidate labels whose number of occurrences meets the first preset condition to obtain a first number of classification labels for the text to be classified.

[0183] In an exemplary embodiment, the ES retrieval module 404 is also used to define data fields of the training set and define corresponding data types based on the data fields; create multiple formatted training texts based on the data fields, data types and each training text; and create a target ES index corresponding to the data set identifier based on each formatted training text.

[0184] In an exemplary embodiment, the above-mentioned classification module 406 is also used to input the text to be classified and each candidate text into the first classification module, so that the first classification module compresses the text to be classified and each candidate text respectively, and obtains the length of the compressed text to be classified and the length of each compressed candidate text respectively, and calculates the distance measurement between the text to be classified and each candidate text based on the length of the compressed text to be classified and the length of each compressed candidate text, and obtains a first number of first labels based on the candidate labels corresponding to the first number of candidate texts that meet the second preset condition based on the distance measurement; the text to be classified and the candidate text list are input into the second classification module and the third classification module respectively, and a first number of second labels and a first number of third labels are obtained respectively.

[0185] In an exemplary embodiment, the classification module 406 is also used to input the text to be classified and each candidate text into a second classification module, so that the second classification module constructs a vocabulary of the text to be classified and a vocabulary of each candidate text, determines the TF-IDF vector of the text to be classified based on the vocabulary of the text to be classified, determines the TF-IDF matrix corresponding to multiple candidate texts based on the vocabulary of each candidate text, and calculates the cosine similarity between the text to be classified and each candidate text based on the TF-IDF vector of the text to be classified and the TF-IDF matrix, and obtains a first number of second labels based on the candidate labels corresponding to the first number of candidate texts whose cosine similarity meets the third preset condition; the text to be classified and each candidate text are respectively input into the first classification module and the third classification module, and a first number of first labels and a first number of third labels are obtained respectively.

[0186] In an exemplary embodiment, the above-mentioned classification module 406 is also used to input the text to be classified and each candidate text into a third classification module, so that the third classification module constructs a vocabulary list of the text to be classified and a vocabulary list of each candidate text, and based on the vocabulary list of the text to be classified and the vocabulary list of each candidate text, determines the number of intersection words between the text to be classified and each candidate text and the number of union words between the text to be classified and each candidate text, and calculates the Jaccard similarity coefficient between the text to be classified and each candidate text based on the number of intersection words and the number of union words, and obtains a first number of third labels based on the candidate labels corresponding to the first number of candidate texts that meet the fourth preset condition based on the Jaccard similarity coefficient; the text to be classified and each candidate text are respectively input into the first classification module and the second classification module, and a first number of first labels and a first number of second labels are obtained respectively.

[0187] Each module in the above unsupervised non-parametric text classification device can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0188] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store text data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an unsupervised non-parametric text classification method is implemented.

[0189] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an unsupervised non-parametric text classification method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0190] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0191] In an exemplary embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0192] In an exemplary embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0193] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0194] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0195] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0196] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0197] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. An unsupervised non-parametric text classification method, characterized in that: The method comprises: Obtaining a training set and a test set; the training set includes a plurality of training texts respectively corresponding to training labels; Determining a text to be classified from a plurality of test texts included in the test set; Retrieving a list of candidate texts in the training set based on the text to be classified and a target ES index; the target ES index corresponds to the training set; Inputting the text to be classified and the candidate text list into a first classification module, a second classification module and a third classification module respectively, and obtaining a first number of first labels, a first number of second labels and a first number of third labels respectively; the first classification module, the second classification module and the third classification module respectively correspond to different unsupervised non-parametric text classification algorithms; Based on a first number of the first labels, a first number of the second labels, and a first number of the third labels, a first number of classification labels for the text to be classified is calculated.

2. The method according to claim 1, characterized in that The retrieving a candidate text list in the training set based on the text to be classified and the target ES index includes: Based on the data set identifier of the text to be classified, determining the target ES index corresponding to the data set identifier; The training text in the target ES index is retrieved based on the text to be classified, so as to retrieve the candidate text list in the training set; the candidate text list includes a second number of candidate texts.

3. The method according to claim 2, characterized in that The step of retrieving the training text in the target ES index based on the text to be classified to retrieve the candidate text list in the training set includes: The text to be classified is input into the target ES index so that the target ES index performs word segmentation processing on the text to be classified to obtain multiple words included in the text to be classified, and retrieves each word in the text to be classified based on the inverted index to obtain at least one initial candidate text, and obtains the text similarity between the text to be classified and each of the initial candidate texts based on a text similarity algorithm, and, based on the text similarity between the text to be classified and each of the initial candidate texts, sorts each of the initial candidate texts in descending order of similarity, so as to determine the second number of initial candidate texts ranked first as candidate texts, and obtain the candidate text list.

4. The method according to claim 1, characterized in that: The calculating a first number of classification labels for the to-be-classified text based on a first number of the first labels, a first number of the second labels, and a first number of the third labels includes: Summarizing a first number of the first tags, a first number of the second tags, and a first number of the third tags to obtain a candidate tag list; Traversing multiple candidate tags included in the candidate tag list, and counting the number of occurrences of each candidate tag; A first number of candidate tags whose occurrence times satisfy a first preset condition is calculated to obtain a first number of classification tags for the text to be classified.

5. The method according to claim 1, characterized in that The method comprises: Defining data fields of the training set, and defining corresponding data types based on the data fields; Creating a plurality of formatted training texts based on the data field, the data type and each of the training texts; Based on each formatted training text, the target ES index corresponding to the data set identifier is created.

6. The method according to claim 1, characterized in that The step of inputting the to-be-classified text and the candidate text list into a first classification module, a second classification module, and a third classification module, respectively, to obtain a first number of first labels, a first number of second labels, and a first number of third labels, respectively, includes: Inputting the text to be classified and each candidate text into the first classification module, so that the first classification module compresses the text to be classified and each candidate text respectively, obtains the length of the compressed text to be classified and the length of each compressed candidate text respectively, and calculates the distance metric between the text to be classified and each candidate text based on the length of the compressed text to be classified and the length of each compressed candidate text, and obtains a first number of first labels based on the candidate labels corresponding to the first number of candidate texts that meet the second preset condition based on the distance metric; The text to be classified and the candidate text list are respectively input into a second classification module and a third classification module to obtain a first number of second labels and a first number of third labels respectively.

7. The method according to claim 1, characterized in that The step of inputting the to-be-classified text and the candidate text list into a first classification module, a second classification module, and a third classification module, respectively, to obtain a first number of first labels, a first number of second labels, and a first number of third labels, respectively, includes: Input the text to be classified and each candidate text into the second classification module, so that the second classification module constructs a vocabulary of the text to be classified and a vocabulary of each candidate text, determines a TF-IDF vector of the text to be classified based on the vocabulary of the text to be classified, determines a TF-IDF matrix corresponding to a plurality of candidate texts based on the vocabulary of each candidate text, calculates cosine similarity between the text to be classified and each candidate text based on the TF-IDF vector of the text to be classified and the TF-IDF matrix, and obtains a first number of second labels based on candidate labels corresponding to a first number of candidate texts whose cosine similarity satisfies a third preset condition; The text to be classified and each candidate text are input into a first classification module and a third classification module respectively, to obtain a first number of first labels and a first number of third labels respectively.

8. The method according to claim 1, characterized in that The step of inputting the to-be-classified text and the candidate text list into a first classification module, a second classification module, and a third classification module, respectively, to obtain a first number of first labels, a first number of second labels, and a first number of third labels, respectively, includes: Input the text to be classified and each candidate text into the third classification module, so that the third classification module constructs a vocabulary of the text to be classified and a vocabulary of each candidate text, determines the number of intersection words between the text to be classified and each candidate text and the number of union words between the text to be classified and each candidate text based on the vocabulary of the text to be classified and the vocabulary of each candidate text, calculates the Jaccard similarity coefficient between the text to be classified and each candidate text based on the number of intersection words and the number of union words, and obtains a first number of third labels based on the candidate labels corresponding to the first number of candidate texts whose Jaccard similarity coefficients meet the fourth preset condition; The text to be classified and each candidate text are input into a first classification module and a second classification module respectively, and a first number of first labels and a first number of second labels are obtained respectively.

9. An unsupervised non-parametric text classification device, characterized in that: The device comprises: An acquisition module is used to acquire a training set and a test set; the training set includes a plurality of training texts respectively corresponding to training labels; and a text to be classified is determined from a plurality of test texts included in the test set; An ES retrieval module, configured to retrieve a candidate text list in the training set based on the text to be classified and a target ES index; the target ES index corresponds to the training set; A classification module, used for inputting the text to be classified and the candidate text list into a first classification module, a second classification module and a third classification module respectively, to obtain a first number of first labels, a first number of second labels and a first number of third labels respectively; the first classification module, the second classification module and the third classification module respectively correspond to different unsupervised non-parametric text classification algorithms; The voting module is used to calculate a first number of classification labels for the text to be classified based on a first number of the first labels, a first number of the second labels, and a first number of the third labels.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images