Text classification, clustering and retrieval method and device and computer equipment

Abstract

The invention relates to a text method, a text classification method, a text clustering method, a text retrieval method, a text classification device, a text retrieval device and computer equipment. The method comprises the steps of obtaining a to-be-classified text, and determining a first code length of a variable-length code of the to-be-classified text; respectively obtaining reference texts from a reference text set, and respectively splicing the reference texts with the to-be-classified texts to obtain first spliced texts; obtaining a second code length of the variable-length code of the reference text and a third code length of the variable-length code of the first spliced text; and based on the first coding length, the second coding length and the third coding length, obtaining a target compression distance between the to-be-classified text and a reference text. And determining a target number of reference texts similar to the to-be-classified text from the reference text set based on a target compression distance, and determining the category of the to-be-classified text based on the category of the target number of reference texts. The text classification efficiency is improved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a text classification, clustering and retrieval method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] Text classification is a basic task in the field of Natural Language Processing (NLP). Text classification is usually achieved by using traditional machine learning-based classification methods or deep learning-based classification methods. Among them, the traditional machine learning classification method includes: extracting features, and inputting the extracted features into the classifier to obtain classification results. The feature representation of this method is relatively sparse, the feature expression ability is very weak, and the classification effect is poor. The classification method based on deep learning includes: extracting vectors, and automatically extracting features and classifying text using the extracted vectors through convolutional neural networks or recurrent neural networks or their variants to obtain classification results. The classification effect of this method has been improved, but it needs to be trained or fine-tuned on labeled samples, which requires a lot of time and cost, and cannot be directly used for online business. Summary of the invention

[0003] Based on this, it is necessary to provide a text classification, clustering and retrieval method, apparatus, computer equipment, storage medium and computer program product that can improve processing efficiency in response to the above technical problems.

[0004] In a first aspect, the present application provides a text classification method, the method comprising:

[0005] Acquire a text to be classified, and determine a first encoding length of a variable-length encoding of the text to be classified;

[0006] Obtain reference texts from the reference text set respectively, and concatenate the reference texts with the text to be classified respectively to obtain first concatenated texts;

[0007] Obtaining a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text;

[0008] Based on the first coding length, the second coding length and the third coding length, a similarity distance between the text to be classified and a reference text is obtained; based on the first coding length and the second coding length, a reference distance is obtained; a ratio calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text, and the target compression distance is used to characterize the similarity between the text to be classified and the reference text;

[0009] A target number of reference texts similar to the text to be classified is determined from the reference text set based on the target compression distance, and a category of the text to be classified is determined based on the categories of the target number of reference texts.

[0010] In a second aspect, the present application further provides a text clustering method, the method comprising:

[0011] Acquire the text to be clustered, and determine a first encoding length of the variable-length encoding of the text to be clustered;

[0012] Acquire reference text from a preset reference text set, and concatenate the reference text with the text to be clustered to obtain a first concatenated text;

[0013] Obtaining a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text;

[0014] Based on the first coding length, the second coding length and the third coding length, a similarity distance between the text to be clustered and the reference text is obtained; based on the first coding length and the second coding length, a reference distance is obtained; a ratio calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the clustered text and the reference text;

[0015] A target number of reference texts similar to the text to be clustered is determined from the reference text set based on the target compression distance, and the cluster where the text to be clustered is located is determined based on the clusters where the target number of reference texts are located.

[0016] In a third aspect, the present application also provides a text retrieval method, the method comprising:

[0017] Acquire a text to be retrieved, and determine a first encoding length of a variable-length encoding of the text to be retrieved;

[0018] Acquire a reference text from a preset reference text set, and concatenate the reference text with the text to be searched to obtain a first concatenated text;

[0019] Obtaining a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text;

[0020] Based on the first coding length, the second coding length and the third coding length, a similarity distance between the text to be retrieved and the reference text is obtained; based on the first coding length and the second coding length, a reference distance is obtained; a ratio calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the text to be retrieved and the reference text;

[0021] Based on the target compression distance, a target number of reference texts similar to the text to be classified are determined from the reference text set as target texts corresponding to the text to be retrieved.

[0022] In a fourth aspect, the present application further provides a text classification device, the device comprising:

[0023] A first acquisition module, used for acquiring a text to be classified and determining a first encoding length of a variable-length encoding of the text to be classified;

[0024] A splicing module, used to obtain reference texts from the reference text set, respectively, and splice the reference texts with the text to be classified to obtain a first spliced ​​text;

[0025] A second acquisition module, used for acquiring a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text;

[0026] A generation module, configured to obtain a similarity distance between the text to be classified and a reference text based on the first encoding length, the second encoding length, and the third encoding length; obtain a reference distance based on the first encoding length and the second encoding length; perform a proportional calculation on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text, wherein the target compression distance is used to characterize the similarity between the text to be classified and the reference text;

[0027] The first determination module is used to determine a target number of reference texts similar to the text to be classified from the reference text set based on a target compression distance, and determine the category of the text to be classified based on the categories of the target number of reference texts.

[0028] In one embodiment, the first concatenated text includes a first concatenated subtext and a second concatenated subtext, and the concatenating module includes:

[0029] A first concatenation submodule is used to concatenate the reference text and the text to be classified in a first order to obtain a first concatenated subtext; in the first concatenated subtext, the reference text precedes the text to be classified;

[0030] A second concatenation submodule is used to concatenate the reference text and the text to be classified in a second order to obtain a second concatenated subtext; in the second concatenated subtext, the text to be classified precedes the reference text;

[0031] The generation module comprises:

[0032] A first acquisition submodule, used for acquiring a first code sublength of the variable length code of the first concatenated subtext and a second code sublength of the variable length code of the second concatenated subtext;

[0033] A first generating submodule, used for obtaining a similarity distance between the text to be classified and a reference text based on the first encoding sublength and the second encoding sublength;

[0034] A second generating submodule, configured to obtain a reference distance based on the first coding length and the second coding length;

[0035] The third generation submodule is used to calculate the ratio of the similarity distance and the reference distance to obtain the target compression distance between the text to be classified and the reference text.

[0036] In one embodiment, the first generation submodule includes:

[0037] A first generating unit, configured to obtain a similarity distance between the text to be classified and a reference text based on a maximum value of the first encoding sub-length and the second encoding sub-length;

[0038] The second generation submodule includes:

[0039] The second generating unit is configured to obtain a reference distance based on a maximum value of the first coding length and the second coding length.

[0040] In one embodiment, the generating module comprises:

[0041] A comparison submodule, configured to obtain a first component based on a minimum value between the first encoding length and the second encoding length;

[0042] A fourth generating submodule, configured to obtain a second component based on the third coding length;

[0043] A fifth generating submodule, configured to obtain a similarity distance between the text to be classified and a reference text based on a difference between the first component and the second component;

[0044] a sixth generating submodule, configured to obtain a reference distance based on a maximum value of the first coding length and the second coding length;

[0045] The seventh generation submodule is used to perform a ratio calculation on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text.

[0046] In one embodiment, the target text includes at least one of a text to be classified, a reference text, a first concatenated text, and a second concatenated text, and the text classification device further includes:

[0047] The first word segmentation module is used to obtain a target text, perform word segmentation processing on the target text, and obtain each character corresponding to the target text;

[0048] A second determination module is used to determine the Huffman code corresponding to each of the characters based on a preset association relationship between the characters and the Huffman code;

[0049] The third determination module is used to determine that the encoding length of the Huffman code of the target text is the sum of the encoding lengths of the Huffman codes corresponding to the characters.

[0050] In one embodiment, the text classification device further includes:

[0051] A second word segmentation module is used to perform word segmentation processing on the reference text in the reference text set to obtain each character corresponding to the reference text set;

[0052] A statistical module, used for counting the frequency of each character in each character corresponding to the reference text set appearing in the reference text set;

[0053] The fourth determination module is used to construct a Huffman tree based on the characters corresponding to the reference text set and the frequencies corresponding to each of the characters, and determine the Huffman code corresponding to each of the characters based on the Huffman tree.

[0054] In one embodiment, the splicing module includes:

[0055] The second acquisition submodule is used to obtain the target compression distance between the text to be classified and each cluster center in the reference text set; wherein the cluster center is the target reference text in the cluster obtained by clustering the reference texts in the reference text set in advance;

[0056] A first determination submodule is used to determine, from each of the cluster centers, a target cluster center that is most similar to the text to be classified based on the target compression distance;

[0057] The third acquisition submodule is used to respectively acquire reference texts from the target clusters corresponding to the target cluster centers.

[0058] In one embodiment, the second acquisition submodule is used to

[0059] The first acquisition unit is used to acquire the target compression distance between the text to be classified and each cluster center in the reference text set, including:

[0060] A concatenation unit, used for concatenating each of the cluster centers with the text to be classified to obtain a second concatenated text;

[0061] A second acquisition unit is used to acquire a fourth code length of the variable length code of the cluster center and a fifth code length of the variable length code of the second concatenated text;

[0062] The third generating unit is used to obtain the similarity distance between the text to be classified and the cluster center based on the first coding length, the fourth coding length and the fifth coding length; obtain the reference distance based on the first coding length and the fourth coding length; perform proportional calculation on the similarity distance between the text to be classified and the cluster center and the reference distance to obtain the target compression distance between the text to be classified and the cluster center.

[0063] In one embodiment, the third acquisition submodule includes:

[0064] A third acquisition unit is used to acquire a target compression distance between the text to be classified and the center of each sub-cluster in the target cluster;

[0065] A determination unit, configured to determine, from the centers of the subclusters, the center of the target subcluster closest to the text to be classified based on the target compression distance;

[0066] The fourth acquisition unit is used to respectively acquire reference texts from the target sub-clusters corresponding to the centers of the target sub-clusters.

[0067] In one embodiment, the first determining module includes:

[0068] A construction submodule, used for traversing a target number of reference texts from the reference text set, taking a target compression distance between the text to be classified and each reference text in the target number of reference texts as a node of a small root heap, and constructing the small root heap;

[0069] A traversal submodule, configured to continue traversing reference texts from the remaining reference texts in the reference text set except the target number of reference texts, and insert the target compression distance corresponding to the traversed reference text into the small root heap when the target compression distance of the traversed reference text is greater than the target compression distance of the root node of the small root heap;

[0070] A sorting submodule is used to sort the nodes in the inserted small root heap to obtain an updated small root heap, until all the remaining reference texts are traversed to obtain the last updated small root heap;

[0071] The eighth generation submodule is used to obtain a target number of reference texts with the largest target compression distance based on the reference texts corresponding to the last updated small root heap.

[0072] In one embodiment, the eighth generation submodule includes:

[0073] A sorting unit, used for sorting the nodes in the last updated small root heap to obtain a large root heap;

[0074] A deletion unit, used for storing the root node in the big root heap, and deleting the root node from the big root heap to obtain the remaining nodes;

[0075] A storage unit, configured to construct an updated big root heap based on the remaining nodes, store the root nodes of the updated big root heap, and delete the corresponding root nodes from the updated big root heap until the number of stored root nodes reaches the target number;

[0076] The fourth generating unit is used to sequentially obtain the reference text corresponding to each root node from the stored root nodes to obtain the arranged reference text.

[0077] In one embodiment, the determining module includes:

[0078] A fourth acquisition submodule is used to acquire the number of reference texts corresponding to each category based on the categories corresponding to the target number of reference texts;

[0079] The second determination submodule is used to take the category where the number of reference texts is the largest as the category of the text to be classified.

[0080] In one embodiment, the determining module includes:

[0081] A fifth acquisition submodule is used to acquire the categories corresponding to the target number of reference texts and the weights of the reference texts; wherein the weight of the reference text is the inverse of the compression distance between the reference text and the target text to be classified;

[0082] The third determination submodule is used to obtain a hit score for each category based on the categories corresponding to the target number of reference texts and the weights of the reference texts, and to take the category with the largest hit score as the category of the text to be classified.

[0083] In a fifth aspect, the present application further provides a text clustering device, the device comprising:

[0084] A first acquisition module, used for acquiring the text to be clustered, and determining a first encoding length of the variable-length encoding of the text to be clustered;

[0085] A splicing module, used for obtaining a reference text from a preset reference text set, and splicing the reference text with the text to be clustered to obtain a first spliced ​​text;

[0086] A second acquisition module, used for acquiring a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text;

[0087] A generation module, configured to obtain a similarity distance between the text to be clustered and a reference text based on the first coding length, the second coding length, and the third coding length; obtain a reference distance based on the first coding length and the second coding length; and perform a proportional calculation on the similarity distance and the reference distance to obtain a target compression distance between the clustered text and the reference text;

[0088] The determination module is used to determine a target number of reference texts similar to the text to be clustered from the reference text set based on a target compression distance, and determine the cluster where the text to be clustered is located based on the clusters where the target number of reference texts are located.

[0089] In a sixth aspect, the present application further provides a text retrieval device, the device comprising:

[0090] A first acquisition module, used for acquiring a text to be retrieved, and determining a first encoding length of a variable-length encoding of the text to be retrieved;

[0091] A splicing module, used for obtaining a reference text from a preset reference text set, and splicing the reference text with the text to be searched to obtain a first spliced ​​text;

[0092] A second acquisition module, used for acquiring a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text;

[0093] A generating module, configured to obtain a similarity distance between the text to be retrieved and the reference text based on the first encoding length, the second encoding length, and the third encoding length; obtain a reference distance based on the first encoding length and the second encoding length; and perform a proportional calculation on the similarity distance and the reference distance to obtain a target compression distance between the text to be retrieved and the reference text;

[0094] The determination module is used to determine a target number of reference texts similar to the text to be retrieved from the reference text set based on the target compression distance as target texts corresponding to the text to be retrieved.

[0095] In a seventh aspect, the present application further provides a computer device. The computer device comprises a memory and a processor, the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0096] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0097] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0098] The above-mentioned text classification method, device, computer equipment, storage medium and computer program product determine the similarity distance between the text to be classified and the reference text through the first code length of the variable length code of the text to be classified, the second code length of the variable length code of the reference text, and the third code length of the variable length code of the first concatenated text of the text to be classified and the reference text. The similarity distance can characterize the similarity between the text to be classified and the reference text, that is, two texts with greater similarity have closer codes and code lengths than other texts of different categories. Further, the embodiment of the present disclosure obtains the reference distance based on the first code length and the second code length, wherein the role of the reference distance is to limit the value of the target compression distance within a preset range, that is, normalization processing, so that the target compression distance has a normalized property. Thereby solving the comparability of data of different dimensions and more conveniently calculating the size of each target compression distance. Finally, the embodiment of the present disclosure determines the target number of reference texts similar to the text to be classified from the reference text set based on the target compression distance, and determines the category of the text to be classified based on the category of the target number of reference texts. In this way, the embodiment of the present disclosure does not need to construct a complex initial classification model, and when a new category is added, there is no need to train and update the initial classification model. Therefore, the embodiment of the present disclosure improves the efficiency of text classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 A schematic diagram of a traditional machine learning classification method in one embodiment;

[0100] Figure 2 A schematic diagram of a deep learning classification method in one embodiment;

[0101] Figure 3 An application environment diagram of a text classification method in an embodiment;

[0102] Figure 4 A flowchart of a text classification method in one embodiment;

[0103] Figure 5 A schematic diagram of a text splicing process in an embodiment;

[0104] Figure 6 A schematic diagram of a flow chart of calculating a target compression distance between a text to be classified and a reference text in one embodiment;

[0105] Figure 7A schematic diagram of a Huffman encoding process in one embodiment;

[0106] Figure 8 A schematic diagram of a result of Huffman encoding in one embodiment;

[0107] Fig. 9 A schematic diagram of a flow chart of obtaining reference text in one embodiment;

[0108] Fig.10 A schematic diagram of a process for calculating a target compression distance between a text to be classified and a cluster center in one embodiment;

[0109] Fig.11 A method for obtaining a target number of reference texts in an embodiment;

[0110] Fig.12 A schematic diagram of a small root pile in one embodiment;

[0111] Fig.13 A schematic diagram of a first processing process of a small root pile in one embodiment;

[0112] Fig.14 is a schematic diagram of a second processing process of a small root pile in one embodiment;

[0113] Fig.15 is a schematic diagram of a third processing process of a small root pile in one embodiment;

[0114] Fig.16 is a schematic diagram of a fourth processing process of a small root pile in one embodiment;

[0115] Fig.17 is a schematic diagram of a fifth processing process of a small root pile in one embodiment;

[0116] Fig.18 is a schematic diagram of the sixth processing process of a small root pile in one embodiment;

[0117] Fig.19 is a schematic diagram of the seventh processing process of a small root pile in one embodiment;

[0118] Fig. 20 A schematic diagram of a process of a text clustering method in one embodiment;

[0119] Fig.21 A flowchart of a text retrieval method in one embodiment;

[0120] Fig. 22 is an architecture diagram of a text classification method in one embodiment;

[0121] Fig.23 is a structural block diagram of a text classification device in one embodiment;

[0122] Fig.24 is a structural block diagram of a text clustering device in one embodiment;

[0123] Fig.25 is a structural block diagram of a text retrieval device in one embodiment;

[0124] Fig.26 is an internal structure diagram of a computer device in one embodiment;

[0125] Fig. 27 FIG. 4 is a diagram showing the internal structure of a computer device in another embodiment. DETAILED DESCRIPTION

[0126] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0127] In order to facilitate those skilled in the art to understand the technical solution provided by the embodiments of the present disclosure, the technical environment in which the technical solution is implemented is described below.

[0128] Traditional machine learning classification methods, see Figure 1 As shown, Figure 1 Feature engineering in .txt includes text preprocessing, feature extraction, and text representation. Feature extraction mainly includes One-hot encoding, TF-IDF (term frequency–inverse document frequency) algorithm, bag-of-words model, etc., which converts text into a computer-understandable format, inputs the extracted features into the classifier for training, and obtains a usable text classification model. Common classifiers include Naive Bayes, K-nearest neighbor, support vector machine, logistic regression, etc.

[0129] In traditional machine learning classification methods, the representation of text features is high-dimensional, sparse, and has semantic gaps. The feature expression ability is very weak and is not conducive to model building. The classification method based on deep learning expresses each word as a multi-dimensional dense, continuous real number vector through the distributed representation of text, which becomes a word vector. This representation method has evolved from early static representations such as word2vec (word vector model) and fasttext (fast text classification algorithm) to BERT (Bidirectional Encoder Representations from Transformers). Based on word representation, the classification method based on deep learning uses convolutional neural networks or recurrent neural networks or their variants to realize automatic feature extraction. Reference Figure 2As shown in the figure, taking BERT as an example, BERT, as the epitome of dynamic word embedding identification learning, is a bidirectional fusion feature model with only an encoder but no decoder. BERT proposes two tasks: occluding language model and predicting the next sentence. Regarding the occluding language model, it specifically includes randomly occluding a certain proportion of words, using the model to predict the word, and training a deep bidirectional word embedding vector identification. Regarding predicting the next sentence, it specifically includes connecting the sequence of two sentences in parallel as the input part of the model, adding a segmentation symbol in the middle, and adding symbol marks to the beginning and end of the sentence. It can learn the relationship between sentences and capture sentence-level identification. By fine-tuning the model pre-trained on a large-scale corpus, the modeling of downstream text tasks can be completed.

[0130] Traditional machine learning classification methods usually have poor classification results, and feature extraction and construction are relatively complex, requiring more manual input. Classification methods based on deep learning need to be trained or fine-tuned on labeled samples. The training process requires a lot of computing resources and time costs, and cannot be directly used for online business. The above two models require retraining of the basic model and the newly added categories, and have poor scalability.

[0131] Based on actual technical needs similar to those described above, the present application proposes a text classification method, apparatus, computer device, storage medium and computer program product.

[0132] The text classification method provided in the embodiment of the present application can be applied to Figure 3 In the application environment shown. Among them, the terminal 302 communicates with the server 304 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 304, or it can be placed on the cloud or other servers. Among them, the terminal 302 can be but is not limited to various desktop computers, laptops, smart phones, tablets, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The server 304 can be implemented with an independent server or a server cluster consisting of multiple servers.

[0133] Both the terminal and the server can be used independently to execute the text classification method, text clustering method and text retrieval method provided in the embodiments of the present application.

[0134] Taking the text classification method as an example, the terminal can send the text to be classified to the server, or the server can directly obtain the text to be classified. The server determines the first code length of the variable length code of the text to be classified. For example, the text to be classified is encoded, and the number of characters in the variable length code is calculated. In an exemplary embodiment, a reference text set can be stored in a data storage system, and the server obtains reference texts from the reference text set respectively. The server splices the obtained reference text with the text to be classified to obtain a first spliced ​​text. The server further obtains the second code length of the variable length code of the reference text and the third code length of the variable length code of the first spliced ​​text. In an exemplary embodiment, because the reference text is pre-saved in the reference text set, the second code length corresponding to each reference text can be pre-calculated.

[0135] The server obtains the similarity distance between the text to be classified and the reference text based on the first coding length, the second coding length and the third coding length. The server obtains the reference distance based on the first coding length and the second coding length. A proportional calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text. The target compression distance is used to characterize the distance between the text to be classified and the reference text. Based on the target compression distance, a target number of reference texts similar to the text to be classified are determined from the text set, and based on the categories of the target number of reference texts, the category of the text to be classified is determined, for example, the text to be classified is determined as a sentiment analysis category, a news category, or a spam filtering category, etc.

[0136] In one embodiment, Figure 4 As shown, a text classification method is provided, and the method is applied to a computer device as an example. The computer device may include a terminal or a server. The method may be executed by the terminal or the server itself, or may be implemented through interaction between the terminal and the server. Figure 4 As shown, the text classification method includes the following steps:

[0137] Step S401, obtaining a text to be classified, and determining a first code length of a variable-length code of the text to be classified.

[0138] Among them, the text to be classified may include text to be classified. In a specific implementation, the text to be classified may be text in the public account of the application, text in a post bar, a forum, text in a search engine, etc. In an exemplary embodiment, the text to be retrieved in the above scenario may be text that has not been converted in format, such as an article in a public account, and the present application exists in the form of text. In another exemplary embodiment, the text to be retrieved may also be text that has been converted in format. For example, the classification of e-commerce products, after expressing the product object in the form of text, realizes the classification of e-commerce products by classifying the text to be classified. Optionally, in the case where the object to be classified involves pictures, videos or audio, the above-mentioned object to be classified can be converted into text, for example, the above-mentioned conversion can be realized by a pre-trained artificial intelligence model, and the present disclosure does not limit this. In addition, the above-mentioned objects to be classified involving pictures, videos or audio can also be directly classified, and it is only necessary to select a compression code that matches the format of the object to be classified so that similar objects have similar codes and code lengths. It is understandable that technical personnel in the relevant field may make other changes based on the technical essence of this application, but as long as the functions and effects achieved are the same or similar to those of this application, they should be included in the scope of protection of this application.

[0139] Among them, variable length coding is a coding method, different symbols or data elements are represented by sequences of different lengths. Compared with fixed length coding, variable length coding can achieve higher compression efficiency. In the specific implementation, variable length coding can be selected, such as Huffman coding (Huffman), LZ coding (Lempel-Ziv), BWT (Burrows-Wheeler Transform) and other data compression coding. Among them, the coding length of variable length coding refers to the number of bits of coding symbols after the text is converted into variable length coding. For example, the variable length code of text B is 10010101, and the length of the variable length code of text B is 8 in this example. In the embodiment of the present disclosure, the first coding length of the text to be classified can be obtained by encoding the text to be classified, obtaining the variable length code corresponding to the text to be classified, and calculating the number of bits of the coding symbols in the variable length code.

[0140] Step S403, respectively obtaining reference texts from the reference text set, and respectively concatenating the reference texts with the text to be classified to obtain a first concatenated text.

[0141] Among them, the reference text set includes reference texts of multiple categories, for example, the reference text set is constructed in the form of content-label. In the process of constructing the reference text set, the reference texts of the corresponding categories can be widely collected according to the specific application scenario, and the categories of the reference texts can be annotated in combination with the specific application scenario. In an exemplary embodiment, the reference texts of various categories can be collected for the application scenario of text classification. For example, when the resource recommendation category is annotated, the text belonging to the resource recommendation category can be annotated, such as the resource recommendation category; the text not belonging to the resource recommendation category can also be annotated, such as not belonging to the resource recommendation category. Optionally, the text of the resource recommendation category can be further subdivided into categories, for example, divided into: resource recommendation category of external link mode, resource recommendation category of telephone mode, resource recommendation category of instant messaging mode, etc. Of course, the divided categories can be further subdivided, and the present disclosure does not limit this. In another exemplary embodiment, texts other than the preset categories can also be collected, which can be annotated as "other categories" and added to the reference text set. In one embodiment, the collected reference texts are preprocessed, wherein the preprocessing methods include but are not limited to removing duplicates, punctuation marks, spaces, meaningless characters such as \n, \t, \r, and rare characters, etc., so as to facilitate compression coding. Optionally, the reference text collection can be scalable to continuously adapt to new categories of text to be classified.

[0142] Specifically, the reference text is spliced ​​with the text to be classified to obtain a first spliced ​​text. In an exemplary embodiment, the reference text can be spliced ​​after the text to be classified. For example, the reference text content includes: A1A2A3A4A5A6, and the text to be classified content includes: B1B2B3B4B5B6, then the first spliced ​​text can include: B1B2B3B4B5B6 A1A2A3A4A5A6. In another exemplary embodiment, the reference text can be spliced ​​before the text to be classified. For example, the reference text content includes: A1A2A3A4A5A6, and the text to be classified content includes: B1B2B3B4B5B6, then the first spliced ​​text can include: A1A2A3A4A5A6B1B2B3B4B5B6.

[0143] Step S405, obtaining the second code length of the variable length code of the reference text and the third code length of the variable length code of the first concatenated text.

[0144] In one embodiment, similar to the method of determining the first code length of the variable length code of the text to be classified in the above-mentioned embodiment, the second code length of the variable length code of the reference text in the embodiment of the present disclosure and the third code length of the variable length code of the first concatenated text are determined. Optionally, the second code length of the variable length code of the reference text can be pre-calculated and stored, and directly obtained from the storage area when used, thereby improving classification efficiency. For example, a key-value pair data structure is used for storage, wherein the key can be used to store the identification information of the reference text, and the value can be used to store the second code length corresponding to the reference text.

[0145] Step S407, based on the first coding length, the second coding length and the third coding length, obtain the similarity distance between the text to be classified and the reference text; based on the first coding length and the second coding length, obtain the reference distance; perform proportional calculation on the similarity distance and the reference distance to obtain the target compression distance between the text to be classified and the reference text, and the target compression distance is used to characterize the similarity between the text to be classified and the reference text.

[0146] Specifically, based on the first coding length, the second coding length and the third coding length, the method for obtaining the similarity distance between the text to be classified and the reference text may include performing mathematical operations on the first coding length, the second coding length and the third coding length. For example: based on the different splicing methods of the reference text and the text to be classified, the value of the third coding length obtained may be different. According to the different values ​​of the third coding length, the similarity distance between the text to be classified and the reference text is determined. Optionally, the largest value is selected from the different values ​​corresponding to the third coding length as the similarity distance between the text to be classified and the reference text. For another example: based on the minimum value of the first coding length and the second coding length, the first component is obtained; based on the third coding length, the second component is obtained; based on the difference between the first component and the second component, the similarity distance between the text to be classified and the reference text is obtained. In the disclosed embodiment, the similarity between the text to be classified and the reference text is characterized by the similarity distance, and the text to be classified, the reference text and the first concatenated text are introduced. The encoding length of each text can quantitatively characterize the Kolmogorov complexity of converting the text to be classified into the reference text. The lower the Kolmogorov complexity, the more similar the text to be classified is to the reference text, and the higher the complexity, the less similar the text to be classified is to the reference text. Among them, the Kolmogorov complexity is used to characterize the shortest binary program length required to generate text x, denoted as K(x), which is the theoretical lower limit of this information measurement. In order to measure the information content shared between two objects, the information distance E(x,y) is defined as the minimum binary length required to generate entity y from entity x. As shown in the following formula:

[0147] E(x, y) = max{K(x|y), K(y|x)} = K(xy) - min{K(x), K(y)} (1)

[0148] Among them, K(x) represents the Kolmogorov complexity, K(y|x) represents the Kolmogorov complexity of generating y from x, and K(xy) represents the Kolmogorov complexity of concatenating x and y as a new object.

[0149] In the disclosed embodiment, obtaining the reference distance based on the first coding length and the second coding length may include performing mathematical operations on the first coding length and the second coding length. For example, the maximum value of the first coding length and the second coding length is taken as the reference distance. For another example, the sum of the first coding length and the second coding length is taken as the reference distance. Among them, the role of the reference distance is to limit the value of the target compression distance within a preset range, that is, normalization processing, so that a proportional calculation is performed for the similarity distance and the reference distance, and the target compression distance between the text to be classified and the reference text has a normalized property. Thereby solving the comparability of data of different dimensions and more conveniently calculating the size of each target compression distance.

[0150] In the disclosed embodiment, a ratio calculation is performed for the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text. For example, the target compression distance between the text to be classified and the reference text is obtained by dividing the similarity distance by the reference distance. Optionally, a constant may be added or subtracted from the similarity distance by the reference distance.

[0151] Step S409, determining a target number of reference texts similar to the text to be classified from the reference text set based on the target compression distance, and determining the category of the text to be classified based on the categories of the target number of reference texts.

[0152] Among them, the number of target quantities may include one or two or more. In an exemplary embodiment, when the target quantity is one, according to a specific expression of a specific target compression distance, a reference text with the smallest target compression distance with the text to be classified can be selected from the reference text set, and the category corresponding to the smallest reference text is used as the category of the text to be classified; or a reference text with the largest target compression distance with the text to be classified is selected, and the largest reference text is used as the category of the text to be classified.

[0153] In another exemplary embodiment, when the target number is two or more, the category of the text to be classified can be determined based on the categories of two or more reference texts. For example: based on the categories corresponding to two or more reference texts, the number of reference texts corresponding to each category is obtained; the category with the largest number of reference texts is used as the category of the text to be classified. Another example: the categories corresponding to two or more reference texts and the weights of the reference texts are obtained; wherein the weight of the reference text is the inverse of the compression distance between the reference text and the target text to be classified; based on the categories corresponding to the target number of reference texts and the weights of the reference texts, the hit score of each category is obtained, and the category with the largest hit score is used as the category of the text to be classified.

[0154] The above-mentioned text classification method determines the similarity distance between the text to be classified and the reference text through the first code length of the variable length code of the text to be classified, the second code length of the variable length code of the reference text, and the third code length of the variable length code of the first concatenated text of the text to be classified and the reference text. The similarity distance can characterize the similarity between the text to be classified and the reference text, that is, two texts with greater similarity have closer codes and code lengths than other texts of different categories. Further, the embodiment of the present disclosure obtains the reference distance based on the first code length and the second code length, wherein the role of the reference distance is to limit the value of the target compression distance within a preset range, that is, normalization processing, so that the target compression distance has a normalized property. Thereby solving the comparability of data of different dimensions and more conveniently calculating the size of each target compression distance. Finally, the embodiment of the present disclosure determines the target number of reference texts similar to the text to be classified from the reference text set based on the target compression distance, and determines the category of the text to be classified based on the category of the target number of reference texts. In this way, the embodiment of the present disclosure does not need to construct a complex initial classification model, and when a new category is added, there is no need to train and update the initial classification model. Therefore, the embodiment of the present disclosure improves the efficiency of text classification.

[0155] In one embodiment, reference Figure 5 As shown, the first concatenated text includes a first concatenated sub-text and a second concatenated sub-text, and the reference text and the text to be classified are concatenated to obtain the concatenated text, including steps S501 and S503, wherein:

[0156] Step S501 , respectively concatenating the reference text and the text to be classified in a first order to obtain a first concatenated sub-text; in the first concatenated text, the reference text precedes the text to be classified.

[0157] Step S503, concatenating the reference text and the text to be classified in a second order to obtain a second concatenated sub-text; in the second concatenated text, the text to be classified precedes the reference text.

[0158] Among them, in the first spliced ​​subtext, the reference text precedes the text to be classified, for example, the reference text content includes: A1A2A3A4A5A6, and the text to be classified content includes: B1B2B3B4B5B6, then the first spliced ​​subtext may include: B1B2B3B4B5B6 A1A2A3A4A5A6. In the second spliced ​​subtext, the text to be classified precedes the reference text, for example, the reference text content includes: A1A2A3A4A5A6, and the text to be classified content includes: B1B2B3B4B5B6, then the second spliced ​​subtext may include: A1A2A3A4A5A6B1B2B3B4B5B6.

[0159] In one embodiment, reference Figure 6 As shown, the similarity distance between the text to be classified and the reference text is obtained based on the first code length, the second code length and the third code length; the reference distance is obtained based on the first code length and the second code length; the target compression distance between the text to be classified and the reference text is obtained by performing proportional calculation on the similarity distance and the reference distance, including step S601, step S603, step S605 and step S607. Step S601, obtaining the first code sub-length of the variable length code of the first concatenated sub-text and the second code sub-length of the variable length code of the second concatenated sub-text.

[0160] The method of obtaining the first code sub-length of the variable length code of the first spliced ​​sub-text and the second code sub-length of the second spliced ​​sub-text in the embodiment of the present disclosure may be the same as the method of obtaining the first code length of the text to be classified in the above embodiment, and the present disclosure will not elaborate on this.

[0161] Step S603: obtaining a similarity distance between the text to be classified and the reference text based on the first codifier length and the second codifier length.

[0162] In a specific implementation, based on the first codifier length and the second codifier length, the method of obtaining the similarity distance between the text to be classified and the reference text may include performing a mathematical operation on the first codifier length and the second codifier length. For example, the maximum value is selected from the first codifier length and the second codifier length as the similarity distance between the text to be classified and the reference text.

[0163] Step S605: obtaining a reference distance based on the first coding length and the second coding length.

[0164] Step S607: performing a ratio calculation on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text.

[0165] In a specific implementation, obtaining the reference distance based on the first coding length and the second coding length may include performing a mathematical operation on the first coding length and the second coding length. For example, taking the maximum value of the first coding length and the second coding length as the reference distance. For another example, taking the sum of the first coding length and the second coding length as the reference distance.

[0166] In one embodiment, based on the first code sub-length and the second code sub-length, obtaining the similarity distance between the text to be classified and the reference text; based on the first code length and the second code length, obtaining the reference distance, comprises:

[0167] Based on the maximum value of the first encoding sub-length and the second encoding sub-length, obtaining a similarity distance between the text to be classified and the reference text;

[0168] A reference distance is obtained based on a maximum value of the first code length and the second code length.

[0169] For example: x represents the text to be classified, y represents the reference text, C(yx) represents the first code sub-length, C(xy) represents the second code sub-length, C(x) represents the first code length, and C(y) represents the second code length. Based on the maximum value of the first code sub-length and the second code sub-length, the similarity distance between the text to be classified and the reference text is obtained. For example, the similarity distance can be expressed as max{C(xy), C(yx)}. Based on the maximum value of the first code length and the second code length, the reference distance is obtained. For example, the reference distance can be expressed as max{C(x), C(y)}. In an exemplary embodiment, the target compression distance SNCD(x,y) between the text to be classified and the reference text is expressed as follows:

[0170] SNCD(x,y)=max{C(xy),C(yx)} / max{C(x),C(y)}-1 (2)

[0171] Among them, SNCD (x, y) in a specific variable length code has a value range of ,in Represents the theoretical error of the distance metric operator. When y is empty, SNCD = 0; when y = x, The larger the SNCD value, the closer the sample to be classified is to the reference sample. SNCD (Symmetrical Normalized Compression Distance) can be called the symmetric normalized compression distance.

[0172] In the above embodiment, the first concatenated subtext and the second concatenated subtext are obtained by concatenating the reference text and the text to be classified in different orders. Based on the first coding sub-length corresponding to the first concatenated text and the second coding sub-length corresponding to the second concatenated text, the similarity distance between the text to be classified and the reference text is obtained. Thus, the target compression distance is symmetrical. For example, SNCD(x,y)=SNCD(y,x), which ensures the consistency of the classification task.

[0173] In one embodiment, based on the first coding length, the second coding length and the third coding length, a similarity distance between the text to be classified and a reference text is obtained; based on the first coding length and the second coding length, a reference distance is obtained, and a ratio calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text, including:

[0174] A first component is obtained based on a minimum value of the first code length and the second code length.

[0175] Based on the third code length, a second component is obtained.

[0176] Based on the difference between the first component and the second component, a similarity distance between the text to be classified and the reference text is obtained.

[0177] A reference distance is obtained based on a maximum value of the first code length and the second code length.

[0178] A ratio calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text.

[0179] For example: x represents the text to be classified, y represents the reference text, C(x) represents the first coding length, and C(y) represents the second coding length. C(xy) represents the third coding length of the variable length coding corresponding to the first concatenated text of x and y. Of course, C(yx) can also represent the above third coding length. The first component can be expressed as min{C(x), C(y)}, and the second component can be expressed as C(xy) or C(yx). The similarity distance can be expressed as C(xy)–min{C(x), C(y)}, and the reference distance can be expressed as max{C(x), C(y)}. In an exemplary embodiment, the target compression distance NCD(x,y) between the text to be classified and the reference text is expressed as follows:

[0180] NCD(x,y)=(C(xy)–min{C(x),C(y)}) / max{C(x),C(y)} (3)

[0181] Among them, NCD (x, y) in a specific variable length code has a value range of ,in, Represents the theoretical error of the distance measurement operator. The smaller the value of NCD(x,y), the more similar the text to be classified is to the reference text. Among them, the abbreviation NCD (Normalized Compression Distance) may mean a symmetrical normalized compression distance. It is understandable that, based on the technical essence of the present application, technicians in the relevant field may make other changes to the target compression distance, but as long as the functions and effects achieved are the same or similar to those of the present application, they should be covered within the scope of protection of the present application.

[0182] In one embodiment, the target text includes at least one of a text to be classified, a reference text, a first concatenated text, and a second concatenated text, and the method for determining the encoding length of the variable-length encoding of the target text includes:

[0183] Obtain a target text, perform word segmentation on the target text, and obtain each character corresponding to the target text.

[0184] Based on the preset association relationship between the characters and the Huffman codes, the Huffman codes corresponding to each of the characters are determined.

[0185] The coding length of the Huffman coding of the target text is determined to be the sum of the coding lengths of the Huffman coding corresponding to each of the characters.

[0186] Specifically, the target text includes a text to be classified, a reference text, a first concatenated text, a second concatenated text, and may also include a first concatenated subtext and a second concatenated subtext. Therefore, the above texts can be encoded using the same variable length coding method to obtain the corresponding coding length. Among them, the preset association relationship between characters and Huffman codes can be pre-constructed. In the disclosed embodiment, the target text is segmented. For example, the target text includes ABCDE, and the target text is segmented to obtain the characters of the target text, "A", "B", "C", "D", and "E". According to the preset association relationship between the character and the Huffman code, the Huffman code of the character "A" is, for example, 0; the Huffman code of the character "B" is, for example, 100; the Huffman code of the character "C" is, for example, 101; the Huffman code of the character "D" is, for example, 110; the Huffman code of the character "E" is, for example, 111. It can be understood that the Huffman code of the target text is the sum of the Huffman codes corresponding to the above characters: 0100101110111. Furthermore, the Huffman coding length of the target text is 13 in the above example.

[0187] Huffman Coding is a data compression algorithm, which is a type of variable-length coding. It achieves data compression coding by representing characters with higher frequency of occurrence with shorter codes and characters with lower frequency of occurrence with longer codes.

[0188] In the embodiment of the above-mentioned text classification method, each text is encoded by Huffman coding, the Huffman coding length of each text is determined, and then the target compression distance is constructed based on the Huffman coding length. Since Huffman coding has the property of non-repetitive prefixes, the code corresponding to each character is not a prefix of the code corresponding to other characters, which is conducive to accurately restoring the original data during decoding. The encoding method of Huffman coding enables the Huffman coding length to more recognizablely represent the corresponding text. Therefore, the closer the Huffman coding length of the text to be classified is to the Huffman coding length of the reference text, the more similar the two texts are. Therefore, the disclosed embodiment uses the Huffman coding length to construct the target compression distance, which can improve the accuracy of text classification.

[0189] In one embodiment, the preset association relationship between the character and the Huffman code is obtained in the following manner:

[0190] The reference texts in the reference text set are segmented to obtain characters corresponding to the reference text set.

[0191] The frequency of occurrence of each character in the reference text set among the characters corresponding to the reference text set is counted.

[0192] A Huffman tree is constructed based on the characters corresponding to the reference text set and the frequencies corresponding to each of the characters, and a Huffman code corresponding to each of the characters is determined based on the Huffman tree.

[0193] Specifically, the reference text in the text set is segmented to obtain each character, and the frequency of each character in the text set is counted to obtain a character frequency pair. In an exemplary embodiment, the character frequency pair can be stored in a mapping word table. Sort each character in ascending order according to the size of the frequency; take the two characters with the smallest frequency as two leaf nodes, and combine the two leaf nodes into a new node, and the frequency of the new node is the sum of the frequencies of the two leaf nodes; sort the new node and the remaining characters in ascending order, take the two smallest characters after sorting as two leaf nodes, and combine the two leaf nodes into a new node. Among them, if one of the two smallest characters after sorting is the above-mentioned new node, then the new node and the other character of the two smallest characters after sorting are respectively used as two leaf nodes. Recombine the two leaf nodes into a new node. And so on, until all nodes are merged into a root node.

[0194] For example, refer to Figure 7 As shown, the character frequency pairs include {"A": 15, "B": 7, "C": 6, "D": 6, "E": 5}. In process 701, the characters are arranged in ascending order according to the frequency. In process 503, the two characters D and E with the smallest frequencies are selected as two leaf nodes, and leaf node D and leaf node E are combined into a new node. The frequency of the new node is the sum of the frequencies of leaf node D and leaf node E. The dotted circle in process 703 represents the new node, and the frequency is 11. The new node and the remaining characters A, B, and C are arranged in ascending order. After sorting, the two smallest characters are B and C. In step 705, the two smallest characters B and C are used as two leaf nodes, and leaf node B and leaf node C are combined into a new node. The frequency of the new node is the sum of the frequencies of leaf node B and leaf node C, which is 13. The new node corresponding to the frequency 13, the new node corresponding to the frequency 11, and the remaining character A are sorted in ascending order. The two smallest nodes after sorting are the new node corresponding to the frequency 13 and the new node corresponding to the frequency 11. In process 707, the new node corresponding to the frequency 13 and the new node corresponding to the frequency 11 are combined to obtain a new node with a frequency of 24. Since there is one character A left at this time, in process 709, character A and frequency 24 are combined into a new node with a frequency of 39. The node corresponding to frequency 39 can be called a root node.

[0195] Based on the Huffman tree generated in the above embodiment, refer to Figure 8 As shown, starting from the root node, traverse the Huffman tree, the left branch represents the code as 0, the right branch represents the code as 1, and the code of each character is transferred from the root node to the corresponding leaf node to obtain a unique code. For example, in this embodiment, the Huffman code of character A is represented as 0, the Huffman code of character B is represented as 100, the Huffman code of character C is represented as 101, the Huffman code of character D is represented as 110, and the Huffman code of character E is represented as 111.

[0196] In the above embodiment, the Huffman tree on which the preset association between characters and Huffman codes is built can be built in advance, thereby saving the calculation time of text classification. Further, the reference texts in the reference text set can also be expanded at a predetermined time, and the Huffman tree can be updated after the expansion is completed.

[0197] In one embodiment, reference Fig. 9 As shown, obtaining reference texts from the reference text set respectively includes step S901, step S903, and step S905. Among them:

[0198] Step S901, obtaining a target compression distance between the text to be classified and each cluster center in the reference text set; wherein the cluster center is a target reference text in a cluster obtained by clustering the reference texts in the reference text set in advance.

[0199] Specifically, the reference texts in the reference text set can be clustered to obtain various clusters. Among them, the text clustering methods include but are not limited to hierarchical clustering, partition clustering, density clustering, spectral clustering and model clustering. Taking hierarchical clustering as an example, starting from an initial single cluster, the data is gradually divided into smaller and smaller sub-clusters until each sub-cluster contains only one text. This method can identify the clustering results through a dendrogram. Among them, the cluster center can be a representative point or center point of each cluster cluster. In text clustering, the cluster center can include the representative reference text of the cluster. Therefore, in the embodiment of the present disclosure, the cluster center is the target reference text in the cluster obtained after the reference text in the reference text set is clustered in advance. In the embodiment of the present disclosure, when obtaining the reference text from the reference text set, the target compression distance between the text to be classified and each cluster center is calculated to obtain each target compression distance.

[0200] Step S903, based on the target compression distance, determining the target cluster center that is most similar to the text to be classified from the cluster centers.

[0201] Step S905 , obtaining reference texts respectively from the target clusters corresponding to the target cluster centers.

[0202] For example: cluster center 1: target compression distance 1; cluster center 2: target compression distance 2; cluster center 3: target compression distance 3. When SNCD is used to represent the above target compression distance, the cluster center corresponding to the largest target compression distance can be selected. For example, if target compression distance 2 is equal to target compression distance 1, and target compression distance 2 is also greater than the target compression distance, then target compression distance 2 is selected. It can be understood that cluster center 2 corresponding to target compression distance 2 is the cluster center most similar to the text to be classified, and is called the target cluster center. Reference texts are obtained from the target clusters corresponding to the target cluster centers.

[0203] In the disclosed embodiment, since the target cluster corresponding to the target cluster center has a higher similarity with the reference text to be classified than other clusters, the reference text is obtained from the target cluster corresponding to the target cluster center, and the reference text in other clusters is no longer obtained and calculated, which can effectively save the time of text classification and improve the efficiency of text classification.

[0204] In one embodiment, reference Fig.10As shown, obtaining the target compression distance between the text to be classified and each cluster center in the reference text set includes step S1001, step S1003, and step S1005, wherein:

[0205] Step S1001, respectively concatenate each of the cluster centers with the text to be classified to obtain a second concatenated text.

[0206] Specifically, the cluster center is spliced ​​with the text to be classified to obtain a first spliced ​​text. In an exemplary embodiment, the cluster center can be spliced ​​after the text to be classified. For example, the content of the cluster center includes: C1C2C3C4C5C6, and the content of the text to be classified includes: B1B2B3B4B5B6, then the first spliced ​​text can include: B1B2B3B4B5B6C1C2C3C4C5C6. In another exemplary embodiment, the cluster center can be spliced ​​before the text to be classified. For example, the content of the cluster center includes: C1C2C3C4C5C6, and the content of the text to be classified includes: B1B2B3B4B5B6, then the first spliced ​​text can include: C1C2C3C4C5C6 B1B2B3B4B5B6.

[0207] Step S1003, obtaining the fourth code length of the variable length code of the cluster center and the fifth code length of the variable length code of the second concatenated text.

[0208] In one embodiment, similar to the method of determining the first code length of the variable length code of the text to be classified in the above embodiment, the fourth code length of the variable length code of the cluster center in the embodiment of the present disclosure and the fifth code length of the variable length code of the second concatenated text are determined.

[0209] Step S1005, based on the first coding length, the fourth coding length and the fifth coding length, obtain the similarity distance between the text to be classified and the cluster center; based on the first coding length and the fourth coding length, obtain the reference distance; perform proportional calculation on the similarity distance between the text to be classified and the cluster center and the reference distance to obtain the target compression distance between the text to be classified and the cluster center.

[0210] In one embodiment, based on the first coding length, the fourth coding length and the fifth coding length, the method for obtaining the similarity distance between the text to be classified and the reference text may include performing mathematical operations on the first coding length, the fourth coding length and the fifth coding length. For example: based on different splicing methods of the cluster center and the text to be classified, the value of the fifth coding length obtained may be different, and the similarity distance between the text to be classified and the reference text is determined according to different values ​​of the fifth coding length. Optionally, the largest value is selected from the different values ​​corresponding to the fifth coding length as the similarity distance between the text to be classified and the reference text. For another example: based on the minimum value of the first coding length and the fourth coding length, the first component is obtained; based on the fifth coding length, the second component is obtained; based on the difference between the first component and the second component, the similarity distance between the text to be classified and the reference text is obtained.

[0211] In the disclosed embodiment, obtaining the reference distance based on the first code length and the fourth code length may include performing mathematical operations on the first code length and the fourth code length. For example, the maximum value of the first code length and the fourth code length is taken as the reference distance. For another example, the sum of the first code length and the fourth code length is taken as the reference distance. In an exemplary embodiment, the target compression distance between the text to be classified and the cluster center is the ratio of the similarity distance to the reference distance.

[0212] In one embodiment, obtaining reference texts from clusters corresponding to the target cluster centers respectively includes:

[0213] Obtain a target compression distance between the text to be classified and the center of each sub-cluster in the target cluster.

[0214] Based on the target compression distance, the center of the target subcluster closest to the to-be-classified text is determined from the centers of the subclusters.

[0215] Reference texts are respectively obtained from the target sub-clusters corresponding to the centers of the target sub-clusters.

[0216] Specifically, in hierarchical clustering, the reference texts contained in the clusters can be re-clustered to obtain more refined sub-clusters. The center of the sub-cluster can be the representative reference text or the central reference text of each sub-cluster. Furthermore, when the reference texts are respectively obtained from the clusters corresponding to the target cluster centers, the target compression distances between the texts to be classified and the centers of the sub-clusters are calculated to obtain the target compression distances. When SNCD is used to represent the above-mentioned target compression distances, the center of the sub-cluster corresponding to the largest target compression distance can be selected, which is called the center of the target sub-cluster. Reference texts are respectively obtained from the target sub-clusters corresponding to the centers of the target sub-clusters.

[0217] In the disclosed embodiment, since the target subcluster corresponding to the center of the target subcluster has a higher similarity with the reference text to be classified than other subclusters, the reference text is obtained from the target subcluster corresponding to the center of the target subcluster, and the reference text in other subclusters is no longer obtained and calculated, which can effectively save the time of text classification and further improve the efficiency of text classification.

[0218] It should be noted that the storage of the reference text is not limited to the above examples. For example, segmented storage can also be used as a storage method for reference text. Technical personnel in the relevant field may make other changes inspired by the technical essence of this application, but as long as the functions and effects achieved are the same or similar to those of this application, they should be covered within the scope of protection of this application.

[0219] In one embodiment, reference Fig.11 As shown, determining a target number of reference texts similar to the text to be classified from the reference text set based on the target compression distance includes steps S1101, S1103, S1105 and S1107. Among them:

[0220] Step S1101, traversing a target number of reference texts from the reference text set, taking a target compression distance between the text to be classified and each reference text in the target number of reference texts as a node of a small root heap, and constructing the small root heap.

[0221] Step S1103, continue to traverse the reference texts from the remaining reference texts in the reference text set except the target number of reference texts, and when the target compression distance of the traversed reference text is greater than the target compression distance of the root node of the small root heap, insert the target compression distance corresponding to the traversed reference text into the small root heap.

[0222] Wherein, the small root heap is a binary tree structure, and the root node of the small root heap is the minimum value in the heap. Specifically, taking the target compression distance as SNCD as an example, the target number of reference texts are traversed from the reference text set. In an exemplary embodiment, it can start from the starting storage position of the reference text set, traverse the target number of reference texts, and then traverse the reference texts one by one from the remaining reference texts, and compare the traversed reference texts with the root node of the small root heap. In another exemplary embodiment, it can also start from the reference text at the random middle position of the reference text set, traverse the target number of reference texts, and then traverse the reference texts one by one from the remaining reference texts, and compare the traversed reference texts with the root node of the small root heap. Wherein, the remaining reference texts can include moving the target number of reference texts forward or backward from the middle position to obtain a new traversal position, starting from the new traversal position until all the remaining reference texts in the reference text set are traversed.

[0223] In the disclosed embodiment, the reference texts are continuously traversed from the remaining reference texts in the reference text set except for the target number of reference texts. In an exemplary embodiment, when the target compression distance of the traversed reference text is greater than the target compression distance of the root node of the small root heap, the target compression distance corresponding to the traversed reference text is inserted into the small root heap. In another exemplary embodiment, when the traversed reference text is smaller than the root node of the small root heap, the traversed reference text is ignored.

[0224] Step S1105, sorting the nodes in the inserted small root heap to obtain an updated small root heap, until all the remaining reference texts are traversed to obtain the last updated small root heap.

[0225] In one of the embodiments, the nodes in the inserted rootlet are sorted to obtain an updated rootlet, until all the remaining reference texts are traversed to obtain the last updated rootlet. For example, when the target compression distance of the traversed reference text C is greater than the target compression distance of the root node of the rootlet, the target compression distance of the reference text C replaces the target compression distance of the root node of the rootlet to obtain an updated root node E. The updated root node E is compared with the node connected to it. If the root node E is greater than the node connected to it, the positions of the two nodes need to be exchanged. Until an updated rootlet is generated. Then the next reference text is traversed. Similarly, if the target compression distance of the next reference text is greater than the root node of the updated rootlet, the target compression distance of the next reference text is inserted into the updated rootlet and sorted. Until all the remaining reference texts are traversed to obtain the last updated rootlet.

[0226] Step S1107 , obtaining a target number of reference texts similar to the text to be classified based on the reference text corresponding to the last updated small root heap.

[0227] Based on the reference text corresponding to the last updated small root heap, the reference text with the target number having the largest target compression distance is obtained. When the target compression distance is SNCD, the reference text with the target number having the largest target compression distance is the reference text with the target number similar to the text to be classified. It should be noted that when the target compression distance is NCD, the reference text with the target number similar to the text to be classified may include the reference text with the target number having the smallest target compression distance. It is not difficult to understand that at this time, the reference text with the target number having the smallest target compression distance can be screened by constructing a large root heap, and the present disclosure will not elaborate on this. Inspired by the technical essence of the present application, technical personnel in the relevant field may also make other changes, but as long as the functions and effects achieved are the same or similar to those of the present application, they should be covered within the scope of protection of the present application.

[0228] In the above embodiment, by constructing a small root heap, a target number of reference texts similar to the text to be classified are screened, without sorting all the reference texts in the reference text set, which helps to improve the efficiency of text classification.

[0229] In one embodiment, based on the reference text corresponding to the last updated small root pile, obtaining the target number of reference texts with the largest target compression distance includes:

[0230] The nodes in the last updated small root heap are sorted to obtain a large root heap.

[0231] The root node in the big root heap is stored, and the root node is deleted from the big root heap to obtain remaining nodes.

[0232] An updated big root heap is constructed based on the remaining nodes, the root nodes of the updated big root heap are stored, and the corresponding root nodes are deleted from the updated big root heap until the number of stored root nodes reaches the target number.

[0233] The reference text corresponding to each root node is obtained from the stored root nodes in turn to obtain the arranged reference text.

[0234] Among them, corresponding to the small root heap, the big root heap is also a binary tree structure, and the root node of the big root heap is the maximum value in the heap. For example, the last updated small root heap is Fig.12 As shown in , the target compression distances corresponding to the target number of reference texts are 46859. The sorting of the nodes in the last updated small root heap can be adjusted from left to right and from bottom to top. For example: Fig.12 Starting from the first non-leaf node in , if non-leaf node 6 and its leaf node 9 do not meet the requirement that non-leaf nodes are larger than the leaf nodes connected to them, refer to Fig.13 As shown, the order of non-leaf node 6 is swapped with leaf node 9. Next, find the second non-leaf node 4. Since non-leaf node 4 is smaller than the two nodes of its branch, node 9 and node 8, refer to Fig.14 As shown, non-leaf node 4 and node 9 are swapped. At this time, the swap causes the structure of non-leaf node 4 and its two branch nodes, node 5 and node 6 to be chaotic, and the adjustment continues. Fig.15 As shown, exchange nodes 4 and 6. Now we get a large root heap.

[0235] In an exemplary embodiment, the root node 9 in the large root heap may be deleted. Fig.16 As shown, the root node 9 in the big root heap can also be exchanged with the leaf node 4, and the root node 9 can be deleted. The deletion is indicated by a dotted line in the figure. After deletion, the remaining node 4685 is obtained. Based on the remaining nodes, reorder, for example: Fig.17 As shown, the root node 4 is exchanged with a node 8 of its branch to obtain an updated large root heap. Fig.18 As shown, the root node 8 in the updated big root heap is exchanged with the leaf node 5, and the root node 8 is deleted. And so on, until the number of stored root nodes reaches the target number. Finally, an ordered reference text is obtained. Fig.19 shown.

[0236] In the above embodiment, by sorting the nodes of the last updated small root heap to obtain the big root heap, and continuously deleting and storing the root nodes in the big root heap and the root nodes of the updated big root heap, a sorted reference text can be finally obtained.

[0237] In one embodiment, determining the category of the text to be classified based on the categories corresponding to the target number of reference texts includes:

[0238] Based on the categories corresponding to the target number of reference texts, the number of reference texts corresponding to each category is obtained.

[0239] The category with the largest number of reference texts is taken as the category of the text to be classified.

[0240] Among them, the category corresponding to the reference text can be obtained from the reference text set. In another exemplary embodiment, the target number of reference texts can also be input into the classifier to obtain the category of each reference text. The number of reference texts corresponding to each category is obtained. For example: the categories corresponding to the target number of reference texts include: Category A, Category B, and Category C, where Category A corresponds to 18 reference texts; Category B corresponds to 20 reference texts; and Category C corresponds to 10 reference texts. At this time, the category with the largest number of reference texts is Category B. Therefore, the category of the text to be classified belongs to Category B in this embodiment.

[0241] In the above embodiment, the number of reference texts corresponding to each category is obtained by the categories corresponding to the target number of reference texts; the category with the largest number of reference texts is taken as the category of the text to be classified, so that the category of the text to be classified can be determined conveniently and quickly.

[0242] In an exemplary embodiment, determining the category of the text to be classified based on the categories corresponding to the target number of reference texts includes:

[0243] Obtaining the categories corresponding to the target number of reference texts and the weights of the reference texts; wherein the weight of the reference text is the inverse of the compression distance between the reference text and the target text to be classified;

[0244] Based on the categories corresponding to the target number of reference texts and the weights of the reference texts, a hit score for each category is obtained, and the category with the largest hit score is used as the category of the text to be classified.

[0245] For example, the categories corresponding to the target number of reference texts include: Category A, Category B, Category B, Category C. Category A, in this example, has 5 reference texts. The weight of the corresponding category to which the reference text belongs can be determined by the inverse of the target compression distance between the reference text and the text to be classified, for example, respectively: 0.5, 0.4, 0.3, 0.2, 0.1. It can be understood that the hit score of Category A is: 0.5+0.1=0.6, the hit score of Category B is: 0.4+0.3=0.7, and the hit score of Category C is: 0.2. The category with the largest hit score is Category B, so the category of the text to be classified is Category B.

[0246] In one embodiment, Fig. 20 As shown, a text clustering method is provided, and the method is applied to a computer device as an example. The computer device may include a terminal or a server. The method may be executed by the terminal or the server itself alone, or may be implemented through interaction between the terminal and the server. Fig. 20 As shown, the text clustering method includes the following steps:

[0247] Step S2001, obtaining the text to be clustered, and determining the first encoding length of the variable-length encoding of the text to be clustered.

[0248] The text to be clustered may include text to be clustered. In a specific implementation, the text to be clustered may be text in an application's public account, a post bar, a forum, a search engine, etc. The present disclosure does not limit the data source of the text to be clustered.

[0249] Among them, variable length coding is a coding method, different symbols or data elements are represented by sequences of different lengths. Compared with fixed length coding, variable length coding can achieve higher compression efficiency. In the specific implementation, variable length coding can be selected, such as Huffman coding (Huffman), LZ coding (Lempel-Ziv), BWT (Burrows-Wheeler Transform) and other data compression coding. Among them, the coding length of variable length coding refers to the number of bits of coding symbols after the text is converted into variable length coding. For example, the variable length code of text B is 10010101, and the length of the variable length code of text B is 8 in this example. In the embodiment of the present disclosure, the first coding length of the text to be clustered can be obtained by encoding the text to be clustered, obtaining the variable length code corresponding to the text to be clustered, and calculating the number of bits of the coding symbols in the variable length code.

[0250] Step S2003 , obtaining a reference text from a preset reference text set, and concatenating the reference text with the text to be clustered to obtain a first concatenated text.

[0251] The reference text set includes reference texts of various categories, for example, the reference text set is constructed in the form of content-label.

[0252] Specifically, the reference text is spliced ​​with the text to be clustered to obtain a first spliced ​​text. In an exemplary embodiment, the reference text can be spliced ​​after the text to be clustered. For example, the reference text content includes: A1A2A3A4A5A6, and the text to be clustered content includes: D1 D2D3D4D5D6, then the first spliced ​​text can include: D1 D2D3D4D5D6A1A2A3A4A5A6. In another exemplary embodiment, the reference text can be spliced ​​before the text to be clustered. For example, the reference text content includes: A1A2A3A4A5A6, and the text to be clustered content includes: D1 D2D3D4D5D6, then the first spliced ​​text can include: A1A2A3A4A5A6 D1 D2D3D4D5D6.

[0253] Step S2005, obtaining the second code length of the variable length code of the reference text and the third code length of the variable length code of the first concatenated text.

[0254] In one embodiment, similar to the method of determining the first code length of the variable length code of the text to be clustered in the above-mentioned embodiment, the second code length of the variable length code of the reference text in the embodiment of the present disclosure and the third code length of the variable length code of the first concatenated text are determined. Optionally, the second code length of the variable length code of the reference text can be pre-calculated and stored, and directly obtained from the storage area when used, thereby improving clustering efficiency. For example, a key-value pair data structure is used for storage, wherein the key can be used to store the identification information of the reference text, and the value can be used to store the second code length corresponding to the reference text.

[0255] Step S2007, based on the first coding length, the second coding length and the third coding length, obtain the similarity distance between the text to be classified and the reference text; based on the first coding length and the second coding length, obtain the reference distance; perform proportional calculation on the similarity distance and the reference distance to obtain the target compression distance between the clustered text and the reference text.

[0256] Specifically, based on the first coding length, the second coding length and the third coding length, the method for obtaining the similarity distance between the text to be clustered and the reference text may include performing mathematical operations on the first coding length, the second coding length and the third coding length. For example: based on the different splicing methods of the reference text and the text to be clustered, the value of the third coding length obtained may be different, and the similarity distance between the text to be clustered and the reference text is determined according to the different values ​​of the third coding length. Optionally, the largest value is selected from the different values ​​corresponding to the third coding length as the similarity distance between the text to be clustered and the reference text. For another example: based on the minimum value of the first coding length and the second coding length, the first component is obtained; based on the third coding length, the second component is obtained; based on the difference between the first component and the second component, the similarity distance between the text to be clustered and the reference text is obtained. In the disclosed embodiment, the similarity between the text to be clustered and the reference text is characterized by the similarity distance, and the text to be clustered, the reference text and the first concatenated text are introduced. The encoding length of each text can quantitatively characterize the Kolmogorov complexity of converting the text to be clustered into the reference text. The lower the Kolmogorov complexity, the more similar the text to be clustered is to the reference text, and the higher the complexity, the less similar the text to be clustered is to the reference text.

[0257] In the disclosed embodiment, obtaining the reference distance based on the first code length and the second code length may include performing mathematical operations on the first code length and the second code length. For example, the maximum value of the first code length and the second code length is taken as the reference distance. For another example, the sum of the first code length and the second code length is taken as the reference distance. Among them, the role of the reference distance is to limit the value of the target compression distance within a preset range, that is, normalization processing, so that a proportional calculation is performed for the similarity distance and the reference distance, and the target compression distance between the text to be clustered and the reference text has a normalized property. Thereby solving the comparability of data of different dimensions and more conveniently calculating the size of each target compression distance. In an exemplary embodiment, the target compression distance may include the ratio of the similarity distance to the reference distance.

[0258] Step S2009, determining a target number of reference texts similar to the text to be clustered from the reference text set based on the target compression distance, and determining the cluster where the text to be classified is located based on the clusters where the target number of reference texts are located.

[0259] Among them, the number of target quantities may include one or two or more. In an exemplary embodiment, when the target quantity is one, according to the specific expression of the specific target compression distance, a reference text with the smallest target compression distance with the text to be clustered can be selected from the reference text set, and the cluster corresponding to the smallest reference text is used as the cluster where the text to be clustered is located; or a reference text with the largest target compression distance with the text to be classified is selected, and the largest reference text is used as the cluster where the text to be clustered is located. In another exemplary embodiment, when the target quantity is two or more, the category of the text to be classified can be determined according to the categories of two or more reference texts. For example: based on the clusters corresponding to two or more reference texts, the number of corresponding reference texts under each cluster is obtained; the cluster where the largest number of reference texts is located is used as the cluster where the text to be classified is located.

[0260] In the disclosed embodiment, the cluster where the text to be classified is located can also be expanded again to the reference text set, and data such as the Huffman tree corresponding to the reference text set can be updated.

[0261] The above-mentioned text clustering method determines the similarity distance between the text to be clustered and the reference text by the first code length of the variable length code of the text to be clustered, the second code length of the variable length code of the reference text, and the third code length of the variable length code of the first concatenated text of the text to be clustered and the reference text. The similarity distance can characterize the similarity between the text to be clustered and the reference text, that is, two texts with greater similarity have closer codes and code lengths than other texts of different categories. Further, the embodiment of the present disclosure obtains a reference distance based on the first code length and the second code length, wherein the role of the reference distance is to limit the value of the target compression distance within a preset range, that is, normalization processing, so that the target compression distance has a normalized property. Thereby solving the comparability of data of different dimensions and more conveniently calculating the size of each target compression distance. Finally, the embodiment of the present disclosure determines the target number of reference texts similar to the text to be clustered from the reference text set based on the target compression distance, and determines the cluster where the text to be clustered is located based on the cluster where the target number of reference texts are located. In this way, the embodiment of the present disclosure does not need to construct a complex initial clustering model, and does not need to train and update the initial clustering model. Therefore, the embodiment of the present disclosure improves the efficiency of text clustering.

[0262] In one embodiment, Fig.21 As shown, a text retrieval method is provided, and the method is applied to a computer device as an example. The computer device may include a terminal or a server. The method may be executed by the terminal or the server itself alone, or may be implemented through interaction between the terminal and the server. Fig.21As shown, the text retrieval method includes the following steps:

[0263] Step S2101, obtaining a text to be retrieved, and determining a first encoding length of a variable-length encoding of the text to be retrieved.

[0264] The text to be retrieved may include text to be retrieved. In a specific implementation, the text to be retrieved may be text in an official account of an application, a post bar, a forum, a search engine, etc. The present disclosure does not limit the data source of the text to be retrieved.

[0265] Among them, variable length coding is a coding method, different symbols or data elements are represented by sequences of different lengths. Compared with fixed length coding, variable length coding can achieve higher compression efficiency. In the specific implementation, variable length coding can choose, for example, Huffman coding (Huffman), LZ coding (Lempel-Ziv), BWT (Burrows-Wheeler Transform) and other data compression coding. Among them, the coding length of variable length coding refers to the number of bits of coding symbols after the text is converted into variable length coding. For example, the variable length code of text B is 10010101, then the length of the variable length code of text B is 8 in this example. In the disclosed embodiment, the first coding length of the text to be retrieved can be obtained by encoding the text to be retrieved, obtaining the variable length code corresponding to the text to be retrieved, and calculating the number of bits of the coding symbols in the variable length code.

[0266] Step S2103: obtaining a reference text from a preset reference text set, and concatenating the reference text with the text to be classified to obtain a first concatenated text.

[0267] The reference text set includes reference texts of various categories, for example, the reference text set is constructed in the form of content-label.

[0268] Specifically, the reference text is spliced ​​with the text to be retrieved to obtain a first spliced ​​text. In an exemplary embodiment, the reference text may be spliced ​​after the text to be retrieved. For example, the reference text content includes: A1A2A3A4A5A6, and the text to be retrieved content includes: E1 E2E3E4E5E6, then the first spliced ​​text may include: E1E2E3E4E5E6A1A2A3A4A5A6. In another exemplary embodiment, the reference text may be spliced ​​before the text to be retrieved. For example, the reference text content includes: A1A2A3A4A5A6, and the text to be retrieved content includes: E1E2E3E4E5E6, then the first spliced ​​text may include: A1A2A3A4A5A6 E1E2E3E4E5E6.

[0269] Step S2105, obtaining the second code length of the variable length code of the reference text and the third code length of the variable length code of the first concatenated text.

[0270] In one embodiment, similar to the method of determining the first code length of the variable length code of the text to be retrieved in the above-mentioned embodiment, the second code length of the variable length code of the reference text in the embodiment of the present disclosure and the third code length of the variable length code of the first concatenated text are determined. Optionally, the second code length of the variable length code of the reference text can be pre-calculated and stored, and directly obtained from the storage area when used, thereby improving retrieval efficiency. For example, a key-value pair data structure is used for storage, wherein the key can be used to store the identification information of the reference text, and the value can be used to store the second code length corresponding to the reference text.

[0271] Step S2107, based on the first coding length, the second coding length and the third coding length, obtain the similarity distance between the text to be retrieved and the reference text; based on the first coding length and the second coding length, obtain the reference distance; perform proportional calculation on the similarity distance and the reference distance to obtain the target compression distance between the text to be retrieved and the reference text.

[0272] Specifically, based on the first coding length, the second coding length and the third coding length, the method for obtaining the similarity distance between the text to be retrieved and the reference text may include performing mathematical operations on the first coding length, the second coding length and the third coding length. For example: based on different splicing methods of the reference text and the text to be retrieved, the value of the third coding length obtained may be different, and the similarity distance between the text to be retrieved and the reference text is determined according to different values ​​of the third coding length. Optionally, the largest value is selected from the different values ​​corresponding to the third coding length as the similarity distance between the text to be retrieved and the reference text. For another example: based on the minimum value of the first coding length and the second coding length, the first component is obtained; based on the third coding length, the second component is obtained; based on the difference between the first component and the second component, the similarity distance between the text to be retrieved and the reference text is obtained. In the disclosed embodiment, the similarity between the text to be retrieved and the reference text is represented by the similarity distance, and the text to be retrieved, the reference text and the first concatenated text are introduced. The encoding length of each text can quantitatively represent the Kolmogorov complexity of converting the text to be retrieved into the reference text. The lower the Kolmogorov complexity, the more similar the text to be retrieved is to the reference text, and the higher the complexity, the less similar the text to be retrieved is to the reference text.

[0273] In the disclosed embodiment, obtaining the reference distance based on the first coding length and the second coding length may include performing mathematical operations on the first coding length and the second coding length. For example, the maximum value of the first coding length and the second coding length is taken as the reference distance. For another example, the sum of the first coding length and the second coding length is taken as the reference distance. Among them, the role of the reference distance is to limit the value of the target compression distance within a preset range, that is, normalization processing, so that a proportional calculation is performed for the similarity distance and the reference distance, and the target compression distance between the text to be retrieved and the reference text has a normalized property. Thereby solving the comparability of data of different dimensions and more conveniently calculating the size of each target compression distance. In an exemplary embodiment, the target compression distance may include the ratio of the similarity distance to the reference distance.

[0274] Step S2109 , determining a target number of reference texts similar to the text to be retrieved from the reference text set based on the target compression distance as target texts corresponding to the text to be retrieved.

[0275] Among them, the number of target quantities may include one or two or more. In an exemplary embodiment, when the target quantity is one, according to a specific expression of a specific target compression distance, a reference text with the smallest target compression distance with the text to be retrieved can be selected from the reference text set, and the smallest reference text can be used as the target text corresponding to the text to be retrieved; or a reference text with the largest target compression distance with the text to be classified can be selected, and the largest reference text can be used as the target text corresponding to the text to be retrieved. In another exemplary embodiment, when the target quantity is two or more, two or more reference texts can be used as the target text corresponding to the text to be retrieved.

[0276] The above-mentioned text retrieval method determines the similarity distance between the text to be retrieved and the reference text by the first code length of the variable length code of the text to be retrieved, the second code length of the variable length code of the reference text, and the third code length of the variable length code of the first concatenated text of the text to be retrieved and the reference text. The similarity distance can characterize the similarity between the text to be retrieved and the reference text, that is, two texts with greater similarity have closer codes and code lengths than other texts of different categories. Further, the embodiment of the present disclosure obtains the reference distance based on the first code length and the second code length, wherein the role of the reference distance is to limit the value of the target compression distance within a preset range, that is, normalization processing, so that the target compression distance has a normalized property. Thereby solving the comparability of data of different dimensions and more conveniently calculating the size of each target compression distance. Finally, the embodiment of the present disclosure determines the target number of reference texts similar to the text to be retrieved from the reference text set based on the target compression distance, and determines the retrieval where the text to be retrieved is located based on the retrieval where the target number of reference texts are located. In this way, the embodiment of the present disclosure does not need to construct a complex initial retrieval model, and does not need to train and update the initial retrieval model. Therefore, the embodiment of the present disclosure improves the efficiency of text retrieval.

[0277] In a specific embodiment, the method of the present application can be applied to the application scenario of text classification. In the prior art, the classification model is trained by traditional machine learning or deep learning. When a new category is added, the classification model needs to be retrained, resulting in low classification efficiency. The present application provides a text classification method that can improve the efficiency of text classification.

[0278] refer to Fig. 22 As shown, the acquisition process of the first encoding length of the text to be classified specifically includes: acquiring the text to be classified online. Performing word segmentation processing on the text to be classified to obtain each character corresponding to the text to be classified. Based on the preset association relationship between the character and the Huffman code, determining the first encoding length of the Huffman code of the text to be classified.

[0279] The reference text acquisition link specifically includes: acquiring reference text from a reference text set. Calculating the target compression distance between the text to be classified and each cluster center, and based on the target compression distance, determining the target cluster center that is most similar to the text to be classified from each cluster center. Further, calculating the target compression distance between the text to be classified and the center of each sub-cluster in the target cluster where the target cluster center is located, and based on the target compression distance, determining the center of the target sub-cluster that is most similar to the text to be classified from the center of each sub-cluster. Acquire the reference text from the target sub-cluster corresponding to the center of the target sub-cluster. In an exemplary embodiment, the second encoding length corresponding to the above-mentioned reference text can be obtained from the key-value storage.

[0280] The concatenation of the reference text and the text to be classified specifically includes: in the disclosed embodiment, the reference text and the text to be classified are concatenated in a first order to obtain a first concatenated subtext. The reference text and the text to be classified are concatenated in a second order to obtain a second concatenated subtext. The first code sublength of the variable length code of the first concatenated subtext and the second code sublength of the variable length code of the second concatenated subtext are obtained.

[0281] The determination of the target compression distance specifically includes: obtaining the similarity distance between the text to be classified and the reference text based on the maximum value of the first code sub-length and the second code sub-length; obtaining the reference distance based on the maximum value of the first code length and the second code length. Proportional calculation is performed on the similarity distance and the reference distance to obtain the target compression distance between the text to be classified and the reference text.

[0282] The process of determining a target number of reference texts from a reference text set specifically includes: traversing a target number of reference texts from the reference text set, taking the target compression distance between the text to be classified and each reference text in the target number of reference texts as a node of a small root heap, and constructing the small root heap; continuing to traverse the reference texts from the remaining reference texts in the reference text set except the target number of reference texts, and when the target compression distance of the traversed reference text is greater than the target compression distance of the root node of the small root heap, inserting the target compression distance corresponding to the traversed reference text into the small root heap; sorting the nodes in the inserted small root heap to obtain an updated small root heap, until all the remaining reference texts are traversed and the target number of reference texts similar to the text to be classified obtain the last updated small root heap; based on the reference text corresponding to the last updated small root heap, obtaining the target number of reference texts with the largest target compression distance.

[0283] The process of sorting the target number of reference texts specifically includes: continuing to sort the nodes in the last updated small root heap to obtain a large root heap; storing the root nodes in the large root heap, and deleting the root nodes from the large root heap to obtain the remaining nodes; constructing an updated large root heap based on the remaining nodes, storing the root nodes of the updated large root heap, and deleting the corresponding root nodes from the updated large root heap until the number of stored root nodes reaches the target number; and obtaining the reference text corresponding to each root node from the stored root nodes in turn to obtain the arranged reference text.

[0284] The step of determining the category of the text to be classified specifically includes: obtaining the number of reference texts corresponding to each category based on the categories corresponding to the target number of reference texts; and taking the category with the largest number of reference texts as the category of the text to be classified.

[0285] The above-mentioned text classification method determines the similarity distance between the text to be classified and the reference text by the first code length of the variable length code of the text to be classified, the second code length of the variable length code of the reference text, and the third code length of the variable length code of the first concatenated text of the text to be classified and the reference text. The similarity distance can characterize the similarity between the text to be classified and the reference text, that is, two texts with greater similarity have closer codes and code lengths than other texts of different categories. Further, the embodiment of the present disclosure obtains the reference distance based on the first code length and the second code length, wherein the role of the reference distance is to limit the value of the target compression distance within a preset range, that is, normalization processing, so that the target compression distance has a normalized property. Thereby solving the comparability of data of different dimensions and more conveniently calculating the size of each target compression distance. In addition, the embodiment of the present disclosure selects the Huffman code length. Since the Huffman code has the property of non-repetitive prefixes, the code corresponding to each character is not the prefix of the code corresponding to other characters, which is conducive to accurately restoring the original data during decoding. The encoding method of the Huffman code enables the Huffman code length to more recognizablely characterize the corresponding text. Therefore, the closer the Huffman coding length of the text to be classified is to the Huffman coding length of the reference text, the more similar the two texts are. Therefore, the embodiment of the present disclosure utilizes the Huffman coding length to construct a target compression distance, which can improve the accuracy of text classification. Finally, the embodiment of the present disclosure determines a target number of reference texts similar to the text to be classified from the reference text set based on the target compression distance, and determines the category of the text to be classified based on the category of the target number of reference texts. In this way, the embodiment of the present disclosure does not need to construct a complex initial classification model, and when a new category is added, there is no need to train and update the initial classification model. Therefore, the embodiment of the present disclosure improves the efficiency of text classification.

[0286] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0287] Based on the same inventive concept, the embodiment of the present application also provides a text classification device for implementing the above-mentioned text classification method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more text classification device embodiments provided below can refer to the limitations on the text classification method above, and will not be repeated here.

[0288] In one embodiment, Fig.23 As shown, a text classification device 2300 is provided, including:

[0289] The first acquisition module 2301 is used to acquire the text to be classified and determine the first code length of the variable length code of the text to be classified;

[0290] The splicing module 2303 is used to obtain reference texts from the reference text set, and splice the reference texts with the text to be classified to obtain first spliced ​​texts;

[0291] The second acquisition module 2305 is used to acquire the second code length of the variable length code of the reference text and the third code length of the variable length code of the first concatenated text;

[0292] A generating module 2307 is used to obtain a similarity distance between the text to be classified and a reference text based on the first coding length, the second coding length and the third coding length; obtain a reference distance based on the first coding length and the second coding length; perform a proportional calculation on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text, wherein the target compression distance is used to characterize the similarity between the text to be classified and the reference text;

[0293] The first determination module is used to determine a target number of reference texts similar to the text to be classified from the reference text set based on a target compression distance, and determine the category of the text to be classified based on the categories of the target number of reference texts.

[0294] In one embodiment, the first concatenated text includes a first concatenated subtext and a second concatenated subtext, and the concatenating module includes:

[0295] A first concatenation submodule is used to concatenate the reference text and the text to be classified in a first order to obtain a first concatenated subtext; in the first concatenated subtext, the reference text precedes the text to be classified;

[0296] A second concatenation submodule is used to concatenate the reference text and the text to be classified in a second order to obtain a second concatenated subtext; in the second concatenated subtext, the text to be classified precedes the reference text;

[0297] The generation module comprises:

[0298] A first acquisition submodule, used for acquiring a first code sublength of the variable length code of the first concatenated subtext and a second code sublength of the variable length code of the second concatenated subtext;

[0299] A first generating submodule, used for obtaining a similarity distance between the text to be classified and a reference text based on the first encoding sublength and the second encoding sublength;

[0300] A second generating submodule, configured to obtain a reference distance based on the first coding length and the second coding length;

[0301] The third generation submodule is used to calculate the ratio of the similarity distance and the reference distance to obtain the target compression distance between the text to be classified and the reference text.

[0302] In one embodiment, the first generation submodule includes:

[0303] A first generating unit, configured to obtain a similarity distance between the text to be classified and a reference text based on a maximum value of the first encoding sub-length and the second encoding sub-length;

[0304] The second generation submodule includes:

[0305] The second generating unit is configured to obtain a reference distance based on a maximum value of the first coding length and the second coding length.

[0306] In one embodiment, the generating module comprises:

[0307] A comparison submodule, configured to obtain a first component based on a minimum value between the first encoding length and the second encoding length;

[0308] A fourth generating submodule, configured to obtain a second component based on the third coding length;

[0309] A fifth generating submodule, configured to obtain a similarity distance between the text to be classified and a reference text based on a difference between the first component and the second component;

[0310] a sixth generating submodule, configured to obtain a reference distance based on a maximum value of the first coding length and the second coding length;

[0311] The seventh generation submodule is used to perform a ratio calculation on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text.

[0312] In one embodiment, the target text includes at least one of a text to be classified, a reference text, a first concatenated text, and a second concatenated text, and the text classification device further includes:

[0313] The first word segmentation module is used to obtain a target text, perform word segmentation processing on the target text, and obtain each character corresponding to the target text;

[0314] A second determination module is used to determine the Huffman code corresponding to each of the characters based on a preset association relationship between the characters and the Huffman code;

[0315] The third determination module is used to determine that the encoding length of the Huffman code of the target text is the sum of the encoding lengths of the Huffman codes corresponding to the characters.

[0316] In one embodiment, the text classification device further includes:

[0317] A second word segmentation module is used to perform word segmentation processing on the reference text in the reference text set to obtain each character corresponding to the reference text set;

[0318] A statistical module, used for counting the frequency of each character in each character corresponding to the reference text set appearing in the reference text set;

[0319] The fourth determination module is used to construct a Huffman tree based on the characters corresponding to the reference text set and the frequencies corresponding to each of the characters, and determine the Huffman code corresponding to each of the characters based on the Huffman tree.

[0320] In one embodiment, the splicing module includes:

[0321] The second acquisition submodule is used to obtain the target compression distance between the text to be classified and each cluster center in the reference text set; wherein the cluster center is the target reference text in the cluster obtained by clustering the reference texts in the reference text set in advance;

[0322] A first determination submodule is used to determine, from each of the cluster centers, a target cluster center that is most similar to the text to be classified based on the target compression distance;

[0323] The third acquisition submodule is used to respectively acquire reference texts from the target clusters corresponding to the target cluster centers.

[0324] In one embodiment, the second acquisition submodule is used to

[0325] The first acquisition unit is used to acquire the target compression distance between the text to be classified and each cluster center in the reference text set, including:

[0326] A concatenation unit, used for concatenating each of the cluster centers with the text to be classified to obtain a second concatenated text;

[0327] A second acquisition unit is used to acquire a fourth code length of the variable length code of the cluster center and a fifth code length of the variable length code of the second concatenated text;

[0328] The third generating unit is used to obtain the similarity distance between the text to be classified and the cluster center based on the first coding length, the fourth coding length and the fifth coding length; obtain the reference distance based on the first coding length and the fourth coding length; perform proportional calculation on the similarity distance between the text to be classified and the cluster center and the reference distance to obtain the target compression distance between the text to be classified and the cluster center.

[0329] In one embodiment, the third acquisition submodule includes:

[0330] A third acquisition unit is used to acquire a target compression distance between the text to be classified and the center of each sub-cluster in the target cluster;

[0331] A determination unit, configured to determine, from the centers of the subclusters, the center of the target subcluster closest to the text to be classified based on the target compression distance;

[0332] The fourth acquisition unit is used to respectively acquire reference texts from the target sub-clusters corresponding to the centers of the target sub-clusters.

[0333] In one embodiment, the first determining module includes:

[0334] A construction submodule, used for traversing a target number of reference texts from the reference text set, taking a target compression distance between the text to be classified and each reference text in the target number of reference texts as a node of a small root heap, and constructing the small root heap;

[0335] A traversal submodule, configured to continue traversing reference texts from the remaining reference texts in the reference text set except the target number of reference texts, and insert the target compression distance corresponding to the traversed reference text into the small root heap when the target compression distance of the traversed reference text is greater than the target compression distance of the root node of the small root heap;

[0336] A sorting submodule is used to sort the nodes in the inserted small root heap to obtain an updated small root heap, until all the remaining reference texts are traversed to obtain the last updated small root heap;

[0337] The eighth generation submodule is used to obtain a target number of reference texts with the largest target compression distance based on the reference texts corresponding to the last updated small root heap.

[0338] In one embodiment, the eighth generation submodule includes:

[0339] A sorting unit, used for sorting the nodes in the last updated small root heap to obtain a large root heap;

[0340] A deletion unit, used for storing the root node in the big root heap, and deleting the root node from the big root heap to obtain the remaining nodes;

[0341] A storage unit, configured to construct an updated big root heap based on the remaining nodes, store the root nodes of the updated big root heap, and delete the corresponding root nodes from the updated big root heap until the number of stored root nodes reaches the target number;

[0342] The fourth generating unit is used to sequentially obtain the reference text corresponding to each root node from the stored root nodes to obtain the arranged reference text.

[0343] In one embodiment, the determining module includes:

[0344] A fourth acquisition submodule is used to acquire the number of reference texts corresponding to each category based on the categories corresponding to the target number of reference texts;

[0345] The second determination submodule is used to take the category where the number of reference texts is the largest as the category of the text to be classified.

[0346] In one embodiment, the determining module includes:

[0347] A fifth acquisition submodule is used to acquire the categories corresponding to the target number of reference texts and the weights of the reference texts; wherein the weight of the reference text is the inverse of the compression distance between the reference text and the target text to be classified;

[0348] The third determination submodule is used to obtain a hit score for each category based on the categories corresponding to the target number of reference texts and the weights of the reference texts, and to take the category with the largest hit score as the category of the text to be classified.

[0349] In one embodiment, Fig.24 As shown, a text clustering device 2400 is provided, and the device 2400 includes:

[0350] A first acquisition module 2401 is used to acquire the text to be clustered and determine a first code length of the variable-length code of the text to be clustered;

[0351] A splicing module 2403 is used to obtain a reference text from a preset reference text set, and splice the reference text with the text to be clustered to obtain a first spliced ​​text;

[0352] A second acquisition module 2405 is used to acquire a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text;

[0353] A generating module 2407 is used to obtain a similarity distance between the text to be clustered and the reference text based on the first coding length, the second coding length and the third coding length; obtain a reference distance based on the first coding length and the second coding length; perform a proportional calculation on the similarity distance and the reference distance to obtain a target compression distance between the clustered text and the reference text;

[0354] The determination module 2409 is used to determine a target number of reference texts similar to the text to be clustered from the reference text set based on the target compression distance, and determine the cluster where the text to be clustered is located based on the clusters where the target number of reference texts are located.

[0355] In one embodiment, Fig.25 As shown, a text search device 2500 is provided, and the device 2500 includes:

[0356] The first acquisition module 2501 is used to acquire the text to be retrieved and determine the first encoding length of the variable length encoding of the text to be retrieved;

[0357] The splicing module 2503 is used to obtain a reference text from a preset reference text set, and splice the reference text with the text to be searched to obtain a first spliced ​​text;

[0358] A second acquisition module 2505 is used to acquire a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text;

[0359] A generating module 2507 is used to obtain a similarity distance between the text to be retrieved and the reference text based on the first encoding length, the second encoding length and the third encoding length; obtain a reference distance based on the first encoding length and the second encoding length; perform a proportional calculation on the similarity distance and the reference distance to obtain a target compression distance between the text to be retrieved and the reference text;

[0360] The determination module 2509 is used to determine a target number of reference texts similar to the text to be retrieved from the reference text set based on the target compression distance as target texts corresponding to the text to be retrieved.

[0361] Each module in the above-mentioned text classification device, text clustering device, and text retrieval device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0362] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Fig.26 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store reference text data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a text classification, clustering and retrieval method is implemented.

[0363] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig. 27As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a text classification, clustering and retrieval method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0364] Those skilled in the art will understand that Fig. 27 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0365] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0366] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0367] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A text classification method, characterized in that: The method comprises: Acquire a text to be classified, and determine a first encoding length of a variable-length encoding of the text to be classified; Obtain reference texts from the reference text set respectively, and concatenate the reference texts with the text to be classified respectively to obtain first concatenated texts; Obtaining a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text; Based on the first coding length, the second coding length and the third coding length, a similarity distance between the text to be classified and a reference text is obtained; based on the first coding length and the second coding length, a reference distance is obtained; a ratio calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text, and the target compression distance is used to characterize the similarity between the text to be classified and the reference text; A target number of reference texts similar to the text to be classified is determined from the reference text set based on the target compression distance, and a category of the text to be classified is determined based on the categories of the target number of reference texts.

2. The method according to claim 1, characterized in that The first concatenated text includes a first concatenated subtext and a second concatenated subtext. The reference text and the text to be classified are concatenated to obtain the first concatenated text, including: The reference text and the text to be classified are respectively concatenated in a first order to obtain a first concatenated sub-text; in the first concatenated sub-text, the reference text precedes the text to be classified; The reference text and the text to be classified are respectively concatenated in a second order to obtain a second concatenated sub-text; in the second concatenated sub-text, the text to be classified precedes the reference text; The method of obtaining a similarity distance between the text to be classified and a reference text based on the first coding length, the second coding length, and the third coding length; obtaining a reference distance based on the first coding length and the second coding length; and performing a proportional calculation on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text includes: Obtaining a first code sub-length of the variable length encoding of the first concatenated sub-text and a second code sub-length of the variable length encoding of the second concatenated sub-text; Based on the first code sub-length and the second code sub-length, obtaining a similarity distance between the text to be classified and the reference text; Obtaining a reference distance based on the first coding length and the second coding length; A ratio calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text.

3. The method according to claim 2, characterized in that Based on the first code sub-length and the second code sub-length, obtaining a similarity distance between the text to be classified and the reference text; Obtaining a reference distance based on the first coding length and the second coding length includes: Based on the maximum value of the first encoding sub-length and the second encoding sub-length, obtaining a similarity distance between the text to be classified and the reference text; A reference distance is obtained based on a maximum value of the first code length and the second code length.

4. The method according to claim 1, characterized in that: Based on the first coding length, the second coding length, and the third coding length, a similarity distance between the text to be classified and a reference text is obtained; based on the first coding length and the second coding length, a reference distance is obtained, and a ratio calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text, including: Obtaining a first component based on a minimum value between the first encoding length and the second encoding length; Based on the third code length, obtaining a second component; Based on the difference between the first component and the second component, obtaining a similarity distance between the text to be classified and the reference text; Obtaining a reference distance based on a maximum value of the first code length and the second code length; A ratio calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text.

5. The method according to claim 1, characterized in that The target text includes at least one of a text to be classified, a reference text, a first concatenated text, and a second concatenated text. The method for determining the encoding length of the variable-length encoding of the target text includes: Acquire a target text, perform word segmentation on the target text, and obtain characters corresponding to the target text; Determine the Huffman code corresponding to each of the characters based on a preset association relationship between the characters and the Huffman code; The coding length of the Huffman coding of the target text is determined to be the sum of the coding lengths of the Huffman coding corresponding to each of the characters.

6. The method according to claim 5, characterized in that The preset association relationship between the character and the Huffman code is obtained in the following manner: Performing word segmentation processing on the reference texts in the reference text set to obtain each character corresponding to the reference text set; Counting the frequency of each character in each character corresponding to the reference text set appearing in the reference text set; A Huffman tree is constructed based on the characters corresponding to the reference text set and the frequencies corresponding to each of the characters, and a Huffman code corresponding to each of the characters is determined based on the Huffman tree.

7. The method according to claim 1, characterized in that Get reference texts from the reference text collection separately, including: Obtaining a target compression distance between the text to be classified and each cluster center in the reference text set; wherein the cluster center is a target reference text in a cluster obtained by pre-clustering the reference texts in the reference text set; Based on the target compression distance, determining the target cluster center that is most similar to the text to be classified from each of the cluster centers; Reference texts are respectively obtained from the target clusters corresponding to the target cluster centers.

8. The method according to claim 7, characterized in that Obtaining a target compression distance between the text to be classified and each cluster center in the reference text set includes: Respectively concatenating each of the cluster centers with the text to be classified to obtain a second concatenated text; Obtaining a fourth code length of the variable length code of the cluster center and a fifth code length of the variable length code of the second concatenated text; Based on the first coding length, the fourth coding length and the fifth coding length, the similarity distance between the text to be classified and the cluster center is obtained; based on the first coding length and the fourth coding length, the reference distance is obtained; and a proportional calculation is performed on the similarity distance between the text to be classified and the cluster center and the reference distance to obtain a target compression distance between the text to be classified and the cluster center.

9. The method according to claim 7, characterized in that: The obtaining reference texts respectively from the clusters corresponding to the target cluster centers includes: Obtaining a target compression distance between the text to be classified and the center of each sub-cluster in the target cluster; Based on the target compression distance, determining the center of the target subcluster closest to the text to be classified from the centers of each subcluster; Reference texts are respectively obtained from the target sub-clusters corresponding to the centers of the target sub-clusters.

10. The method according to claim 1, characterized in that Determining a target number of reference texts similar to the text to be classified from the reference text set based on the target compression distance includes: Traversing a target number of reference texts from the reference text set, taking a target compression distance between the text to be classified and each reference text in the target number of reference texts as a node of a small root heap, and constructing the small root heap; Continue traversing reference texts from the remaining reference texts in the reference text set except the target number of reference texts, and when the target compression distance of the traversed reference text is greater than the target compression distance of the root node of the small root heap, insert the target compression distance corresponding to the traversed reference text into the small root heap; The nodes in the inserted small root heap are sorted to obtain an updated small root heap, until all the remaining reference texts are traversed to obtain the last updated small root heap; Based on the reference text corresponding to the last updated small root pile, a target number of reference texts with the largest target compression distance is obtained.

11. The method according to claim 10, characterized in that Based on the reference text corresponding to the last updated small root heap, a reference text of a target quantity with a maximum target compression distance is obtained, including: Sorting the nodes in the last updated small root heap to obtain a large root heap; Storing the root node in the big root heap, and deleting the root node from the big root heap to obtain remaining nodes; Building an updated big root heap based on the remaining nodes, storing the root nodes of the updated big root heap, and deleting the corresponding root nodes from the updated big root heap until the number of stored root nodes reaches the target number; The reference text corresponding to each root node is obtained in turn from the stored root nodes to obtain the arranged reference text.

12. The method according to claim 1, characterized in that Determining the category of the text to be classified based on the categories corresponding to the target number of reference texts includes: Based on the categories corresponding to the target number of reference texts, obtain the number of reference texts corresponding to each category; The category with the largest number of reference texts is taken as the category of the text to be classified.

13. A text clustering method, characterized in that: The method comprises: Acquire the text to be clustered, and determine a first encoding length of the variable-length encoding of the text to be clustered; Acquire a reference text from a preset reference text set, and concatenate the reference text with the text to be clustered to obtain a first concatenated text; Obtaining a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text; Based on the first coding length, the second coding length and the third coding length, a similarity distance between the text to be clustered and the reference text is obtained; based on the first coding length and the second coding length, a reference distance is obtained; a ratio calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the clustered text and the reference text; A target number of reference texts similar to the text to be clustered is determined from the reference text set based on the target compression distance, and the cluster where the text to be clustered is located is determined based on the clusters where the target number of reference texts are located.

14. A text retrieval method, characterized in that: The method comprises: Acquire a text to be retrieved, and determine a first encoding length of a variable-length encoding of the text to be retrieved; Acquire a reference text from a preset reference text set, and concatenate the reference text with the text to be searched to obtain a first concatenated text; Obtaining a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text; Based on the first coding length, the second coding length and the third coding length, a similarity distance between the text to be retrieved and the reference text is obtained; based on the first coding length and the second coding length, a reference distance is obtained; a ratio calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the text to be retrieved and the reference text; Based on the target compression distance, a target number of reference texts similar to the text to be classified are determined from the reference text set as target texts corresponding to the text to be retrieved.

15. A text classification device, characterized in that: The device comprises: A first acquisition module, used for acquiring a text to be classified and determining a first encoding length of a variable-length encoding of the text to be classified; A splicing module, used to obtain reference texts from the reference text set, respectively, and splice the reference texts with the text to be classified to obtain a first spliced ​​text; A second acquisition module, used for acquiring a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text; A generation module, configured to obtain a similarity distance between the text to be classified and a reference text based on the first encoding length, the second encoding length, and the third encoding length; obtain a reference distance based on the first encoding length and the second encoding length; perform a proportional calculation on the similarity distance and the reference distance to obtain a target compression distance between the text to be classified and the reference text, wherein the target compression distance is used to characterize the similarity between the text to be classified and the reference text; The determination module is used to determine a target number of reference texts similar to the text to be classified from the reference text set based on a target compression distance, and determine the category of the text to be classified based on the categories of the target number of reference texts.

16. A text clustering device, characterized in that: The device comprises: A first acquisition module, used for acquiring the text to be clustered, and determining a first encoding length of the variable-length encoding of the text to be clustered; A splicing module, used for obtaining a reference text from a preset reference text set, and splicing the reference text with the text to be clustered to obtain a first spliced ​​text; A second acquisition module, used for acquiring a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text; A generation module, configured to obtain a similarity distance between the text to be clustered and a reference text based on the first coding length, the second coding length, and the third coding length; obtain a reference distance based on the first coding length and the second coding length; and perform a proportional calculation on the similarity distance and the reference distance to obtain a target compression distance between the clustered text and the reference text; The determination module is used to determine a target number of reference texts similar to the text to be clustered from the reference text set based on a target compression distance, and determine the cluster where the text to be clustered is located based on the clusters where the target number of reference texts are located.

17. A text retrieval device, characterized in that: The device comprises: Acquire a text to be retrieved, and determine a first encoding length of a variable-length encoding of the text to be retrieved; Acquire a reference text from a preset reference text set, and concatenate the reference text with the text to be searched to obtain a first concatenated text; Obtaining a second code length of the variable length code of the reference text and a third code length of the variable length code of the first concatenated text; Based on the first coding length, the second coding length and the third coding length, a similarity distance between the text to be retrieved and the reference text is obtained; based on the first coding length and the second coding length, a reference distance is obtained; a ratio calculation is performed on the similarity distance and the reference distance to obtain a target compression distance between the text to be retrieved and the reference text; Based on the target compression distance, a target number of reference texts similar to the text to be retrieved are determined from the reference text set as target texts corresponding to the text to be retrieved.

18. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12, or the steps of the method according to claim 13, or the steps of the method according to claim 14 are implemented.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12, or the steps of the method according to claim 13, or the steps of the method according to claim 14 are implemented.

20. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12, or the steps of the method according to claim 13, or the steps of the method according to claim 14 are implemented.

Images