Text classification method and related equipment
By performing block-based parallel clustering and cluster merging operations on the text, the problems of memory consumption and computation time when large models process large-scale data are solved, and more efficient text classification is achieved.
Patent Information
- Application Number
- CN202511070783.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-31
- Publication Date
- 2025-11-11
AI Technical Summary
When using large models to classify large amounts of text, the memory usage and computation time of the large models increase linearly with the increase in data volume, resulting in high computational costs and making it difficult to process large-scale data in real time.
By dividing multiple target texts into blocks and performing clustering and cluster merging operations in parallel, the target classification results are obtained.
It improves the speed and efficiency of large models in processing large amounts of data and reduces computational costs.
Smart Images
Figure CN120929985A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing technology, and in particular to text classification methods and related equipment. Background Technology
[0002] With the maturity of large model technology, using large models to handle a wide variety of tasks has become the current mainstream trend.
[0003] However, in relevant text classification scenarios, when using large models to classify a large amount of text, the memory usage and computation time of large models increase linearly with the increase in data volume, resulting in high computational costs and making it difficult to process large-scale data in real time.
[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention
[0005] The main purpose of this application is to provide a text classification method and related equipment, aiming to solve the technical problem of how to improve the speed of large-scale models in processing large-scale data.
[0006] To achieve the above objectives, this application proposes a text classification method, which includes:
[0007] In response to a text classification instruction, obtain multiple target texts to be classified;
[0008] Based on the preset number of texts in each block, the multiple target texts are divided into blocks to obtain multiple target blocks;
[0009] Clustering operations are performed in parallel on the multiple target blocks to obtain multiple first classification results;
[0010] A cluster merging operation is performed on the multiple first classification results to obtain the target classification result.
[0011] In one embodiment, the step of performing clustering operations in parallel on the plurality of target blocks to obtain a plurality of first classification results further includes:
[0012] Based on the preset number of texts in each block, coarse-grained clustering operations are performed in parallel on the multiple target blocks to obtain a second classification result;
[0013] The number of texts corresponding to each category in the second classification result is dynamically adjusted to perform fine-grained clustering operations and obtain multiple first classification results.
[0014] In one embodiment, the step of obtaining multiple target texts to be classified in response to a text classification instruction further includes:
[0015] In response to a text classification instruction, obtain multiple initial texts to be classified;
[0016] A cleaning operation is performed on the multiple initial texts to obtain multiple first texts;
[0017] Part-of-speech filtering and stop word removal operations are performed sequentially on the multiple first texts to obtain multiple target texts.
[0018] In one embodiment, the step of performing a cluster merging operation on the plurality of first classification results to obtain the target classification result further includes:
[0019] Based on an automatic dictionary, multiple sets of high-frequency compound words corresponding to the multiple target texts are identified;
[0020] Based on the aforementioned multiple sets of high-frequency compound words, the relevance of the target text in different blocks is calculated;
[0021] Based on the correlation, a cluster merging operation is performed on the multiple first classification results to obtain the target classification result.
[0022] In one embodiment, the step of performing a cluster merging operation on the plurality of first classification results based on the correlation degree to obtain the target classification result further includes:
[0023] Based on the correlation degree, a cluster merging operation is performed on the multiple first classification results to obtain the second classification result;
[0024] Based on the preset threshold for the number of texts contained in the target category and the number of texts contained in each category in the second classification result, the target category in the second classification result is determined;
[0025] Delete the target category from the second classification result to obtain the target classification result.
[0026] In one embodiment, after the step of performing a cluster merging operation on the plurality of first classification results to obtain the target classification result, the method further includes:
[0027] Based on the structured prompt, a preset large language model is invoked to process the target classification results and determine the semantic label corresponding to each category in the target classification results;
[0028] Based on the semantic labels, each category in the target classification result is labeled to obtain the target classification result with semantic labels.
[0029] In one embodiment, prior to the structured prompt-based step, the method further includes:
[0030] Based on the cosine similarity algorithm, N groups of second texts with the highest semantic diversity for each category in the target classification results are determined, where the value of N is predetermined.
[0031] Based on the preset prompt template, the second text is processed to obtain a structured prompt.
[0032] Furthermore, to achieve the above objectives, this application also proposes a text classification device, which includes:
[0033] An acquisition module is used to acquire multiple target texts to be classified in response to a text classification instruction.
[0034] The segmentation module is used to segment the multiple target texts into multiple target segments based on a preset number of texts in each segment.
[0035] A clustering module is used to perform clustering operations on the multiple target blocks in parallel to obtain multiple first classification results;
[0036] The cluster merging module is used to perform a cluster merging operation on the multiple first classification results to obtain the target classification result.
[0037] In one embodiment, the clustering module further includes:
[0038] The clustering unit is used to perform coarse-grained clustering operations in parallel on the multiple target blocks based on the preset number of texts in each block, so as to obtain a second classification result;
[0039] The adjustment unit is used to dynamically adjust the number of texts corresponding to each category in the second classification result in order to perform fine-grained clustering operations and obtain multiple first classification results.
[0040] In one embodiment, the acquisition module further includes:
[0041] The acquisition unit is used to acquire multiple initial texts to be classified in response to a text classification instruction;
[0042] A cleaning unit is used to perform cleaning operations on the plurality of initial texts to obtain a plurality of first texts;
[0043] The first data processing unit is used to sequentially perform part-of-speech filtering and stop word removal operations on the plurality of first texts to obtain a plurality of target texts.
[0044] In one embodiment, the cluster merging module further includes:
[0045] The first determining unit is used to determine multiple sets of high-frequency compound words corresponding to the multiple target texts based on an automatic dictionary;
[0046] The calculation unit is used to calculate the relevance of the target text in different blocks based on the multiple sets of high-frequency compound words;
[0047] The first cluster merging unit is used to perform a cluster merging operation on the multiple first classification results based on the correlation degree to obtain the target classification result.
[0048] In one embodiment, the cluster merging module further includes:
[0049] The second cluster merging unit is used to perform a cluster merging operation on the multiple first classification results based on the correlation degree to obtain a second classification result;
[0050] The second determining unit is used to determine the target category in the second classification result based on a preset threshold for the number of texts contained in the target category and the number of texts contained in each category in the second classification result;
[0051] The deletion unit is used to delete the target category from the second classification result to obtain the target classification result.
[0052] In one embodiment, the text classification device further includes a semantic tag generation module, which further includes:
[0053] The second data processing unit is used to process the target classification results by calling a preset large language model based on the structured prompt, and to determine the semantic label corresponding to each category in the target classification results;
[0054] A labeling unit is used to label each category in the target classification result based on the semantic label, so as to obtain a target classification result with semantic labels.
[0055] In one embodiment, the text classification device further includes a prompt generation module, which further includes:
[0056] The third determining unit is used to determine, based on the cosine similarity algorithm, the N groups of second texts with the highest semantic diversity for each category in the target classification results, wherein the value of N is preset.
[0057] The structuring unit is used to perform structuring processing on the second text based on a preset prompt template to obtain a structured prompt.
[0058] In addition, to achieve the above objectives, this application also proposes a text classification device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the text classification method as described above.
[0059] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and which, when executed by a processor, implements the steps of the text classification method described above.
[0060] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the text classification method described above.
[0061] One or more technical solutions proposed in this application have at least the following technical effects:
[0062] This application proposes a text classification method and related equipment, relating to the field of natural language processing technology. Compared with related technologies, where large models are used to classify large amounts of text, the memory usage and computation time of the large models increase linearly with the increase of data volume, resulting in high computational costs and difficulty in real-time processing of large-scale data, in this application, firstly, in response to a text classification instruction, multiple target texts to be classified are obtained; then, based on a preset number of texts in each block, the multiple target texts are divided into blocks to obtain multiple target blocks; further, clustering operations are performed in parallel on the multiple target blocks to obtain multiple first classification results; finally, cluster merging operations are performed on the multiple first classification results to obtain the target classification result.
[0063] Understandably, this application divides multiple target texts into blocks and performs clustering operations on multiple target blocks in parallel to improve the efficiency of clustering, thereby increasing the speed at which large models can process large amounts of data. Attached Figure Description
[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0065] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0066] Figure 1This is a flowchart illustrating an embodiment of the text classification method of this application.
[0067] Figure 2 This is a schematic diagram of the text classification process provided in Embodiment 1 of the text classification method of this application;
[0068] Figure 3 This is a flowchart illustrating Embodiment 2 of the text classification method of this application;
[0069] Figure 4 This is a schematic diagram of the module structure of the text classification device according to an embodiment of this application;
[0070] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the text classification method in the embodiments of this application.
[0071] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0072] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.
[0073] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.
[0074] The main solution in this application embodiment is:
[0075] In this embodiment, for ease of description, the following description uses a text classification device as the execution subject.
[0076] Due to existing technologies, when using large models to classify large amounts of text, the memory usage and computation time of large models increase linearly with the increase in data volume, resulting in high computational costs and making it difficult to process large-scale data in real time.
[0077] This application provides a solution in which: first, in response to a text classification instruction, multiple target texts to be classified are obtained; then, based on a preset number of texts in each block, the multiple target texts are divided into blocks to obtain multiple target blocks; further, a clustering operation is performed in parallel on the multiple target blocks to obtain multiple first classification results; finally, a cluster merging operation is performed on the multiple first classification results to obtain a target classification result.
[0078] Understandably, this application divides multiple target texts into blocks and performs clustering operations on these blocks in parallel to improve clustering efficiency, thereby increasing the speed at which large models can process large amounts of data. ......
[0079] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or text classification device capable of performing the above functions. The following description uses a text classification device as an example to illustrate this embodiment and the subsequent embodiments.
[0080] Based on this, embodiments of this application provide a text classification method, referring to... Figure 1 and Figure 2 , Figure 1 This is a flowchart illustrating the first embodiment of the text classification method of this application. Figure 2 This is a schematic diagram of the text classification process provided in Embodiment 1 of the text classification method of this application.
[0081] In this embodiment, the text classification method includes steps S100 to S400:
[0082] Step S100: In response to the text classification instruction, obtain multiple target texts to be classified;
[0083] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or text classification device capable of performing the above functions. The following description uses a text classification device as an example to illustrate this embodiment and the subsequent embodiments.
[0084] It's important to note that in the computer field, an instruction is a command that a computer uses to perform an operation. Here, "text classification instruction" refers to a specific instruction whose purpose is to instruct the system to perform the task of text classification. For example, in a text processing system, a user might initiate a text classification operation by clicking a button or entering a specific command; this initiation of the operation is the "instruction."
[0085] In this embodiment, when the system receives a text classification instruction, it will initiate a series of pre-set programs or algorithms to execute the instruction. For example, the system may check the legality of the instruction, prepare the necessary resources (such as memory, computing power, etc.), and then begin to execute the relevant text classification operations.
[0086] Target text refers to the text content that needs to be classified. This text can be of various types, such as news articles, social media posts, user comments, product descriptions, etc. It is the input data for text classification tasks.
[0087] "Multiple" indicates that there is not just one text, but multiple texts that need to be classified. This means that the system is processing a batch of text data, rather than a single text. This is common in practical applications because it is often necessary to perform batch classification on large amounts of text to improve efficiency and processing capacity.
[0088] "Acquisition" refers to the process by which a system obtains target text from a certain place or in a certain way. There can be many ways to acquire text, such as reading from a database, loading from a file system, or receiving it through a network interface. For example, a text classification system might acquire the text to be classified from a database storing news articles, or read text files uploaded by users from a folder.
[0089] Specifically, the step of obtaining multiple target texts to be classified in response to a text classification instruction further includes steps S110 to S130:
[0090] Step S110: In response to the text classification instruction, obtain multiple initial texts to be classified;
[0091] It should be noted that initial text refers to the text data that the system first acquires at the start of the text classification task. This text is raw text without any preprocessing or filtering and is the starting point of the text classification process. For example, text content read directly from a text data source; this text may contain various formats, lengths, and contents, and is raw data that has not been filtered or preprocessed.
[0092] Step S120: Perform a cleaning operation on the plurality of initial texts to obtain a plurality of first texts;
[0093] It should be noted that in text processing, cleaning refers to preprocessing the original text to remove noise, correct errors, and standardize formatting, making the text data cleaner and more standardized, thus facilitating subsequent processing and analysis. In this embodiment, cleaning includes, but is not limited to, deleting null values from the initial text.
[0094] The first text refers to the text data obtained after cleaning operations. This text has been cleaned, with noise removed, errors corrected, and formatted to be cleaner and more standardized, making it suitable for subsequent text processing tasks such as classification, clustering, and sentiment analysis.
[0095] Step S130: Perform part-of-speech filtering and stop word removal operations sequentially on the plurality of first texts to obtain a plurality of target texts.
[0096] The词性filter operation refers to screening the words in the text according to the word type (such as nouns, verbs, adjectives, etc.), and retaining or removing words of certain specific word types. For example, if the task is to extract entities (usually nouns) in the text, nouns can be retained and words of other word types can be removed. The词性filter operation usually needs to借助natural language processing tools (such as word segmentation tools) to实现. For example, using a word segmentation tool to segment and label the word type of the sentence "I love natural language processing", the result is "I (pronoun) / love (verb) / natural language processing (noun)", and then retain or remove words of certain word types according to the needs.
[0097] The stop word removal operation refers to deleting some common words from the text that are not helpful for text processing tasks (such as classification, sentiment analysis, etc.), such as "的", "是", "在", etc. These words appear frequently in the text but usually do not carry important semantic information. The stop word list is usually predefined, and different languages and application fields may have different stop word lists. For example, in Chinese texts, common stop words include "的", "是", "在", "和", etc.; in English texts, common stop words include "the", "is", "in", "and", etc.
[0098] The target text refers to the text data obtained after the词性filter and stop word removal operations. These texts are more concise and targeted, retaining the words useful for text processing tasks and removing irrelevant words, and are suitable for subsequent text processing tasks, such as classification, clustering, sentiment analysis, etc.
[0099] It should also be noted that in this application, first, after cleaning the input short text (deleting special categories of null values), then a sparse matrix is generated by TF-IDF vectorization, and a high-dimensional feature space is constructed through词性filtering (retaining nouns, verbs, interrogative words) and stop word removal. Finally, the feature matrix is stored in a compressed manner (such as CSR format) to obtain a compressed text vector. This compressed vector is the basis for clustering in the subsequent process.
[0100] Step S200, based on the preset number of texts in each block, divide the multiple target texts into blocks to obtain multiple target blocks;
[0101] The preset number of texts refers to the number of texts that should be included in each block preset before block division. This number can be determined according to actual needs and processing capabilities. For example, if each thread of the system can efficiently process 1000 texts each time, then the size of each block can be preset to 1000 texts.
[0102] Chunking refers to dividing the target text into multiple smaller blocks according to a preset number of texts. The purpose of chunking is to facilitate subsequent processing, such as parallel processing and batch processing, and to improve processing efficiency. For example, if there are 10,000 target texts and the preset block size is 1,000 texts, then these 10,000 texts can be divided into 10 blocks, each containing 1,000 texts.
[0103] Target chunks refer to the text blocks obtained after chunking operations. Each target chunk contains a certain number of target texts, which can be processed separately to facilitate subsequent text analysis, classification, clustering, and other tasks.
[0104] In this embodiment, the system divides multiple target texts into multiple target blocks according to a preset number of texts that each block should contain. Each target block contains a fixed number of target texts, and these blocks can be processed individually, facilitating subsequent text processing tasks such as parallel processing and batch processing, thereby improving processing efficiency.
[0105] Step S300: Perform clustering operations in parallel on the multiple target blocks to obtain multiple first classification results;
[0106] Parallel execution refers to performing clustering operations on multiple target blocks simultaneously, rather than processing them sequentially. Parallel processing can significantly improve processing efficiency, especially when dealing with large amounts of data. Parallel execution typically requires multithreading or distributed computing resources. For example, in a distributed computing environment, each target block can be distributed across different computing nodes for simultaneous clustering operations.
[0107] Clustering is the process of grouping text data into different groups (or categories) based on their similarity or features. Clustering is an unsupervised learning method, and common clustering algorithms include K-means, DBSCAN, and hierarchical clustering. In text clustering, the similarity between texts is typically calculated based on text features (such as term frequency, TF-IDF values, etc.), and then similar texts are grouped into the same category.
[0108] The first classification result refers to the classification result obtained after clustering each target block. These classification results are preliminary clustering results and require further processing or verification. The clustering result of each target block is independent, that is, the text within each block is divided into different categories.
[0109] In this embodiment, the system performs clustering operations on multiple target blocks simultaneously, with each block being clustered independently, ultimately yielding multiple preliminary classification results. Parallel execution of clustering operations can significantly improve processing efficiency, especially when processing large amounts of data.
[0110] Specifically, the step of performing clustering operations in parallel on the multiple target blocks to obtain multiple first classification results further includes steps S310 to S320:
[0111] Step S310: Based on the preset number of texts in each block, perform coarse-grained clustering operations in parallel on the multiple target blocks to obtain the second classification result;
[0112] Coarse-grained clustering refers to dividing text data into broad categories rather than very fine classifications. It's typically used for initial classification, aiming to quickly group text into several large categories for subsequent, more detailed processing. For example, in news classification, coarse-grained clustering might divide text into broad categories like "sports," "finance," and "technology," rather than specific subcategories like "basketball," "football," "stocks," and "funds." Coarse-grained clustering operations usually use simple clustering algorithms or fewer features to improve processing speed.
[0113] The second classification result refers to the classification result obtained after coarse-grained clustering of each target block. These classification results are preliminary clustering results, usually representing relatively broad categories. The clustering result for each target block is independent, meaning that the text within each block is divided into different major categories.
[0114] Specifically, in this embodiment, based on the preset number of texts in each block, the number of basic clusters k with low complexity is calculated. means_n Then based on the number of basic clusters k means_n Coarse-grained clustering is then performed. The specific calculation formula is as follows:
[0115]
[0116] Where num_vectors is the total number of text vectors, k means_n It is a positive integer.
[0117] In this embodiment, the system performs coarse-grained clustering on multiple target blocks simultaneously, based on the preset number of texts that each block should contain. Each block is clustered independently, resulting in multiple preliminary classification results. These classification results are relatively broad categories, typically used for preliminary classification, to facilitate more detailed processing later.
[0118] Step S320: Dynamically adjust the number of texts corresponding to each category in the second classification result to perform fine-grained clustering operation and obtain multiple first classification results.
[0119] Dynamic adjustment refers to adjusting the number of texts in each category according to certain rules or conditions. This adjustment may be based on factors such as category size, text distribution, and clustering quality. The purpose of dynamic adjustment is to optimize the clustering results, making the number of texts in each category more reasonable, which facilitates subsequent fine-grained clustering operations.
[0120] The number of texts per category refers to the total number of texts contained within each category. After coarse-grained clustering, the number of texts in each category may be uneven, with some categories containing more text and others containing fewer. Dynamic adjustment aims to make these numbers more balanced or to meet specific optimization objectives.
[0121] Fine-grained clustering refers to further refining the clustering of text within each category compared to coarse-grained clustering. Fine-grained clustering typically uses more features, more complex algorithms, or stricter similarity criteria to divide the text into more specific subcategories. For example, in news classification, coarse-grained clustering might divide the text into broad categories such as "sports," "finance," and "technology," while fine-grained clustering might further subdivide the "sports" category into subcategories such as "basketball," "football," and "tennis."
[0122] Specifically, HDBSCAN is then executed separately for each basic cluster, dynamically adjusting the min_cluster_size parameter (e.g., 20% of the basic cluster size) to achieve fine-grained partitioning. Two-stage clustering is accelerated through parallel computation, reducing overall time consumption. The parameters are dynamically determined by the following formula:
[0123]
[0124] Wherein, min_cluster_size is the minimum number of texts per cluster in the HDBSCAN algorithm.
[0125] In this embodiment, the system first dynamically adjusts the results of coarse-grained clustering to optimize the number of texts in each category. Then, based on the adjusted categories, fine-grained clustering is performed, ultimately yielding multiple more detailed classification results. This process aims to improve the accuracy and quality of clustering, making the classification results more consistent with actual needs.
[0126] Step S400: Perform a cluster merging operation on the multiple first classification results to obtain the target classification result.
[0127] It's important to note that cluster merging refers to combining certain clusters (or categories) from multiple fine-grained clustering results into a larger category. The purpose of cluster merging is to reduce the number of categories and improve the interpretability and usability of the classification results. The merging is usually based on the similarity or correlation between clusters. For example, if the subcategories "basketball" and "football" both semantically belong to the broader category of "sports," they can be merged into a larger "sports" category.
[0128] Specifically, the step of performing a cluster merging operation on the plurality of first classification results to obtain the target classification result further includes steps S410 to S430:
[0129] Step S410: Based on an automatic dictionary, determine multiple sets of high-frequency compound words corresponding to the multiple target texts;
[0130] An automatic dictionary is a pre-built dictionary that typically contains a large vocabulary and related information (such as parts of speech, word frequency, and word meaning). Automatic dictionaries can be general-purpose or customized for specific domains or tasks. In text processing, automatic dictionaries are commonly used for tasks such as word segmentation, part-of-speech tagging, and word meaning disambiguation.
[0131] High-frequency compound words refer to phrases or groups of words that appear frequently in text. Compound words usually have specific semantic or grammatical structures, such as "natural language processing," "machine learning," and "deep learning." Identifying high-frequency compound words is crucial for tasks such as text analysis, topic extraction, and sentiment analysis.
[0132] "Multiple groups" means that each target text may contain multiple high-frequency compound words, which can be organized into different groups. For example, a target text may contain two high-frequency compound words, "natural language processing" and "machine learning," which can be considered as a group.
[0133] "Identification" refers to the process of using an algorithm or method to recognize and extract these high-frequency compound words from the target text. This process typically requires the assistance of information from an automatic dictionary, such as using word frequency information in the dictionary to determine which word phrases are high-frequency.
[0134] Step S420: Based on the multiple sets of high-frequency compound words, calculate the relevance of the target text in different blocks;
[0135] The target text within different blocks refers to the target text within each block after the text segmentation operation. This text has been divided into multiple smaller blocks, each containing a certain number of target texts. When calculating relevance, the relationships between the texts in different blocks need to be considered.
[0136] Relevance refers to the degree of similarity or relevance between two or more texts. Relevance can be calculated using various methods, common ones including word frequency-based similarity measures (such as cosine similarity) and topic model-based similarity measures (such as LDA topic modeling). The result of a relevance calculation is usually a numerical value representing the degree of similarity between the texts.
[0137] In this embodiment, the system uses high-frequency compound words extracted from the target text to evaluate the correlation between target texts in different segments. By calculating the correlation, the similarity of texts in different segments in terms of topic or content can be understood, which is very helpful for subsequent text clustering, classification, or other analysis tasks.
[0138] Step S430: Based on the correlation degree, perform a cluster merging operation on the multiple first classification results to obtain the target classification result.
[0139] In this embodiment, the system uses the correlation between texts as a basis to perform cluster merging operations on multiple fine-grained classification results, combining similar or related subcategories into larger categories, ultimately obtaining a higher-level target classification result. This process aims to optimize the classification results to better meet the needs of practical applications.
[0140] Specifically, the step of performing a cluster merging operation on the multiple first classification results based on the correlation degree to obtain the target classification result further includes steps S431 to S433:
[0141] Step S431: Based on the correlation degree, perform a cluster merging operation on the multiple first classification results to obtain a second classification result;
[0142] Obtaining a second classification result involves merging similar or related subcategories into a larger category through cluster merging operations, resulting in a higher-level classification. The second classification result typically has fewer categories than the first classification result, but each category has richer and more complete semantics.
[0143] Step S432: Based on the preset threshold for the number of texts contained in the target category and the number of texts contained in each category in the second classification result, determine the target category in the second classification result;
[0144] The preset threshold for the number of texts in each target category is a pre-defined threshold used to determine which categories are "target categories". This threshold can be determined based on actual needs and application scenarios. For example, if you want each target category to contain at least 10 texts, you can set the threshold to 10.
[0145] The number of texts in each category of the second classification result refers to the total number of texts in each category within the second classification result. By counting the number of texts in each category, it is possible to determine which categories meet the preset threshold.
[0146] In this embodiment, categories that meet certain conditions in the second classification results are selected based on a preset threshold. These categories are referred to as "target categories". For example, if the number of texts in a certain category is less than a preset threshold, then that category is a target category.
[0147] Step S433: Delete the target category from the second classification result to obtain the target classification result.
[0148] Furthermore, the target categories identified in step S432 are removed from the second classification results. The purpose of this step is to further optimize the classification results and remove those categories that do not meet the requirements or are unnecessary.
[0149] After the deletion operation, the final target classification result is obtained. This result is optimized and better reflects the needs of practical applications.
[0150] This application proposes a text classification method and related equipment, relating to the field of natural language processing technology. Compared with related technologies, where large models are used to classify large amounts of text, the memory usage and computation time of the large models increase linearly with the increase of data volume, resulting in high computational costs and difficulty in real-time processing of large-scale data, in this application, firstly, in response to a text classification instruction, multiple target texts to be classified are obtained; then, based on a preset number of texts in each block, the multiple target texts are divided into blocks to obtain multiple target blocks; further, clustering operations are performed in parallel on the multiple target blocks to obtain multiple first classification results; finally, cluster merging operations are performed on the multiple first classification results to obtain the target classification result.
[0151] Understandably, this application divides multiple target texts into blocks and performs clustering operations on multiple target blocks in parallel to improve the efficiency of clustering, thereby increasing the speed at which large models can process large amounts of data.
[0152] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in Embodiment 1 above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 3 After the step of performing a cluster merging operation on the multiple first classification results to obtain the target classification result, the method further includes steps A10 to A20:
[0153] Step A10: Based on the structured prompt, call the preset large language model to process the target classification result and determine the semantic label corresponding to each category in the target classification result;
[0154] A structured prompt is a carefully crafted text prompt used to guide a large language model in generating specific outputs. A structured prompt typically contains explicit instructions and contextual information to ensure the model understands the task requirements and produces the expected results. For example, a structured prompt might include a description of a classification task, example inputs, and the expected output format.
[0155] Pre-trained large language models refer to models that are pre-trained and possess powerful language understanding and generation capabilities, such as GPT-3, GPT-4, and Wenxin Yiyan. These models are typically trained on large amounts of text data and are capable of understanding and generating natural language text.
[0156] A semantic tag is a concise label that summarizes the content of a category. A semantic tag is typically one or a few words that accurately reflect the topic or content of the category. For example, if a category contains multiple texts related to "natural language processing," then "natural language processing" can serve as the semantic tag for that category.
[0157] It should also be noted that after obtaining semantic tags, the system will further filter out low-confidence tags (such as those that deviate too much from the semantics of the input text) and supplement them with manual rules for correction based on the actual scenario.
[0158] In this embodiment, a suitable semantic label is assigned to each category based on the output of the large language model. This process typically requires parsing and filtering the model's output to ensure the accuracy and consistency of the labels.
[0159] Specifically, before the step based on the structured prompt, steps A01 to A02 are also included:
[0160] Step A01: Based on the cosine similarity algorithm, determine the N groups of second texts with the highest semantic diversity for each category in the target classification results, where the value of N is pre-set;
[0161] It's important to note that cosine similarity is a commonly used method for measuring text similarity. It measures the similarity between two text vectors by calculating the cosine of the angle between them. Cosine similarity values range from 0 to 1, with values closer to 1 indicating higher similarity. In this step, the cosine similarity algorithm is used to evaluate the similarity between texts.
[0162] In this embodiment, within each category, N groups of texts with the highest semantic diversity are selected. Semantic diversity can be assessed by calculating the similarity between texts, and those texts with lower similarity to other texts are selected, i.e., those texts that are more semantically representative and diverse.
[0163] Here, N is a pre-defined value representing the number of text groups to be selected in each category. This value can be determined based on actual needs and application scenarios. For example, if you want to select 5 of the most representative text groups in each category, then you can set N to 5.
[0164] Step A02: Based on the preset prompt template, the second text is processed to obtain a structured prompt.
[0165] It's important to note that the preset prompt template is a pre-designed text template used to convert the extracted text into a structured prompt. This template typically contains specific formatting and instructions to ensure that the generated prompt guides the large language model to produce the expected output.
[0166] In this embodiment, the second text extracted in step A01 is transformed according to a preset prompt template to generate a structured prompt. This process typically involves text reorganization, formatting, and supplementing with necessary contextual information.
[0167] The final generated structured prompt is a formatted text that can be directly input into a large language model to generate specific outputs. The structured prompt typically contains explicit instructions and contextual information to ensure the model understands the task requirements and generates the expected results.
[0168] Understandably, in this embodiment, text with high semantic diversity is extracted from the classification results and converted into a structured prompt for subsequent processing and analysis.
[0169] Step A20: Based on the semantic labels, each category in the target classification result is labeled to obtain the target classification result with semantic labels.
[0170] Tagging each category involves assigning semantic tags to each category and associating these tags with the categories. The tagging process typically involves attaching tags to categories for subsequent processing and analysis. The purpose of tagging is to make the content of each category clearer and easier to understand.
[0171] Semantically labeled classification results refer to classification results where each category is accompanied by a semantic label after being tagged. These labels make the classification results easier to understand and use, facilitating subsequent analysis, display, or further processing.
[0172] In this embodiment, the system uses the semantic labels determined in the previous steps to assign these labels to each category in the target classification result and associates the labels with the categories, ultimately obtaining the target classification result with semantic labels. This process makes the classification result clearer and easier to understand, facilitating subsequent processing and analysis.
[0173] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the text classification method of this application. Any simple transformations based on this technical concept are within the protection scope of this application.
[0174] It should be noted that all user-related data involved in this application (such as user attribute data, user behavior data, and user geographical location, etc., the data types here should be modified according to the adaptability of the solution content) were obtained after obtaining the user's permission or consent; that is to say, when this application is applied to specific products or technologies, user permission is required to obtain and process the relevant data, and the processing of the relevant data must comply with the relevant laws, regulations and regulatory standards of the relevant countries and regions.
[0175] For example, when it is necessary to obtain a user's current geographical location, a location acquisition prompt can be displayed on the user's terminal. After receiving confirmation from the user regarding the location acquisition prompt, the terminal can obtain the user's current geographical location.
[0176] This application also provides a text classification device, please refer to... Figure 4 The text classification device includes:
[0177] Acquisition module 10, the acquisition module is used to acquire multiple target texts to be classified in response to a text classification instruction;
[0178] Segmentation module 20, the segmentation module is used to segment the multiple target texts into multiple target segments based on the preset number of texts in each segment;
[0179] Clustering module 30, which is used to perform clustering operations on the multiple target blocks in parallel to obtain multiple first classification results;
[0180] Cluster merging module 40 is used to perform cluster merging operations on the multiple first classification results to obtain the target classification result.
[0181] In one embodiment, the clustering module further includes:
[0182] The clustering unit is used to perform coarse-grained clustering operations in parallel on the multiple target blocks based on the preset number of texts in each block, so as to obtain a second classification result;
[0183] The adjustment unit is used to dynamically adjust the number of texts corresponding to each category in the second classification result in order to perform fine-grained clustering operations and obtain multiple first classification results.
[0184] In one embodiment, the acquisition module further includes:
[0185] The acquisition unit is used to acquire multiple initial texts to be classified in response to a text classification instruction;
[0186] A cleaning unit is used to perform cleaning operations on the plurality of initial texts to obtain a plurality of first texts;
[0187] The first data processing unit is used to sequentially perform part-of-speech filtering and stop word removal operations on the plurality of first texts to obtain a plurality of target texts.
[0188] In one embodiment, the cluster merging module further includes:
[0189] The first determining unit is used to determine multiple sets of high-frequency compound words corresponding to the multiple target texts based on an automatic dictionary;
[0190] The calculation unit is used to calculate the relevance of the target text in different blocks based on the multiple sets of high-frequency compound words;
[0191] The first cluster merging unit is used to perform a cluster merging operation on the multiple first classification results based on the correlation degree to obtain the target classification result.
[0192] In one embodiment, the cluster merging module further includes:
[0193] The second cluster merging unit is used to perform a cluster merging operation on the multiple first classification results based on the correlation degree to obtain a second classification result;
[0194] The second determining unit is used to determine the target category in the second classification result based on a preset threshold for the number of texts contained in the target category and the number of texts contained in each category in the second classification result;
[0195] The deletion unit is used to delete the target category from the second classification result to obtain the target classification result.
[0196] In one embodiment, the text classification device further includes a semantic tag generation module, which further includes:
[0197] The second data processing unit is used to process the target classification results by calling a preset large language model based on the structured prompt, and to determine the semantic label corresponding to each category in the target classification results;
[0198] The labeling unit is used to label each category in the target classification result based on the semantic label, so as to obtain the target classification result with semantic label.
[0199] In one embodiment, the text classification device further includes a prompt generation module, which further includes:
[0200] The third determining unit is used to determine, based on the cosine similarity algorithm, the N groups of second texts with the highest semantic diversity for each category in the target classification results, wherein the value of N is preset.
[0201] The structuring unit is used to perform structuring processing on the second text based on a preset prompt template to obtain a structured prompt.
[0202] The text classification device provided in this application, employing the text classification method in the above embodiments, can solve the technical problem of text classification. Compared with the prior art, the beneficial effects of the text classification device provided in this application are the same as those of the text classification method provided in the above embodiments, and other technical features in the text classification device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0203] This application provides a text classification device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the text classification method in Embodiment 1 above.
[0204] The following is for reference. Figure 5 The diagram illustrates a structural schematic of a text classification device suitable for implementing embodiments of this application. The text classification device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The text classification device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0205] like Figure 5As shown, the text classification device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the text classification device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the text classification device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show text classification devices with various systems, it should be understood that it is not required to implement or possess all of the systems shown. More or fewer systems may be implemented alternatively.
[0206] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0207] The text classification device provided in this application, employing the text classification method described in the above embodiments, can solve the technical problems. Compared with the prior art, the beneficial effects of the text classification device provided in this application are the same as those of the text classification method provided in the above embodiments, and other technical features of this text classification device are the same as those disclosed in the previous embodiment method, and will not be repeated here.
[0208] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0209] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0210] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the text classification method in the above embodiments.
[0211] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0212] The aforementioned computer-readable storage medium may be included in the text classification device; or it may exist independently and not be assembled into the text classification device.
[0213] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the text classification device, cause the text classification device to:
[0214] In response to a text classification instruction, obtain multiple target texts to be classified;
[0215] Based on the preset number of texts in each block, the multiple target texts are divided into blocks to obtain multiple target blocks;
[0216] Clustering operations are performed in parallel on the multiple target blocks to obtain multiple first classification results;
[0217] A cluster merging operation is performed on the multiple first classification results to obtain the target classification result.
[0218] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0219] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0220] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0221] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described text classification method, thereby solving the technical problem of text classification. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the text classification method provided in the above embodiments, and will not be repeated here.
[0222] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the text classification method described above.
[0223] The computer program product provided in this application can solve the technical problem of text classification. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the text classification method provided in the above embodiments, and will not be repeated here.
[0224] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. A text classification method, characterized in that, The text classification method includes: In response to a text classification instruction, obtain multiple target texts to be classified; Based on the preset number of texts in each block, the multiple target texts are divided into blocks to obtain multiple target blocks; Clustering operations are performed in parallel on the multiple target blocks to obtain multiple first classification results; A cluster merging operation is performed on the multiple first classification results to obtain the target classification result.
2. The text classification method as described in claim 1, characterized in that, The step of performing clustering operations in parallel on the multiple target blocks to obtain multiple first classification results further includes: Based on the preset number of texts in each block, coarse-grained clustering operations are performed in parallel on the multiple target blocks to obtain a second classification result; The number of texts corresponding to each category in the second classification result is dynamically adjusted to perform fine-grained clustering operations and obtain multiple first classification results.
3. The text classification method as described in claim 1, characterized in that, The step of obtaining multiple target texts to be classified in response to a text classification instruction further includes: In response to a text classification instruction, obtain multiple initial texts to be classified; A cleaning operation is performed on the multiple initial texts to obtain multiple first texts; Part-of-speech filtering and stop word removal operations are performed sequentially on the multiple first texts to obtain multiple target texts.
4. The text classification method as described in claim 1, characterized in that, The step of performing a cluster merging operation on the plurality of first classification results to obtain the target classification result further includes: Based on an automatic dictionary, multiple sets of high-frequency compound words corresponding to the multiple target texts are identified; Based on the aforementioned multiple sets of high-frequency compound words, the relevance of the target text in different blocks is calculated; Based on the correlation, a cluster merging operation is performed on the multiple first classification results to obtain the target classification result.
5. The text classification method as described in claim 4, characterized in that, The step of performing a cluster merging operation on the multiple first classification results based on the correlation degree to obtain the target classification result further includes: Based on the correlation degree, a cluster merging operation is performed on the multiple first classification results to obtain the second classification result; Based on the preset threshold for the number of texts contained in the target category and the number of texts contained in each category in the second classification result, the target category in the second classification result is determined; Delete the target category from the second classification result to obtain the target classification result.
6. The text classification method as described in claim 1, characterized in that, After the step of performing a cluster merging operation on the multiple first classification results to obtain the target classification result, the method further includes: Based on the structured prompt, a preset large language model is invoked to process the target classification results and determine the semantic label corresponding to each category in the target classification results; Based on the semantic labels, each category in the target classification result is labeled to obtain the target classification result with semantic labels.
7. The text classification method as described in claim 6, characterized in that, Before the structured prompt-based step, the following steps are also included: Based on the cosine similarity algorithm, N groups of second texts with the highest semantic diversity for each category in the target classification results are determined, where the value of N is predetermined. Based on the preset prompt template, the second text is processed to obtain a structured prompt.
8. A text classification device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the text classification method as described in any one of claims 1 to 7.
9. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the text classification method as described in any one of claims 1 to 7.
10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the text classification method as described in any one of claims 1 to 7.