Short text clustering method and device based on large language model

Abstract

The invention discloses a short text clustering method and device based on a large language model, and belongs to the technical field of natural language processing, and the method comprises the steps: converting a short text into vector representation, and carrying out the preliminary clustering of the vector representation through a clustering algorithm, and obtaining an initial category; for all the corresponding short texts in each initial category, generating an abstract by using a large language model, generating a plurality of theme titles for summarizing the core meanings of the texts according to the abstract, respectively taking the theme titles as sub-titles, inputting the plurality of sub-titles into the large language model, and carrying out theme combination to obtain a parent title, constructing the parent title and the child titles into a hierarchical theme system; the large language model allocates to-be-classified short texts to one or more categories according to the hierarchical topic system to complete short text clustering; and performing quality evaluation and error pattern analysis on a short text clustering result to optimize a clustering algorithm and a large language model. According to the method, the theme of the short text can be automatically found and classified, and the classification accuracy and efficiency are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of natural language processing, and in particular relates to a short text clustering method and device based on a large language model. Background Art

[0002] In the field of natural language processing, automatic classification of short texts is an important and challenging task. With the rapid development of the Internet and social media, the speed of information generation and dissemination has reached an unprecedented level, and a huge amount of short text data is generated every day. These short texts contain rich information and have important application value in data analysis, information retrieval, knowledge management and other aspects. Therefore, how to automatically classify these short texts efficiently and accurately has become a key problem that needs to be solved in the field of natural language processing.

[0003] Traditional text classification methods rely heavily on predefined category systems. They perform well when dealing with text classification tasks with a clear and fixed set of categories, and can learn the feature differences between categories based on training data to achieve classification. However, in the face of dynamically changing text content, especially those scenarios where new topics and new concepts frequently appear, the adaptability of traditional methods is insufficient. The limitation of the predefined category system is that it cannot capture and respond to emerging trends or nuances in text content in a timely manner, which leads to a decline in classification results. In addition, traditional methods also face many challenges when dealing with short texts. This is because short texts usually have a small vocabulary and lack of contextual information, which makes it particularly difficult to extract effective features from limited texts.

[0004] In recent years, deep learning technology has been widely used in the field of short text classification. Deep learning models, such as convolutional neural networks (CNN), recurrent neural networks (RNN) and their variants, long short-term memory networks (LSTM), gated recurrent units (GRU), and Transformer models, can automatically learn high-level feature representations of texts, effectively solving the problem of sparse features in short texts. These models can not only capture the local features of words, but also understand the contextual information and semantic relationships of texts through hierarchical structures.

[0005] However, in practical applications, we often need to process a large number of short texts of unknown categories, which may come from different fields, different language styles, and even contain noise and erroneous information. This requires the classification system to not only have powerful text processing capabilities, but also be able to automatically discover the topic categories in the text and effectively classify texts of unknown categories. Traditional supervised learning methods rely on a large amount of labeled data to train the model, but it is often difficult to obtain sufficient labeled data in practical applications.

[0006] Therefore, how to combine unsupervised learning methods or weakly supervised learning methods to automatically discover topic categories in texts and classify texts of unknown categories has become a hot topic and difficulty in current research. It is necessary to continuously explore new technologies and methods to improve the accuracy and efficiency of automatic classification of short texts of unknown categories. Summary of the invention

[0007] In view of the above, the purpose of the present invention is to provide a short text clustering method and device based on a large language model. By combining the clustering algorithm and the semantic analysis ability of the large language model, it is possible to realize automatic discovery and classification of the topics of short texts, which not only improves the accuracy of classification, but also realizes the automatic construction and dynamic optimization of a multi-layer classification system, providing strong technical support for various text processing applications.

[0008] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows: In a first aspect, an embodiment of the present invention provides a short text clustering method based on a large language model, comprising the following steps: After preprocessing the acquired short texts, vector representations of the texts are obtained, and the vector representations of the texts are preliminarily clustered by a clustering algorithm to obtain initial categories; For all the short texts corresponding to each initial category, a large language model is used to generate summaries and multiple topic titles that summarize the core meaning of the text are generated based on the summaries as subtitles. Multiple subtitles are input into the large language model for topic merging to obtain a parent title, and the parent title and subtitles are constructed into a hierarchical topic system. The short text to be classified is input into the large language model, and the large language model assigns the short text to be classified into one or more categories according to the hierarchical subject system to complete the short text clustering; Perform quality assessment and error pattern analysis on short text clustering results to optimize clustering algorithms and large language models.

[0009] Preferably, the step of preprocessing the acquired short texts to obtain vector representations of the texts includes: The obtained short texts are first cleaned, and then each cleaned short text is encoded and converted into a fixed-dimensional vector representation using a pre-trained language model. For long texts, they are converted into short texts through sliding windows or truncation, and then cleaned and encoded into vector representations.

[0010] Preferably, the step of performing preliminary clustering of the vector representation of the text by a clustering algorithm to obtain initial categories includes: The DBSCAN clustering algorithm is used to perform preliminary clustering based on the vector representation of the text to obtain multiple initial categories. The vector representation classified as noise points by the DBSCAN clustering algorithm is processed separately to be assigned to the nearest initial category or retained as a separate small category.

[0011] Preferably, when constructing a hierarchical subject system, a tree structure is used to display the hierarchical relationship between parent titles and subtitles, and an upper limit on the number of subtitles contained under each parent title is set.

[0012] Preferably, when constructing a hierarchical subject system, it also includes merging semantically similar parent titles or subtitles into higher-level categories, and generating a new parent title for each higher-level category.

[0013] Preferably, when short texts to be classified are clustered, if the large language model cannot determine the classification, it is marked as a pending label and the reason is explained. The clustering algorithm is iteratively executed for the short texts with the pending label to perform preliminary clustering, and the large language model is used to generate topic titles and construct a hierarchical topic system until the final short text clustering is completed or the specified number of iterations is reached.

[0014] Preferably, after short texts to be classified are clustered, for small categories in which the number of short texts contained therein is less than a set lower limit threshold, the small categories are merged into large categories with the closest semantics.

[0015] Preferably, the quality of the short text clustering results is evaluated using internal evaluation indicators including silhouette coefficient and Calinski-Harabasz index, external evaluation indicators including NMI and ARI, and a manual evaluation method including random sampling to check the semantic consistency of categories. Based on the quality evaluation results, common error patterns are summarized by detailed analysis of the misclassified short texts to complete the error pattern analysis.

[0016] Preferably, a method including grid search or Bayesian optimization is used to automatically adjust and optimize the parameters of the clustering algorithm, and further optimize the prompt words and output format of the large language model to dynamically optimize the semantic analysis capability of the large language model.

[0017] In a second aspect, an embodiment of the present invention further provides a short text clustering device based on a large language model, which is implemented using the short text clustering method based on a large language model, and includes: a preliminary clustering module, a topic building module, a short text clustering module and a dynamic optimization module; The preliminary clustering module is used to pre-process the acquired short texts to obtain vector representations of the texts, and to perform preliminary clustering on the vector representations of the texts through a clustering algorithm to obtain initial categories; The topic construction module is used to generate a summary of all the short texts corresponding to each initial category using a large language model, and generate multiple topic titles for summarizing the core meaning of the text according to the summary as subtitles, input the multiple subtitles into the large language model to merge the topics to obtain a parent title, and construct the parent title and the subtitles into a hierarchical topic system; The short text clustering module is used to input the short text to be classified into the large language model, and the large language model assigns the short text to be classified to one or more categories according to the hierarchical subject system to complete the short text clustering; The dynamic optimization module is used to perform quality assessment and error pattern analysis on the results of short text clustering to optimize the clustering algorithm and the large language model.

[0018] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention uses a clustering algorithm to perform preliminary clustering on massive texts to achieve rapid dimensionality reduction, overcomes the contextual limitations of large language models when processing large-scale texts, automatically discovers the potential topic structure in the text, and thus better uses it for the subsequent large language model's automatic topic discovery. The large language model is used to perform semantic refinement on the clustering results to generate more accurate topic categories, realizing the complementary advantages of the hybrid architecture of "clustering dimensionality reduction + large model classification" and ensuring the accuracy of classification.

[0019] (2) The present invention uses a large language model to perform in-depth semantic analysis on the preliminary clustering results, generates accurate topic categories, and realizes the transformation from "unsupervised clustering" to "category definition". It further performs secondary clustering on the generated topics and constructs a hierarchical topic system, realizing a complete understanding system from low-level features to high-level semantics, making the classification more structured and explainable.

[0020] (3) The present invention continuously optimizes the classification system through a dynamic optimization mechanism and the semantic understanding ability of a large language model. It can automatically identify and process texts with blurred classification boundaries, continuously improve classification accuracy through iterative optimization, and dynamically adjust the hierarchical topic system to ensure the semantic clarity of the categories. Through such an adaptive topic discovery mechanism, it can adapt to the ever-changing text content and ensure the accuracy and timeliness of classification.

[0021] (4) The method and apparatus provided by the present invention adopt a modular design, support distributed processing, and can realize incremental updates to adapt to the continuously growing data scale. They also support parallel computing to improve processing efficiency. These features enable the present invention to easily cope with large-scale practical application scenarios.

[0022] (5) The present invention combines clustering algorithms and large language models to achieve deep semantic understanding capabilities. It can apply the cognitive capabilities of large language models to text classification, thereby better understanding the deep semantic connotations of texts, identifying cross-domain conceptual associations, and effectively processing fuzzy and ambiguous expressions. This deep understanding capability makes the classification results more consistent with human cognition.

[0023] (6) The method and device proposed in the present invention have strong generalization capabilities. Since they do not require domain-specific training data, they can automatically adapt to the professional terminology of different industries and support multilingual text processing, so they can be quickly deployed in various application fields. They provide a new technical route for the field of natural language processing and have important theoretical value and broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 It is a flowchart of a short text clustering method based on a large language model provided by an embodiment of the present invention; Figure 2 It is a schematic diagram of a process of preprocessing and preliminary clustering provided by an embodiment of the present invention; Figure 3 It is a schematic diagram of a process for constructing a hierarchical subject system provided by an embodiment of the present invention; Figure 4 It is a structural schematic diagram of a short text clustering device based on a large language model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0027] The inventive concept of the present invention is: in response to the problem of insufficient accuracy and efficiency in classifying short texts of unknown categories in the prior art, the embodiments of the present invention provide a short text clustering method and device based on a large language model. On the basis of using a clustering algorithm for preliminary clustering, a hierarchical topic system is further constructed in combination with the large language model, and the hierarchical topic system is used to complete the automatic classification of all short texts, and the clustering algorithm and the large language model are dynamically adjusted. The efficiency and accuracy of classification are effectively improved through the automatic construction and dynamic optimization of the multi-layer classification system.

[0028] Figure 1 FIG. 1 is a flow chart of a short text clustering method based on a large language model provided by an embodiment of the present invention. Figure 1 As shown, the embodiment provides a short text clustering method based on a large language model, comprising the following steps: S1, pre-process the obtained short texts to obtain vector representations of the texts, and perform preliminary clustering of the vector representations of the texts through a clustering algorithm to obtain initial categories. Figure 2 shown.

[0029] S1.1, Data preprocessing.

[0030] S1.1.1, text cleaning. The obtained short texts are first cleaned to remove special characters, extra spaces and other noise, unify the case (such as converting to lowercase), and remove stop words (optional, depending on the specific application).

[0031] S1.1.2, text encoding. Use a pre-trained language model (including BERT model, RoBERTa model or SBERT model) to encode each cleaned short text into a fixed-dimensional vector representation. For long texts, convert them into short texts through sliding windows or truncation, then clean the text and encode them into vector representations.

[0032] S1.2, Preliminary clustering.

[0033] S1.2.1, DBSCAN clustering. Use the DBSCAN clustering algorithm to perform preliminary clustering based on the vector representation of the text to obtain multiple initial categories. The DBSCAN clustering algorithm parameters are selected as follows: Distance threshold eps: Adaptively selected according to data distribution, using K-distance graph to assist in determination; Sample number threshold min_samples: It is set according to the expected minimum class size, and is set to 4-5 in the embodiment.

[0034] S1.2.2, Anomaly Detection. Identify the vector representations classified as noise points by the DBSCAN clustering algorithm, and process the short texts corresponding to these vector representations separately, assigning them to the nearest initial category or retaining them as separate small categories.

[0035] S2: For all the short texts corresponding to each initial category, a large language model is used to generate a summary and multiple topic titles are generated based on the summary to summarize the core meaning of the text as subtitles. Multiple subtitles are input into the large language model for topic merging to obtain a parent title, and the parent title and subtitles are constructed into a hierarchical topic system.

[0036] S2.1, generate summaries based on short texts. For all the short texts corresponding to each initial category, use a large language model (including the GPT-3 model, ChatGPT model, or a custom fine-tuned large language model) to generate a concise summary.

[0037] S2.2, generate subtitles based on the abstract. Generate multiple topic titles that summarize the core meaning of the text based on the abstract and the prompt words as subtitles.

[0038] Prompt word design: “Please summarize the common themes of the following texts and provide 3-5 brief topic headings that capture the core meaning of these texts”.

[0039] S2.3, generate parent titles based on subtitles and build a hierarchical theme system. For the multiple subtitles obtained in S2.2, input the large language model, use prompt words to merge semantically similar titles, generate a parent title for the merged titles, and build the parent title and subtitles into a hierarchical theme system.

[0040] Prompt word design: "You are a professional text classification expert. Here is a set of topic titles: [list of subject titles]; Please complete the following tasks: 1. Analyze the semantic associations between these titles; 2. Merge semantically similar titles into higher-level categories; 3. Generate a parent title for each high-level category; 4. Build a clear hierarchical relationship; Output requirements: Use a tree structure to display hierarchical relationships; Each parent heading can contain up to 5 sub-headings.

[0041] S3, input the short text to be classified into the large language model, and the large language model assigns the short text to be classified to one or more categories according to the hierarchical subject system to complete the short text clustering.

[0042] S3.1, Category Arrangement. Based on the hierarchical topic system obtained in S2.3, a large language model is used to classify all short texts to be classified. A short text can belong to one or more categories. For short texts that cannot be classified, they are marked with the "pending" label and the reason is explained.

[0043] Prompt word design: "You are a professional text classification expert. Based on the following classification system: [Hierarchical theme system]; Please categorize this text: [Text to be classified]; Require: 1. Assign the text to one or more of the most appropriate categories; 2. If the classification cannot be determined, mark it as 'pending' and state the reason."

[0044] S3.2 Iterative optimization. For short texts marked with the "pending" label, iteratively execute S2.1~S3.1, re-perform preliminary clustering using the clustering algorithm, generate topic titles using the large language model, and adjust the hierarchical topic system until the final short text clustering is completed or the specified number of iterations (such as 5 times) is reached.

[0045] S3.3, small category processing. For small categories whose number of short texts is less than the set lower limit threshold (such as less than 3 short texts), the small categories are merged into the large category with the closest semantics.

[0046] S4, quality assessment and error pattern analysis are performed on the results of short text clustering to optimize the clustering algorithm and large language model.

[0047] S4.1 Quality assessment. The quality of the short text clustering results is assessed using internal evaluation indicators including the silhouette coefficient and the Calinski-Harabasz index, external evaluation indicators including NMI (Normalized Mutual Information) and ARI (Adjusted Rand Index), and a manual evaluation method including random sampling to check the semantic consistency of categories.

[0048] S4.2, Error analysis. According to the quality assessment results, the incorrectly classified short texts are analyzed in detail, and common error patterns are summarized to complete the error pattern analysis. According to the error analysis results, the algorithm can be improved or the model parameters can be adjusted in a targeted manner.

[0049] S4.3, hyperparameter tuning. Use methods including grid search or Bayesian optimization to automatically adjust the parameters of the clustering algorithm, and further optimize the prompt words and output format of the large language model to dynamically optimize the semantic analysis ability of the large language model and improve the accuracy of classification.

[0050] In summary, the short text clustering method based on a large language model provided by the embodiment of the present invention can effectively combine traditional clustering algorithms and advanced large language models to achieve high-quality short text semantic clustering. The method has strong adaptability and scalability, can process text data of various fields and scales, provides a reliable foundation for natural language processing applications, and has broad and far-reaching application value.

[0051] The short text clustering method based on a large language model provided by the embodiment of the present invention can play an important role in multiple fields and scenarios and significantly improve the efficiency and accuracy of text processing: 1. Standardization of terminology.

[0052] In various professional fields, the standardization of terminology is an important and complex task. The present invention provides an innovative solution through automatic classification, which can be further applied to: (1) Automatic term classification: by identifying semantically similar term variants and classifying them into a unified standard category, even if these terms differ in expression form; (2) Intelligent standardization: Use a large language model to perform semantic analysis on each category and automatically generate the most representative standard terms.

[0053] (3) Interdisciplinary mapping: By discovering semantically similar concepts in different disciplines, an interdisciplinary terminology comparison table is established.

[0054] (4) Multilingual correspondence: Automatically establish term mapping relationships between different languages ​​and support the integration of multilingual knowledge systems.

[0055] (5) Terminology evolution analysis: By regularly updating the classification system, tracking the evolution trend of terminology usage, and updating the standard terminology database in a timely manner.

[0056] 2. Theme discovery and analysis.

[0057] Automatic topic discovery is a key task in large-scale text data processing. The present invention provides a complete solution, which can be further applied to: (1) Intelligent topic identification: Combining the advantages of clustering and large language models, it can accurately identify core topics from massive texts.

[0058] (2) Hierarchical theme system: Through multi-level classification, a clear theme hierarchy structure is constructed to show the relationship between themes.

[0059] (3) Emerging topic warning: Continuously monitor and analyze text flows to promptly identify and classify emerging discussion topics.

[0060] (4) Cross-domain topic association: Automatically discover and analyze the topic connections between different fields or data sources.

[0061] (5) Tracking the evolution of themes: By analyzing the classification results of different periods, the development of themes can be depicted.

[0062] 3. Intelligent text classification.

[0063] In application scenarios that require automatic text classification, this method provides a more flexible and intelligent solution that can achieve adaptive classification without pre-defining a fixed category system. It can dynamically generate appropriate classification standards based on the text content and achieve fine-grained classification. Through a hierarchical classification system, it can achieve refined classification of text content and achieve real-time updates. It can dynamically adjust the classification system according to the newly added text content to maintain the timeliness of classification. It can be further applied to: (1) News information classification: Automatically build a news topic system to achieve accurate news classification and push.

[0064] (2) User feedback analysis: Intelligently identify the types of problems in user feedback and support accurate problem analysis and processing.

[0065] (3) Knowledge management: Establish a dynamic classification system for corporate documents to optimize knowledge retrieval and management efficiency.

[0066] Based on the same inventive concept, Figure 4 As shown, the embodiment of the present invention further provides a short text clustering device 400 based on a large language model, including: a preliminary clustering module 410, a topic construction module 420, a short text clustering module 430 and a dynamic optimization module 440.

[0067] The preliminary clustering module 410 is used to pre-process the acquired short texts to obtain vector representations of the texts, and to perform preliminary clustering on the vector representations of the texts through a clustering algorithm to obtain initial categories.

[0068] The topic construction module 420 is used to generate summaries for all short texts corresponding to each initial category using a large language model and generate multiple topic titles that summarize the core meaning of the text as subtitles based on the summaries, input the multiple subtitles into the large language model for topic merging to obtain a parent title, and construct the parent title and subtitles into a hierarchical topic system.

[0069] The short text clustering module 430 is used to input the short text to be classified into the large language model, and the large language model assigns the short text to be classified to one or more categories according to the hierarchical subject system to complete the short text clustering.

[0070] The dynamic optimization module 440 is used to perform quality assessment and error pattern analysis on the results of short text clustering to optimize the clustering algorithm and the large language model.

[0071] It should be noted that the short text clustering device based on a large language model provided in the above embodiment belongs to the same inventive concept as the short text clustering method based on a large language model. The specific implementation process is detailed in the embodiment of the short text clustering method based on a large language model, which will not be repeated here.

[0072] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A short text clustering method based on a large language model, characterized in that: The following steps are involved: After preprocessing the acquired short texts, vector representations of the texts are obtained, and the vector representations of the texts are preliminarily clustered by a clustering algorithm to obtain initial categories; For all the short texts corresponding to each initial category, a large language model is used to generate summaries and multiple topic titles that summarize the core meaning of the text are generated based on the summaries as subtitles. Multiple subtitles are input into the large language model for topic merging to obtain a parent title, and the parent title and subtitles are constructed into a hierarchical topic system. The short text to be classified is input into the large language model, and the large language model assigns the short text to be classified into one or more categories according to the hierarchical subject system to complete the short text clustering; Perform quality assessment and error pattern analysis on short text clustering results to optimize clustering algorithms and large language models.

2. The short text clustering method based on a large language model according to claim 1, characterized in that: The method of preprocessing the obtained short texts to obtain vector representations of the texts includes: The obtained short texts are first cleaned, and then each cleaned short text is encoded and converted into a fixed-dimensional vector representation using a pre-trained language model. For long texts, they are converted into short texts through sliding windows or truncation, and then cleaned and encoded into vector representations.

3. The short text clustering method based on a large language model according to claim 1, characterized in that: The method of performing preliminary clustering of the vector representation of the text by a clustering algorithm to obtain initial categories includes: The DBSCAN clustering algorithm is used to perform preliminary clustering based on the vector representation of the text to obtain multiple initial categories. The vector representation classified as noise points by the DBSCAN clustering algorithm is processed separately to be assigned to the nearest initial category or retained as a separate small category.

4. The short text clustering method based on a large language model according to claim 1, characterized in that: When building a hierarchical topic system, use a tree structure to display the hierarchical relationship between parent titles and subtitles, and set an upper limit on the number of subtitles that can be included under each parent title.

5. The short text clustering method based on a large language model according to claim 1 or 4, characterized in that: When constructing a hierarchical subject system, it also includes merging semantically similar parent titles or subtitles into higher-level categories and generating a new parent title for each higher-level category.

6. The short text clustering method based on a large language model according to claim 1, characterized in that: When clustering short texts to be classified, if the large language model cannot determine the classification, it will be marked as a pending label and the reason will be explained. The clustering algorithm will be iteratively executed for the short texts with pending labels to perform preliminary clustering, and the large language model will be used to generate topic titles and build a hierarchical topic system until the final short text clustering is completed or the specified number of iterations is reached.

7. The short text clustering method based on a large language model according to claim 1, characterized in that: After the short texts to be classified are clustered, for the small categories in which the number of short texts contained therein is less than the set lower limit threshold, the small categories are merged into the large category with the closest semantics.

8. The short text clustering method based on a large language model according to claim 1, characterized in that: The quality of short text clustering results is evaluated using internal evaluation indicators including silhouette coefficient and Calinski-Harabasz index, external evaluation indicators including NMI and ARI, and manual evaluation methods including random sampling to check category semantic consistency. Based on the quality assessment results, common error patterns are summarized through detailed analysis of the misclassified short texts to complete the error pattern analysis.

9. The short text clustering method based on a large language model according to claim 1, characterized in that: Methods including grid search or Bayesian optimization are used to automatically adjust and optimize the parameters of the clustering algorithm, and the prompt words and output format of the large language model are further optimized to dynamically optimize the semantic analysis capability of the large language model.

10. A short text clustering device based on a large language model, implemented by the short text clustering method based on a large language model according to any one of claims 1 to 9, characterized in that: include: Preliminary clustering module, topic building module, short text clustering module and dynamic optimization module; The preliminary clustering module is used to pre-process the acquired short texts to obtain vector representations of the texts, and to perform preliminary clustering on the vector representations of the texts through a clustering algorithm to obtain initial categories; The topic construction module is used to generate a summary of all the short texts corresponding to each initial category using a large language model, and generate multiple topic titles for summarizing the core meaning of the text according to the summary as subtitles, input the multiple subtitles into the large language model to merge the topics to obtain a parent title, and construct the parent title and the subtitles into a hierarchical topic system; The short text clustering module is used to input the short text to be classified into the large language model, and the large language model assigns the short text to be classified to one or more categories according to the hierarchical subject system to complete the short text clustering; The dynamic optimization module is used to perform quality assessment and error pattern analysis on the results of short text clustering to optimize the clustering algorithm and the large language model.

Images