A topic clustering method and storage medium driven by a large language model

Abstract

The present invention relates to the field of artificial intelligence natural language processing, and in particular to a topic clustering method driven by a large language model; the method comprises the following steps: S1) arranging document set data; S2) using the HDBSCAN clustering algorithm to perform preliminary clustering on the document set D to obtain a plurality of clusters {C k}; S3) Model the document topics using the LDA topic model to obtain the topic distribution of each document; S4) Adjust the clustering results based on the results obtained from the LDA model; S5) Further refine the topics of each cluster using LLM; S6) Establish effective and explainable associations between the generated topic list and the documents in the data set. The topic clustering system driven by the large language model of the present invention achieves a deep understanding and fine clustering of document topics by integrating HDBSCAN clustering, LDA and the large language model, providing a solid foundation for subsequent information retrieval, text mining and knowledge graph construction.

Description

Technical Field

[0001] The invention of the present application relates to the field of artificial intelligence natural language processing, and in particular to a topic clustering method driven by a large language model. Background Art

[0002] In recent years, the development of natural language processing (NLP) technology, especially deep learning and language models, has brought revolutionary breakthroughs to text data analysis. Among them, topic models such as Latent Dirichlet Allocation (LDA) and clustering algorithms such as K-means and HDBSCAN clustering have become important tools for mining text topic structures. However, these traditional methods have certain limitations. For example, LDA assumes that the document topic distribution follows the Dirichlet prior, which may not match the actual data distribution; and hard clustering algorithms such as K-means may lead to local optimal solutions due to initialization problems.

[0003] Traditional text analysis methods, such as keyword-based search and simple statistical analysis, can no longer meet the needs for in-depth understanding of large-scale, multi-domain text data.

[0004] How to effectively integrate LLM with classic topic models and clustering algorithms, how to quantify and optimize the interaction between models, and how to design a reasonable topic evaluation and update mechanism are all urgent issues to be solved. Therefore, an innovative method is needed that can fully utilize the advantages of LLM while combining the stability of traditional algorithms to achieve smarter and more dynamic topic clustering analysis. Summary of the invention

[0005] In view of the above problems, the present invention proposes a topic clustering method driven by a large language model, and the specific process of the method is as follows:

[0006] S1, first construct a document set D, with d representing each document in the document set, then the document set can be expressed as D = {d1, d2, ..., d M Each document d can be represented as a sequence of N words, that is, d = [w1, w2, ..., w N ], and words are the basic units of discrete document data, which can be represented as items in the vocabulary indexed by {1,2,...,V}, where V is the total number of words. Then each word can be represented by a one-hot unit vector of dimension V.

[0007] S2, document collection data organization: This stage involves data preprocessing, including but not limited to text cleaning, stop word removal, stemming, word segmentation, and possible spelling correction, to ensure the quality and consistency of document data.

[0008] S3, use the HDBSCAN clustering algorithm to perform preliminary clustering on the document set D, and obtain multiple clusters. This step uses the TF-IDF algorithm to convert the document into a vector representation. The HDBSCAN clustering algorithm does not require pre-specified clusters, and the algorithm can automatically determine the optimal number of clusters. By using data structures such as KD-Tree or BallTree to accelerate the calculation of building a neighborhood graph, a cohesive hierarchical clustering tree is built based on the neighborhood graph, which involves gradually merging the nearest points or clusters until all points are merged into one cluster. Then calculate the persistence of each cluster, that is, the number of levels at which the cluster persists. Clusters with high persistence are usually more stable clusters. The present invention selects the cluster with the highest persistence as the final clustering result, and regards points that are not included in any cluster as noise points.

[0009] S4, then use the LDA (Latent Dirichlet Allocation) topic model to perform topic modeling on the preliminary clustered documents to obtain the topic distribution of each document. The LDA model is a generative model based on Bayesian statistics, which can identify the potential topic structure in the document and give the weight of each topic in the document. This method solves the LDA model based on the Gibbs sampling algorithm. The model is as follows Figure 2 As shown, α and β are known prior inputs, and the goal is to obtain the topic distribution θ and topic word distribution of the document , based on which the topic of the article is predicted. The following is the process of topic modeling:

[0010] S4.1, define the number of topics K, and the hyperparameters α and β.

[0011] S4.2, randomly assign a topic to each word in each document.

[0012] S4.3, for each word in the document collection, use the Gibbs sampling formula to update the topic to which the selected word belongs. The formula is as follows:

[0013]

[0014] z i is the topic to which the i-th word in the corpus belongs, k is the topic index, For word w i The document d is assigned the topic z i = k times, For word w i The number of words in the document. For word w i =t is given to the subject z i = k times, All words are assigned topic z i= k times, all times calculations exclude the current item

[0015] S4.4, repeat the previous step until Gibbs sampling converges.

[0016] S4.5, count the topics of each word in each document in the corpus, and get the topic distribution θ and the topic word distribution

[0017] S5, dynamically adjust the clustering results according to the topic distribution obtained by the LDA topic model. This step allows documents to migrate between different clusters by calculating the topic similarity between the document and other documents in the cluster to optimize the clustering effect. The specific method is as follows:

[0018] S5.1, calculate the probability distribution p of the LDA topic model between all documents LDA (d) KL divergence Construct a two-dimensional matrix

[0019] S5.2, calculate each cluster C k The average KL divergence value between documents

[0020] S5.3, traverse each document d and calculate the average KL divergence value of its probability distribution with other documents in the cluster about the LDA topic model And attribute document d to the cluster with the lowest average KL divergence value:

[0021]

[0022] where z d is the topic index of document d.

[0023] S5.4, repeat steps S5.2 and S5.3 until the results converge to obtain the topic cluster results.

[0024] S6, using the large language model (LLM) to further refine and refine the topics of each cluster. LLM is used to identify and merge repeated topics, remove uncommon topics, and generate easy-to-understand topic descriptions. In addition, LLM can also identify and merge semantically similar topics to ensure the coherence and non-redundancy of the topic list. The specific method is to use Sentence-Transformer to load the text embedding model, calculate the text representation of the topic words, and calculate the cosine similarity between different topic representations:

[0025]

[0026] in, Cluster C iThe topic words are represented by the vectors obtained based on the text embedding model. Then, the cosine similarity results between the topic set and the topic pairs are used as LLM inputs, and LLM performs combined prediction.

[0027] S7,Finally, the topic assignment process ensures a valid and interpretable association between documents and topics.,By providing the generated topic list, example topics, and individual documents,,LLM is able to assign the most appropriate topic label and description to each,document, so that the final output includes both the assigned topic,labels and specific topic descriptions for each document.

[0028] The beneficial effects of this application are:

[0029] 1. Co-optimization of topic model and clustering algorithm: By combining LDA topic modeling with HDBSCAN clustering algorithm, the present invention aims to solve the limitations of single algorithm in topic discovery, such as LDA's assumptions about topic distribution. Through iterative adjustment and optimization, more accurate document classification and topic boundary determination can be achieved.

[0030] 2. Deepen topic understanding using large language models (LLM): By introducing LLM as a high-level interpreter and topic optimizer, the present invention solves the shortcomings of traditional topic models in generating human-readable and semantically rich topic descriptions. LLM can generate more accurate and detailed topic labels and their definitions, thereby improving the interpretability of topic clustering results.

[0031] 3. Adjustment mechanism of topic clustering: Considering the continuous changes in document topic sets and the emergence of new topics, the present invention provides a dynamic adjustment strategy for topic clustering. By monitoring the generation frequency of new topics, the system can update the topic list in a timely manner, remove inactive or duplicate topics, and ensure the timeliness and relevance of the topic list.

[0032] 4. Similarity and redundancy elimination between topics: By calculating the similarity of topic embeddings through deep learning models such as Sentence-Transformer, the present invention can identify and merge topics that are too similar, avoid redundancy in the topic list, and ensure the diversity and independence of the output topics.

[0033] 5. Refined management of document-topic association: The present invention realizes refined matching between documents and topics through LLM, which not only assigns the most relevant topic label to each document, but also provides a specific topic description for the document, thereby enhancing the user's understanding of the document content and retrieval efficiency.

[0034] 6. Balance between system performance and efficiency: Through carefully designed algorithm flow and iterative update strategy, the present invention not only ensures the quality of topic clustering, but also takes into account the computational efficiency and resource consumption, and strives to find the best balance between accuracy and real-time performance.

[0035] In summary, the topic clustering system driven by the large language model of the present invention, by integrating HDBSCAN clustering, LDA and a large language model, achieves a deep understanding and fine clustering of document topics, providing a solid foundation for subsequent applications such as information retrieval, text mining and the construction of knowledge graphs. The above steps comprehensively utilize a variety of machine learning techniques and natural language processing algorithms to improve the accuracy and interpretability of topic clustering in order to cope with the complexity and dynamism of large-scale document collections. Through this series of steps, the present invention can effectively reveal the topic structure in the document collection, providing strong support for information organization and retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0037] Figure 1 It is a flowchart of a method for constructing a topic clustering system driven by a large language model provided in an embodiment of this specification;

[0038] Figure 2 It is a probabilistic graphical model of the text generation process of the provided LDA topic model. DETAILED DESCRIPTION

[0039] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention, so the present invention is not limited by the specific implementation disclosed below.

[0040] The technical solutions provided by the embodiments of this specification are described in detail below in conjunction with the accompanying drawings.

[0041] Figure 1 A flowchart of a method for constructing a topic clustering system based on a large language model provided in an embodiment of this specification includes the following steps:

[0042] S1, obtain and organize document datasets:

[0043] This process first organizes the document data set. This embodiment usually involves processing a large amount of text data, which requires high computing power of the device.

[0044] In the embodiment of the present invention, a document set D is first constructed, and d represents each document in the document set. Then the document set can be expressed as D = {d1, d2, ..., d M}, where M represents the total number of documents in the document collection. Each document d can be represented as a sequence of N words, that is, d = [w1, w2, ..., w N ], words are the basic units of discrete document data and can be represented as items in the vocabulary indexed by {1,2,...,V}, where V is the total number of non-repeated words in the document collection. Then each word can be represented by a one-hot unit vector of dimension V.

[0045] In the embodiments of this specification, the document data set needs to be preprocessed, including data operation methods such as special character processing, stop word processing, and text segmentation, so as to facilitate the subsequent clustering algorithm and topic model to extract and cluster the topic of the document, wherein the stop word processing and text segmentation can be processed according to the stop word dictionary and the word segmentation dictionary respectively.

[0046] S2, HDBSCAN preliminary clustering:

[0047] In an embodiment of the present specification, the HDBSCAN clustering algorithm is used to cluster the document collection preliminarily. The present invention can use the pandas library of python to read and process the data set, followed by the preprocessing of the document set, including the removal of stop words, tokenization and stem extraction, etc. Then the text data needs to be converted into a numerical vector. In order to calculate the similarity. The bag of words model is used here, in which each word is regarded as a feature, and the number of occurrences of each word in each text is calculated. Then the TF-IDF algorithm is used to weight each word to better distinguish important words from common words. Next, the HDBSCAN algorithm is used to cluster the text data. The main steps of the HDBSCAN algorithm are to calculate the nearest neighbor point set of each point, and construct a neighborhood graph based on the distance between these points, and then build a cohesive hierarchical clustering tree based on the neighborhood graph, and then calculate the persistence of each cluster, and select the cluster with the highest persistence as the final clustering result. Finally, the document vector of each cluster is averaged and reconverted into text to represent the central theme of the cluster.

[0048] S3, LDA topic modeling:

[0049] In an embodiment of the present invention, Figure 2 The probabilistic graphical model of the LDA topic model is presented, showing the modeling process of text generation, where α and β are hyperparameters that control the distribution of topics in documents and the distribution of words within topics, respectively, and θ and is the target parameter to be solved.

[0050] The LDA topic model can give the topic of each document in a document collection in the form of probability. After analyzing some documents and extracting their topic distribution, topic clustering or text classification can be performed based on the topic distribution. At the same time, it is a typical bag-of-words model, that is, a document is composed of a group of words, and there is no order relationship between words. In addition, a document can contain multiple topics, and each word in the document is generated by one of the topics.

[0051] In an embodiment of the present invention, after the text data set has been preprocessed in the previous step, a bag-of-words model needs to be constructed: first, a vocabulary is created, which includes all the words in this document data set; then a document-word frequency matrix is ​​constructed, and for each document, the number of times each word appears is recorded to form a bag-of-words representation.

[0052] In an embodiment of the present invention, it is necessary to set the parameters of the LDA topic model. The first is the number of topics: this is usually based on domain knowledge or can be determined by cross-validation. Hyperparameters α and β: α controls the sparsity of topic distribution in the document, and β controls the sparsity of vocabulary distribution within the topic.

[0053] In an embodiment of the present invention, training the LDA model generally includes the following steps:

[0054] (1) Model training: Use the training data set to train the LDA model, usually using the Gibbs sampling algorithm;

[0055] (2) Iterative optimization: The model gradually optimizes the topic-vocabulary distribution and document-topic distribution through multiple iterations.

[0056] In an embodiment of the present invention, topics are extracted from a trained LDA topic model, and each topic is represented by a set of high-probability words.

[0057] S4, clustering result adjustment:

[0058] In the embodiment of the present invention, after completing the preliminary HDBSCAN clustering and LDA topic modeling, it is a very important key step to adjust the modeling results to improve the clustering quality and relevance. The following is a detailed process of this step:

[0059] S4.1, topic distance calculation:

[0060] First, KL divergence is used as a distance metric to measure the difference between topic distributions. Next, the document topic distance is calculated. Based on the initial clustering results of HDBSCAN, the average distance metric of other documents in each cluster to the cluster center document is calculated, and the average distance metric of each document in other different clusters is calculated. According to the minimum value of the average distance metric between the document and documents in different clusters, the document is moved to the cluster. The specific method is as follows:

[0061] S4.1, calculate the probability distribution p of the LDA topic model between all documents LDA (d) KL divergence Construct a two-dimensional matrix .

[0062] S4.2, calculate each cluster C k The average KL divergence value between documents

[0063] S4.3, traverse each document d and calculate the average KL divergence value of the probability distribution of the LDA topic model with other documents in the cluster And attribute document d to the cluster with the lowest average KL divergence value:

[0064]

[0065] S4.4, repeat steps S5.2 and S5.3 until the clustering result is stable or the predetermined maximum number of iterations is reached. In each iteration, the change in clustering quality should be evaluated. The present invention can evaluate by calculating the sum of square errors within the cluster or using external indicators (such as Rand Index, Adjusted Rand Index). When the composition of the cluster no longer changes significantly or the preset number of iterations is reached, the iteration process is stopped. At this point, the clustering result has converged.

[0066] S5, use LLM to further refine the theme of each cluster:

[0067] In the embodiment of the present invention, after completing the clustering result optimization and obtaining the central theme of each cluster and the document theme set, the next step is to use the large language model to refine and refine these themes to produce a clearer and more consistent theme description. The following is the specific operation process:

[0068] S5.1, refine and generate new topics:

[0069] First, each cluster C k The document set and the topic set generated by the LDA model {z k} and two or three sample theme collections Prepare (Initially, It consists of some manually given topic-explanatory labels, which are used as a partial demonstration of the topic generation format, prompting the LLM in what format to generate topics and their explanatory descriptions. At the same time, the manually given example topics do not need to be corpus-specific, nor do they need to represent possible topics in the corpus, but are simply provided as an output format to the large language model), which is provided as input to the LLM. The LLM is prompted to generate a set of topics given an input dataset, and then further refine the topics, removing uncommon topics and merging duplicates. The output of this step can also be optionally fed into the model again to generate more fine-grained sub-topics. The model is instructed to either use D k The document is assigned to the current example subject collection topics in the language, either requiring a large language model combined with a topic collection Generate better descriptions C k and add the generated topic to the sample topic collection In the large language model, The topic in the text provides concise labels and corresponding definitions, such as "Scientific research: refers to the process of exploring natural and social phenomena through systematic observation, experimentation, and theoretical analysis". Here, the "Scientific research" label is the document topic keyword extracted by the LDA model, and the content after the colon is an explanatory description given by the large language model combined with the document content.

[0070] In the embodiment of the present invention, each cluster C is classified by using a large language model. k For analysis, LLM based on C k The content and existing subject sets k} to refine and generate new topic descriptions, LLM analyzes C k The semantic content of the documents in the text is extracted to key concepts and entities, thereby generating and improving topic descriptions; and based on the context and semantic understanding, a brief and precise definition is provided for each topic to enhance the readability and interpretability of the topic.

[0071] S5.2, Refine the generated topics:

[0072] In this process, the present invention uses Sentence-Transformer Embedding technology to generate embedding vectors of topics to quantify the semantic similarity between topics. The generated topics are converted into vector representations Then, calculate the cosine similarity between topic vectors Then the similarity calculation results and the topic textual information are used as LLM input, and LLM performs combined prediction.

[0073] In the embodiment of the present invention, for each topic, the frequency of its occurrence in the document cluster topic set is recorded, and a threshold is set. If the frequency of a topic is lower than the threshold, it is deleted from the final topic list S to ensure the coherence and non-redundancy of the topic list. The whole process needs to go through multiple iterations to continuously optimize the quality of the topic. After each iteration, the generated topics and definitions are manually reviewed to ensure that they are accurate and well-explained.

[0074] Ultimately, this series of steps will help produce a concise, coherent, non-redundant, and well-defined list of topics for better understanding and classification of document collections.

[0075] S6, Topic Allocation:

[0076] In an embodiment of the present invention, after the generation and refinement of the topic set is completed, the large language model is used to complete the assignment of topics. That is, an effective and explainable association is established between the generated topic list and the documents in the data set. The present invention provides the LLM with the generated topic list, 2-3 example topics and a document. The model is instructed to assign one or more topics to a given document. The final output contains the assigned topic labels, document-specific topic descriptions, and references in the document that support this assignment. Network public opinion: mentions the public's increased awareness of data privacy protection ("... the public's demand for data privacy protection is increasing..."). Among them, "network public opinion" is the designated topic label, followed by the topic description and document references in brackets. Compared with the traditional LDA topic modeling method, the quoted text improves the verifiability, explainability and verifiability of traditional tasks.

[0077] In the embodiment of the present invention, a self-correction mechanism is also introduced: In order to solve the problems of incorrect format or poor quality, the present invention uses a self-correction mechanism to modify the wrong topic. The specific method is: a small parameter of the Transformer model based on the Decoder-only architecture is used to identify hallucinations or invalid assignments (such as "None" / "error"). After identifying the wrong topic, the wrongly identified document and the error type are re-provided to the LLM, and the model is prompted to re-assign a valid topic to the document.

[0078] An embodiment of the present invention provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0079] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

Claims

1. A topic clustering method driven by a large language model, characterized in that: The steps include: S1) Document collection data arrangement: It includes constructing a document collection D = {d1, d2, ..., d M }, where each document d=[w1,w2,...,w N ], the document d consists of N words; the word is the basic unit of discrete document data, representing an item in the vocabulary indexed by {1,2,...,V}, where V is the total number of non-repeated words in the document set; each word is represented by a one-hot unit vector of dimension V; S2) using the HDBSCAN clustering algorithm to perform preliminary clustering on the document set D to obtain multiple clusters {C k }; S3) Modeling document topics using the LDA topic model to obtain the topic distribution of each document; S4) adjusting the clustering results based on the results obtained from the LDA model; S5) Use LLM to further refine the theme of each cluster; S6) Establish valid and interpretable associations between the generated topic lists and the documents in the dataset.

2. The topic clustering method based on large language model driving according to claim 1 is characterized in that: Step S2) comprises: S2.1) Preprocessing the document; S2.2) vectorize the documents using the TF-IDF algorithm and calculate the nearest neighbors of each point in the dataset using a distance metric and construct a neighborhood graph; S2.3) Based on the neighborhood graph, construct an agglomerative hierarchical clustering tree by HDBSCAN clustering algorithm, wherein each node represents a cluster of a subset; the root node of the tree represents the entire data set, and the leaf nodes represent individual data points; S2.4) The HDBSCAN clustering algorithm marks those data points that do not belong to any cluster as noise points; S2.5) The iteration ends when the clustering results are stable, otherwise the HDBSCAN clustering algorithm is repeatedly executed. After stabilization, the results at this time are recorded, and the document vector representation in each cluster is averaged to represent the cluster, and then the keywords in the center of each cluster are extracted as the first-level topic of the cluster.

3. The topic clustering method based on large language model driving according to claim 2 is characterized in that: Step S3) includes: the LDA topic model is a three-layer Bayesian model, in which the first layer is documents, the second layer is topics, and the third layer is vocabulary; each document is a mixture of multiple topics; establishing an LDA hypothesis document, which includes first extracting a topic distribution from a Dirichlet distribution, and then for each word in the document, first selecting a topic according to the topic distribution, and then selecting a word according to the vocabulary distribution under the topic; approximating the posterior probability distribution through the EM algorithm to obtain a topic word bag for each document.

4. The topic clustering method based on large language model driving according to claim 3 is characterized in that: Step S4) includes: according to the topic distribution of the document, using KL divergence to calculate the similarity between the document and the cluster center, and optimizing the clustering effect.

5. The topic clustering method based on large language model driving according to claim 4 is characterized in that: Step S5) comprises: S5.1) Refine and generate new topics: First, delete uncommon topics and merge duplicates, and cluster the document clusters C k A set of example topics S obtained by the LDA model and the cluster are input into a large model, and the large model provides an explanatory description for each topic; S5.2) Refine the generated topics: Use the Sentence-Transformer embedding algorithm to identify topic pairs with cosine similarity ≥ 0.5; then use multiple identified topic pairs to prompt the LLM and merge repeated pairs where appropriate; track the generation frequency of each topic, and if a topic frequency is below the deletion threshold, delete it from the final topic list S to ensure that the generated topic list is coherent and non-redundant.

6. The topic clustering method based on large language model driving according to claim 5 is characterized in that: Step S6) includes providing the generated topic list, multiple example topics and a document to the LLM; instructing the LDA model to assign one or more topics to a given document; and the final output includes the assigned topic labels and document-specific topic descriptions.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium comprises instructions, and when the instructions are executed on a computer, the computer is caused to perform the method according to any one of claims 1 to 6.

Images