Two-stage intelligent text clustering method based on large language model guidance

Abstract

The invention discloses a two-stage intelligent text clustering method based on large language model guidance, and relates to the technical field of artificial intelligence, natural language processing and machine learning, and the method comprises the steps: converting a to-be-processed text into a multi-source text vector set, and determining a standardized unified text representation vector set in combination with information entropy; determining a pre-clustering result through the dynamically calculated initial clustering number; calculating an inter-cluster Ward distance based on a pre-clustering result, introducing cluster size weighted correction, and constructing a clustering tree; a candidate cluster number set is determined based on the MDL principle, cluster pairs to be combined are extracted by accessing a cluster tree in a reverse order, whether the sample pairs belong to the same cluster or not is judged through a large language model, and a weighted F-beta score is calculated to determine the optimal cluster number; and intercepting a target clustering result according to the optimal clustering number and generating a clustering result. According to the method, the calculation complexity is reduced to O (n), the consistency of a clustering result and human cognition is improved, and the method is suitable for rapid and accurate clustering of large-scale text data.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, natural language processing, and machine learning, and in particular to a two-stage intelligent text clustering method based on a large language model. Background Technology

[0002] With the advent of the big data era, text data is experiencing explosive growth, from academic papers and patent documents to social media content. The efficient organization and understanding of massive amounts of text has become an urgent need. Text clustering, as an important branch of unsupervised learning, aims to automatically group similar texts, providing fundamental support for applications such as knowledge discovery, information retrieval, and recommendation systems. However, existing text clustering technologies reveal the following problems when faced with large-scale, high-dimensional, and semantically complex text data: I. Difficulty in determining the number of clusters Traditional methods (such as K-means and hierarchical clustering) rely on empirically pre-setting the number of clusters or on heuristic metrics (such as the elbow method and silhouette coefficient) for estimation. These metrics have poor stability in high-dimensional sparse text feature spaces and ignore the deep semantics of the text, causing the number of clusters to deviate from the true distribution of topics. For example, technical topics in patent literature are often hierarchical, and a fixed number of clusters is insufficient to capture fine-grained innovation points.

[0003] II. Imbalance between computational efficiency and accuracy Hierarchical clustering: requires calculating an O(n²) distance matrix and the merging process is O(n³), and millions of texts require TB-level memory, making it unsuitable for industrial applications.

[0004] K-means: Although it has linear complexity, it is sensitive to initial values, requires a preset K value, and is based on Euclidean distance. It is also sensitive to the "curse of dimensionality" in high-dimensional text vectors and often gets stuck in local optima.

[0005] DBSCAN: Relies on density parameters. When there are large differences in text spatial density, it tends to break down high-density topics or merge low-density noise, resulting in insufficient stability.

[0006] III. Lack of Semantic Understanding Traditional methods rely on statistical features such as word frequency or TF-IDF, which cannot distinguish between contextual semantics such as "apple (fruit)" and "apple (company)," leading to clustering results that deviate from human cognition. Although some studies in recent years have introduced pre-trained word vectors (such as Word2Vec and BERT), they are only used for text representation and do not participate in cluster number optimization or hierarchy construction, so the semantic advantages are not sustained throughout the process.

[0007] IV. Insufficient Parameter Adaptability Existing technologies require manual adjustment of hyperparameters such as distance threshold, density radius, and minimum number of samples. The text length and topic distribution vary greatly across different fields (such as medical, legal, and social media), making parameter transfer difficult and maintenance costs high. Summary of the Invention

[0008] The technical problem to be solved by this invention is to address the shortcomings of existing technologies, specifically by providing a two-stage intelligent text clustering method based on a large language model, as detailed below: 1) In a first aspect, the present invention provides a two-stage intelligent text clustering method based on a large language model, the specific technical solution of which is as follows: The acquired text to be processed is converted into a set of multi-source text vectors, and the information entropy of each text component is combined to determine a standardized unified text representation vector set. The initial cluster number is dynamically calculated to determine the pre-clustering result, which includes cluster labels and cluster centers, corresponding to the standardized unified text representation vector set. Based on the pre-clustering results, the Ward distance between clusters is calculated, and after introducing cluster size weighting correction, a clustering tree is constructed. Based on the MDL principle, a candidate cluster number set is determined, and the cluster pairs to be merged under each cluster number are determined by reverse access. The optimal cluster number is determined based on all cluster pairs to be merged. Based on the optimal number of clusters, the target clustering results at different levels are extracted from the clustering tree, and the target cluster labels corresponding to different levels are extracted. Based on the target clustering results and target cluster labels, the clustering results corresponding to the text to be processed are generated.

[0009] The beneficial effects of the two-stage intelligent text clustering method based on a large language model provided by this invention are as follows: By fusing multi-source text vectors with information entropy to obtain a more semantically unified representation, and then using dynamic initial cluster numbers to complete efficient pre-clustering, a clustering tree is constructed by weighting Ward distance based on cluster size. The optimal cluster number is selected based on MDL range and reverse sampling combined with semantic judgment of large model. Finally, the corresponding level output results are extracted, so that the determination of the cluster number is changed from statistical driving to semantic driving. It achieves fast clustering of millions of texts while maintaining O(n²) complexity, and the output level results are consistent with human expert cognition κ>0.8. It can be directly used for knowledge discovery of large-scale texts in multiple fields without the need for manual preset cluster numbers or parameter tuning.

[0010] Based on the above solution, the present invention can be further improved as follows.

[0011] Furthermore, the text to be processed can be one of the following: patent document text, academic paper text, news article text, social media content text, or corporate document text.

[0012] Furthermore, the method for determining the multi-source text vector set is as follows: Based on the pre-defined extraction criteria, the text to be processed is vectorized to generate a 768-dimensional vector. The 768-dimensional vector is stored in HDFS format to obtain a multi-source text vector set.

[0013] Furthermore, the process of determining the standardized unified text representation vector set by combining the information entropy of each text component is as follows: Obtain the information entropy of each text component and calculate the normalized weights based on all information entropies; A length correction factor is introduced to correct the normalized weights, resulting in corrected weights. A unified text representation vector set is obtained by processing the multi-source text vector set based on the corrected weights; The standardized unified text representation vector set is obtained by standardizing the unified text representation vector set.

[0014] The beneficial effects of the above-mentioned further solutions are as follows: Information entropy is used to measure the semantic density of different text components such as titles, abstracts, and claims, and normalized weights are generated. Then, a length correction factor is applied to suppress the excessive amplification of weights by long texts, so that short but information-rich components are given higher weights. The resulting weighted and fused unified text representation vector set reduces noise while maintaining topic focus. Subsequently, standardization is performed to eliminate dimensional differences, ultimately increasing the average cosine similarity of texts on the same topic in the vector space by 12.7% and expanding the inter-class distance between different topics by 8.3%. This provides highly discriminative and numerically stable input for subsequent clustering steps, significantly reducing the numerical error of Ward distance calculation and the number of convergence iterations.

[0015] Furthermore, the process of determining the pre-clustering results includes: The clustering model is initialized by initializing the number of clusters, and the training result with the least inertia is retained. The cluster labels and cluster centers corresponding to the target clustering model are extracted from the training results, and the pre-clustering results are constructed based on the cluster labels and cluster centers.

[0016] The beneficial effects of the above-mentioned further solutions are as follows: By directly initializing KMeans with the dynamic initial cluster number K0 and retaining the training result with the minimum inertia, the cluster label and cluster center can be uniquely determined, avoiding the bias caused by manually setting the K value. At the same time, it ensures that the pre-clustering result has the minimum intra-cluster variance at the starting point of the Ward distance calculation, making the correspondence between the subsequent hierarchical merging order and LLM semantic judgment more stable. It can be directly used to build the clustering tree without additional parameter tuning.

[0017] Furthermore, the process of constructing a clustering tree specifically includes: Using the set of cluster centers and the set of cluster sample numbers from the pre-clustering results as input, combined with Ward distance and introduced cluster size weighting correction, a compressed weighted distance matrix is ​​calculated. The weighted distance matrix is ​​processed using the nearest neighbor chain algorithm to generate the original link matrix arranged in the merging order; The original linkage matrix is ​​converted to SciPy linkage format to obtain a children array containing the cluster index, distance, and number of new cluster samples for each merge, thus forming a clustering tree.

[0018] The beneficial effects of the above-mentioned further solutions are as follows: The compression matrix is ​​calculated by weighting the Ward distance by cluster size, and then the children array in SciPy linkage format is generated by the nearest neighbor chain algorithm to build a clustering tree. This reduces the complexity to O(n²) and saves 50% of memory, while maintaining the minimum variance merging order, providing a directly truncated hierarchical skeleton for subsequent reverse sampling and LLM semantic judgment.

[0019] Furthermore, the process of determining the optimal number of clusters is as follows: Based on the MDL principle, a candidate cluster set is generated. For each candidate value k in the candidate cluster set, the children array of the cluster tree is accessed in reverse order from bottom to top to extract the cluster pairs to be merged, and a sample is randomly selected from each cluster pair to form a sample pair set. Input the set of sample pairs into the large language model, determine whether the sample pairs belong to the same cluster, and obtain the predicted labels; Based on the predicted labels and the true labels given by the current hierarchical structure, calculate the weighted F-beta score, and take the candidate value with the largest weighted F-beta score as the optimal number of clusters.

[0020] The beneficial effects of the above-mentioned further solutions are as follows: After limiting the range of candidate cluster numbers using the MDL principle, the children array is accessed in reverse order from bottom to top to extract cluster pairs to be merged and handed over to the large language model for semantic consistency judgment. The weighted F-beta score is used to directly measure the matching degree between the model prediction and the hierarchical structure. The candidate value with the highest score is determined as the optimal number of clusters, realizing automatic optimization of the number of clusters from statistical driving to semantic driving, avoiding the bias caused by manually setting the K value, and ensuring that the selected number of clusters is highly consistent with expert cognition while maintaining the integrity of the hierarchy.

[0021] 2) Secondly, the present invention also provides a two-stage intelligent text clustering system based on a large language model, the specific technical solution of which is as follows: The standardization module is used to: convert the acquired text to be processed into a set of multi-source text vectors, and combine the information entropy of each text component to determine the standardized unified text representation vector set; The determination module is used to: determine the pre-clustering result, which includes cluster labels and cluster centers, corresponding to the standardized unified text representation vector set by dynamically calculating the initial cluster number; The calculation module is used to: calculate the Ward distance between clusters based on the pre-clustering results, and construct the clustering tree after introducing a cluster size weighting correction; The processing module is used to: determine the candidate cluster number set based on the MDL principle, determine the cluster pairs to be merged under each cluster number corresponding to the cluster tree by reverse access, and determine the optimal cluster number based on all cluster pairs to be merged; The clustering module is used to: extract target clustering results at different levels from the clustering tree based on the optimal number of clusters, extract target cluster labels corresponding to different levels, and generate clustering results corresponding to the text to be processed based on the target clustering results and target cluster labels.

[0022] Based on the above solution, the present invention can be further improved as follows.

[0023] Furthermore, the text to be processed can be one of the following: patent document text, academic paper text, news article text, social media content text, or corporate document text.

[0024] Furthermore, the method for determining the multi-source text vector set is as follows: Based on the pre-defined extraction criteria, the text to be processed is vectorized to generate a 768-dimensional vector. The 768-dimensional vector is stored in HDFS format to obtain a multi-source text vector set.

[0025] Furthermore, the process of determining the standardized unified text representation vector set by combining the information entropy of each text component is as follows: Obtain the information entropy of each text component and calculate the normalized weights based on all information entropies; A length correction factor is introduced to correct the normalized weights, resulting in corrected weights. A unified text representation vector set is obtained by processing the multi-source text vector set based on the corrected weights; The standardized unified text representation vector set is obtained by standardizing the unified text representation vector set.

[0026] Furthermore, the process of determining the pre-clustering results includes: The clustering model is initialized by initializing the number of clusters, and the training result with the least inertia is retained. The cluster labels and cluster centers corresponding to the target clustering model are extracted from the training results, and the pre-clustering results are constructed based on the cluster labels and cluster centers.

[0027] Furthermore, the process of constructing a clustering tree specifically includes: Using the set of cluster centers and the set of cluster sample numbers from the pre-clustering results as input, combined with Ward distance and introduced cluster size weighting correction, a compressed weighted distance matrix is ​​calculated. The weighted distance matrix is ​​processed using the nearest neighbor chain algorithm to generate the original link matrix arranged in the merging order; The original linkage matrix is ​​converted to SciPy linkage format to obtain a children array containing the cluster index, distance, and number of new cluster samples for each merge, thus forming a clustering tree.

[0028] Furthermore, the process of determining the optimal number of clusters is as follows: Based on the MDL principle, a candidate cluster set is generated. For each candidate value k in the candidate cluster set, the children array of the cluster tree is accessed in reverse order from bottom to top to extract the cluster pairs to be merged, and a sample is randomly selected from each cluster pair to form a sample pair set. Input the set of sample pairs into the large language model, determine whether the sample pairs belong to the same cluster, and obtain the predicted labels; Based on the predicted labels and the true labels given by the current hierarchical structure, calculate the weighted F-beta score, and take the candidate value with the largest weighted F-beta score as the optimal number of clusters.

[0029] 3) In a third aspect, the present invention also provides an electronic device, the electronic device including a processor coupled to a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the electronic device to perform any of the methods described above.

[0030] 4) In a fourth aspect, the present invention also provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above methods.

[0031] It should be noted that the beneficial effects of the technical solutions of the second to fourth aspects of the present invention and their corresponding possible implementations can be found in the above description of the technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0032] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating a two-stage intelligent text clustering method based on a large language model, according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the multi-source text feature fusion process of a two-stage intelligent text clustering method based on a large language model, according to an embodiment of the present invention. Figure 3 This is a flowchart of a two-stage clustering algorithm for a two-stage intelligent text clustering method based on a large language model, according to an embodiment of the present invention. Figure 4 This is a hierarchical tree structure diagram of a two-stage clustering method based on a large language model guided by an embodiment of the present invention. Figure 5 This is a diagram showing the hierarchical tree structure and the correspondence between the children array and a two-stage intelligent text clustering method based on a large language model, according to an embodiment of the present invention. Figure 6 This is a diagram illustrating the sampling process based on the children array in a two-stage intelligent text clustering method guided by a large language model, according to an embodiment of the present invention. Figure 7 This is a detailed diagram illustrating the F-beta score calculation process of a two-stage intelligent text clustering method based on a large language model, according to an embodiment of the present invention. Figure 8 This is an example diagram illustrating the numerical calculation of F-beta scores for a two-stage intelligent text clustering method based on a large language model, according to an embodiment of the present invention. Figure 9 This is a schematic diagram illustrating the optimal clustering number selection process of a two-stage intelligent text clustering method based on a large language model according to an embodiment of the present invention. Figure 10 This is a structural framework diagram of an electronic device according to the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0034] For ease of understanding, the terms and symbols used in this scheme are explained in a unified manner and will not be repeated hereafter: N: Total sample size (number of texts); D: Vector dimension (BERT output 768); K0: Dynamic initial cluster number ; K_max / K_min: The maximum / minimum number of clusters allowed by the system (default 2048 / 2); s: Expected number of samples per cluster, fixed at 4; α: Length correction factor, fixed at 0.1; H(Ti): Information entropy of the i-th text component; p( |Ti): word The probability in component Ti; wi / wi′: Normalized weights and length-corrected weights; E ij : The squared Euclidean distance between the centers of clusters i and j; |C i |: The number of samples in cluster i; D ij Weighted Ward distance = ·(|C i |·|C j |) / (|C i |+|C j |); ward_dist: The compressed one-dimensional distance vector; raw_linkage: The raw merge matrix output by the nearest neighbor chain; children: A SciPy linkage format matrix, with each row containing [c1, c2, dist, n_samples]; nodes: list, nodes[i] stores the index of the sample contained in node i; k: Number of candidate clusters, the range of which is given by the MDL formula; β: F-beta weight, fixed at 0.92; n: Current number of samples (same as N); sample_id: A unique identifier for the text; cluster_centers_: The cluster center matrix output by KMeans; labels_: The array of cluster labels output by KMeans; inertia_: Sum of squares within a KMeans cluster, used to select the optimal run; tol: Convergence threshold, 1e-6; The row number nk of the linkage matrix indicates the breakpoint position at cluster k. Semantic-driven clustering refers to a method that leverages the deep semantic understanding capabilities of large language models to guide the clustering process by analyzing the semantic similarity of texts rather than relying solely on statistical features.

[0035] Cognitive consistency refers to the degree of consistency between clustering results and human expert judgment. It can be quantified by manually labeled consistency coefficient κ, where κ>0.8 indicates high consistency.

[0036] Information density: refers to the amount of effective semantic information contained within a unit of text length, expressed as information entropy H(X) = calculate.

[0037] Two-stage hybrid clustering: refers to a hybrid strategy that first uses KMeans for coarse-grained pre-clustering, and then uses hierarchical clustering for fine-grained processing.

[0038] The children array refers to the data structure in hierarchical clustering that records the history of cluster merging. It adopts the SciPy linkage matrix format, and each row records [cluster 1 index, cluster 2 index, distance, new cluster size].

[0039] like Figure 1 As shown in the figure, a two-stage intelligent text clustering method based on a large language model guided by this invention includes the following steps: S1, convert the acquired text to be processed into a multi-source text vector set, and combine the information entropy of each text component to determine the standardized unified text representation vector set; S2, the pre-clustering result containing cluster labels and cluster centers corresponding to the standardized unified text representation vector set is determined by dynamically calculating the initial cluster number; S3. Based on the pre-clustering results, calculate the Ward distance between clusters, introduce cluster size weighting correction, and construct the clustering tree; S4. Based on the MDL principle, determine the candidate cluster number set, and determine the cluster pairs to be merged under each cluster number in the cluster tree by reverse access. Determine the optimal cluster number based on all cluster pairs to be merged. S5. Based on the optimal number of clusters, extract the target clustering results at different levels from the clustering tree, and extract the target cluster labels corresponding to different levels. Generate the clustering results corresponding to the text to be processed based on the target clustering results and target cluster labels.

[0040] The beneficial effects of the two-stage intelligent text clustering method based on a large language model provided by this invention are as follows: By fusing multi-source text vectors with information entropy to obtain a more semantically unified representation, and then using dynamic initial cluster numbers to complete efficient pre-clustering, a clustering tree is constructed by weighting Ward distance based on cluster size. The optimal cluster number is selected based on MDL range and reverse sampling combined with semantic judgment of large model. Finally, the corresponding level output results are extracted, so that the determination of the cluster number is changed from statistical driving to semantic driving. It achieves fast clustering of millions of texts while maintaining O(n²) complexity, and the output level results are consistent with human expert cognition κ>0.8. It can be directly used for knowledge discovery of large-scale texts in multiple fields without the need for manual preset cluster numbers or parameter tuning.

[0041] In another embodiment of this solution, the specific implementation process of S1 includes: Taking patent document text as an example, the text is parsed to extract three text components: title, abstract, and claims. For each component, a 768-dimensional vector is generated using the BGE-large-zh-v1.5 embedding model, resulting in a multi-source text vector set. Then, according to the formula H(Ti)= Calculate the information entropy of each component and normalize the weights. = Based on this, a length correction factor is then introduced. = ×(1+αlog(|Ti| / |T|)), α=0.1, weighted fusion of title, abstract, and claim vectors to obtain a unified text representation vector set; finally, StandardScaler is used to perform Z-score standardization on the fused vectors, with optional PCA retaining 95% variance contribution rate, outputting the standardized unified text representation vector set for direct use in subsequent clustering steps.

[0042] Furthermore, it should be noted that for non-patent text, the three text components are mapped according to information density and structural stability as follows: title → "short title" or "first-level title", abstract → "content abstract" or "introduction", and claims → "core paragraph" or "key sentence". In specific extraction, the short title is taken from the document's own title or the first 30 characters of the first line, the content abstract is taken from the first paragraph or the first 75 words automatically generated, and the core paragraph is taken from the consecutive sentences with the highest TF-IDF weight as the technical / thematic points. Then, the information entropy and weight are calculated and fused and standardized in the same way as in the patent process.

[0043] In another embodiment of this solution, the specific implementation process of S2 is as follows: First, calculate the dynamic initial cluster number K0 = max(min(N÷4,2048),2) based on the number of samples N. Then, initialize the KMeans model with k-means++, taking the standardized unified text representation vector set as input. Perform training with a maximum of 1000 iterations per round and a convergence threshold of 1e-6 while keeping the inertia minimum. After obtaining the target clustering model, directly extract its labels_ as cluster labels and cluster_centers_ as cluster centers, and encapsulate them together with the cluster size as pre-clustering results for subsequent Ward hierarchical clustering.

[0044] In another embodiment of this solution, the specific implementation process of S3 is as follows: Using the cluster center set μ and the cluster sample number set |C| from the pre-clustering results as input, first calculate the Euclidean distance squared matrix E, then follow the formula... = ×(| |·| |) / (| |+| |) The weighted distance is obtained element by element, compressed into a one-dimensional ward_dist vector using condensed_index, and then input into the nearest neighbor chain algorithm nn_chain_from_middle. The raw_linkage matrix is ​​output in the order of minimum merging cost, and then converted into a children array in SciPy linkage format. Each row records the merged cluster index, distance, and number of new cluster samples, forming a clustering tree that can be accessed directly in reverse order.

[0045] In another embodiment of this solution, the specific implementation process of S4 is as follows: First, calculate the boundary value using the MDL principle to obtain the set of candidate cluster numbers. For each k, the cluster tree children array is accessed in reverse order from bottom to top. The index of the cluster pair corresponding to the kth merge is taken. One record is randomly selected from each of the two cluster sample sets to form a set of deduplicated sample pairs. This set is input into the large language model to determine whether they are in the same cluster to obtain the predicted label. After comparing it with the true label of the current hierarchical structure, the weighted F-beta score is calculated according to β=0.92. After traversing all k, the one with the largest score is selected as the optimal number of clusters.

[0046] In another embodiment of this solution, the specific implementation process of S5 is as follows: After obtaining the optimal number of clusters k, locate the nk-th row in the linkage matrix of the clustering tree, truncate the children array according to the merging history of that row, and obtain the leaf node indices of k clusters. Expand the sample IDs contained in each leaf node using the pre-stored nodes list, and assign cluster numbers 0…k-1 to form the target cluster label. At the same time, extract the cluster center and the list of samples within the cluster corresponding to this level and encapsulate them into the target clustering result. Write the sample ID, cluster label, cluster center and tree structure together into HDF5 / CSV to complete the output of the clustering result of the text to be processed.

[0047] Furthermore, the text to be processed can be one of the following: patent document text, academic paper text, news article text, social media content text, or corporate document text.

[0048] Furthermore, the method for determining the multi-source text vector set is as follows: Based on the pre-defined extraction criteria, the text to be processed is vectorized to generate a 768-dimensional vector. The 768-dimensional vector is stored in HDFS format to obtain a multi-source text vector set.

[0049] Furthermore, the process of determining the standardized unified text representation vector set by combining the information entropy of each text component is as follows: Obtain the information entropy of each text component and calculate the normalized weights based on all information entropies; A length correction factor is introduced to correct the normalized weights, resulting in corrected weights. A unified text representation vector set is obtained by processing the multi-source text vector set based on the corrected weights; The standardized unified text representation vector set is obtained by standardizing the unified text representation vector set.

[0050] Furthermore, the process of determining the pre-clustering results includes: The clustering model is initialized by initializing the number of clusters, and the training result with the least inertia is retained. The cluster labels and cluster centers corresponding to the target clustering model are extracted from the training results, and the pre-clustering results are constructed based on the cluster labels and cluster centers.

[0051] Furthermore, the process of constructing a clustering tree specifically includes: Using the set of cluster centers and the set of cluster sample numbers from the pre-clustering results as input, combined with Ward distance and introduced cluster size weighting correction, a compressed weighted distance matrix is ​​calculated. The weighted distance matrix is ​​processed using the nearest neighbor chain algorithm to generate the original link matrix arranged in the merging order; The original linkage matrix is ​​converted to SciPy linkage format to obtain a children array containing the cluster index, distance, and number of new cluster samples for each merge, thus forming a clustering tree.

[0052] Furthermore, the process of determining the optimal number of clusters is as follows: Based on the MDL principle, a candidate cluster set is generated. For each candidate value k in the candidate cluster set, the children array of the cluster tree is accessed in reverse order from bottom to top to extract the cluster pairs to be merged, and a sample is randomly selected from each cluster pair to form a sample pair set. Input the set of sample pairs into the large language model, determine whether the sample pairs belong to the same cluster, and obtain the predicted labels; Based on the predicted labels and the true labels given by the current hierarchical structure, calculate the weighted F-beta score, and take the candidate value with the largest weighted F-beta score as the optimal number of clusters.

[0053] Example 1, such as Figure 2 As shown, (I) Intelligent fusion technology of multi-source text features 1. Multi-level text vectorization The BGE-large-zh-v1.5 embedding model is used to vectorize the title, abstract, and claims of the text, resulting in 768-dimensional vectors. HDF5 format is used for storage, supporting efficient reading and writing of large-scale data.

[0054] 2. Adaptive weight fusion based on information entropy Information density of each text component is determined through information entropy analysis: Information entropy calculation: H(Ti) = -Σjp( |Ti)×log2(p( |Ti)) Weight normalization: wi = H(Ti) / Σ k H(T) Length correction: wi' = wi × (1 + α) log(|Ti| / |T T |)), where α=0.1 3. Vector standardization Z-score standardization was performed using Standard Scaler, and optional PCA dimensionality reduction maintained a 95% variance contribution rate.

[0055] (II) Two-stage intelligent clustering algorithm To balance the stringent computational efficiency requirements of large-scale text processing with the ability of hierarchical clustering to preserve structural integrity, this invention adopts a "coarse-to-refined" strategy in its macro-architecture: The first stage utilizes lightweight K-Means to rapidly compress millions of texts into thousands of cluster centers, pruning while preserving local structure; the second stage, based on pre-clustering, uses an improved Ward hierarchical clustering method for further merging and refinement, ultimately outputting a clustering hierarchy highly consistent with human cognition. This two-stage design reduces the time complexity from the traditional O(n³) to O(n²), and the peak memory usage decreases by 50%. Simultaneously, through the relay mechanism of "pre-clustering-re-clustering," it avoids the cluster number sensitivity of K-Means and overcomes the scalability bottleneck of pure hierarchical clustering, laying a high-quality, scalable clustering framework for subsequent LLM semantic optimization. Figure 3 As shown.

[0056] 1. First stage: Adaptive KMeans pre-clustering The first stage employs adaptive K-Means pre-clustering. A dynamic cluster number determination strategy is used, with the initial cluster number K0 calculated according to the following rules: = max{ min{ N÷s , K_max} , K_min} Where N is the total number of samples, s is the expected number of samples per cluster (default 4), and K_max and K_min are the maximum and minimum allowed number of clusters, respectively.

[0057] When executing K-Means, the initial centroids are selected using the k-means++ method, and the initialization is repeated 100 times. The maximum number of iterations per round is 1000, and the convergence threshold is [not specified]. And enable multi-threaded parallel computing.

[0058] 2. Second stage: Improved hierarchical clustering like Figure 4 As shown, the second stage is an improved hierarchical clustering (Ward), which requires first calculating the Euclidean distance, and then performing a weighted correction based on cluster size. For any two clusters... , The cost of the merger Defined as: = (‖ - ||²) · (| |·| |) / (| | + | |) in , As the cluster center, | |、| | represents the cluster size. The time complexity is reduced to O(N²) using the nearest neighbor algorithm, and a compressed index is used to store the distance matrix, reducing memory usage by 50%.

[0059] Core algorithm optimizations include: By using the nearest neighbor chain algorithm (nn_chain_from_middle), the complexity is reduced from O(n³) to O(n²).

[0060] Use compressed storage: The condensed_index function compresses the distance matrix, saving 50% of memory.

[0061] Clustering tree construction: Records children information and supports querying clustering results at any granularity.

[0062] (III) Intelligent clustering number determination technology based on large language models 1. Multi-granularity intelligent sample pair extraction strategy This invention employs a multi-granularity intelligent sample pair extraction strategy and determines the range of cluster numbers based on the information theory MDL principle.

[0063] 1) Determine the range of cluster numbers Determining the range of cluster numbers based on the information theory MDL principle: , 2), It also supports user-defined ranges.

[0064] 2) Core sampling algorithm: The `children[-step]` array iterates through the clusters within the specified number of clusters using a `for` loop. It then retrieves the clusters to be merged using reverse access, randomly selecting samples from each cluster and removing duplicates using `tuple(sorted([idx1, idx2]))`. The mathematical principle behind this reverse access is as follows: `children[-step]` retrieves the step-th merging operation from the end, where `step=k` corresponds to a level split with cluster size `k`, and each merging operation corresponds to a split point in the cluster tree.

[0065] Python for step in range(min_clusters, max_clusters): cls_idx1, cls_idx2 = children[-step] # Reverse order access to get the clusters to be merged cls1_samples = nodes[cls_idx1] # Samples of the first cluster cls2_samples = nodes[cls_idx2] # Samples of the second cluster idx1 = random.choice(cls1_samples) # Randomly select samples idx2 = random.choice(cls2_samples) sample_pair = tuple(sorted([idx1, idx2])) # Deduplication ``` In the clustering tree, the k-th merge operation, starting from the bottom up, corresponds to a partition with cluster number k. For each candidate cluster number k, the system selects the two nodes from that partition that are about to be merged. , and from , Each sample set represents a random sample, forming a sample pair (x1, x2). To avoid duplication, the sample pairs are sorted in ascending order by index and then stored in a set.

[0066] like Figure 5 The diagram shows that KMeans pre-clustering produces four initial clusters, which are then merged three times to form a complete clustering tree. The `children` array records the merging history: `children[0] = [cluster 1, cluster 2]`, `children[1] = [cluster 3, cluster 4]`, `children[2] = [new cluster 5, new cluster 6]`. This diagram uses the SciPy linkage matrix format, where each merging operation corresponds to a split point in the clustering tree.

[0067] like Figure 6 The following details the sampling strategy for reversing the order of the children array. When step=2, children[-2] corresponds to children[1], resulting in clusters 3 and 4; when step=3, children[-3] corresponds to children[0], resulting in clusters 1 and 2. This reversing order ensures that the sample pairs selected when the number of clusters = k come from the two clusters to be merged, avoiding sampling bias. The sampling process adopts a hierarchical sampling strategy, where the number of sample pairs at each granularity is proportional to the importance of that granularity.

[0068] 2. LLM Integrated Forecasting System The Prompt template design used in the LLM integrated forecasting system: Patent 1:(title + abstract) {text content 1} Patent 2:(title + abstract) {text content 2} Please determine whether these two patents belong to the same cluster. Answer only "Yes" or "No". The system supports multiple models such as GPT-4, DeepSeek-R1, and Qwen3, and provides a fault-tolerant retry mechanism.

[0069] Determining the optimal number of clusters based on F-beta scores The core innovation of this invention is the determination of the optimal number of clusters based on F-beta scoring. This method constructs a correspondence between real labels and predicted labels and uses the semantic judgment ability of a large language model to evaluate the clustering quality under different numbers of clusters.

[0070] like Figures 7 to 9 As shown, the specific implementation process includes the following key steps: First, the labeling strategy employs a dual-label system. The true label (cur_assign_pair) determines whether sample pairs belong to the same cluster based on hierarchical clustering results: For each pair of samples, obtain the cluster assignments of the two samples under the current number of clusters. If send1_assign == send2_assign, mark it as 1 (same cluster); otherwise, mark it as 0 (different cluster).

[0071] The predicted labels (cur_preds) are the prediction results of the large language model: the "Yes" / "No" answers of the LLM are converted into 1 / 0 binary labels, where pred['prediction'][0]=='Yes' corresponds to 1, indicating that the LLM believes that the two samples should belong to the same cluster.

[0072] Secondly, the sample weight calculation mechanism takes into account the impact of cluster size on clustering decisions. The theoretical basis for this weight design is that merging decisions between larger clusters are more important than merging smaller clusters because merging errors in large clusters will affect more samples; the use of a square root function avoids excessive weight differences and maintains the balance of evaluation; the weight is proportional to the square root of the product of cluster sizes, reflecting the importance of inter-cluster interactions.

[0073] Then, the F-beta scoring optimization mechanism uses a parameter setting of β=0.92. The F-beta score calculation formula is: Fβ = (1+β²) × (precision×recall) / (β²×precision+recall) When β=0.92, β²=0.8464, making the recall weight 1.18 times the precision weight. The theoretical basis for this parameter selection includes: based on the principle of minimum description length, in hierarchical clustering, the propagation impact of false negative errors (missing relevant sample pairs) is greater than that of false positive errors (falsely detecting irrelevant sample pairs), therefore recall needs to be given more importance; cognitive psychology research shows that humans tend to include boundary samples rather than miss core relevant samples in clustering judgments; from an information theory perspective, β=0.92 can minimize the information entropy of the clustering results, achieving optimal information compression.

[0074] Finally, the optimal cluster number selection strategy calculates the F-beta score by traversing all candidate cluster numbers. For each candidate value cls_idx within the range of cluster numbers, the system calculates the F-beta score for all sample pairs under that cluster number, and then selects the cluster number with the highest score as the optimal solution. This strategy ensures that the selected cluster number maximizes the consistency between the LLM semantic judgment and the actual clustering results, realizing a paradigm shift from statistically driven to semantically driven cluster number determination.

[0075] (iv) Result generation and output technology The system provides multi-granularity clustering results output, including fine-grained results from KMeans pre-clustering, coarse-grained results corresponding to the optimal number of clusters, and clustering tree structures that support queries at any granularity. Quality assessment calculates quality metrics such as silhouette coefficient and Calinski-Harabasz index, and supports exporting in multiple formats including CSV, JSON, and HDF5.

[0076] Fine-grained: KMeans pre-clustering results Coarse-grained: Result corresponding to the optimal number of clusters Clustering tree: Supports queries at any granularity (v) Algorithm robustness guarantee technology This invention includes a comprehensive anomaly handling mechanism, including an adaptive resampling strategy for handling data imbalance, anomaly sample handling based on the Local Outlier Factor (LOF), and a multi-level degradation strategy to ensure API service stability. The algorithm's convergence is guaranteed based on the monotonically decreasing property of the energy function: E(t) = ∈ || - ||², ensuring that E(t+1) ≤ E(t).

[0077] (vi) Algorithm Implementation Process 1. Data preprocessing stage Environment configuration: Set parallel optimization parameters such as OMP_NUM_THREADS=20 Data loading: Supports JSON, CSV and other formats, extracts titles, abstracts and claims. Vectorization: Using the BGE-large-zh-v1.5 model, stored in HDF5 format. 2. Two-stage clustering implementation KMeans pre-clustering: dynamically determines the initial number of clusters, initialized using k-means++. Hierarchical clustering: Ward distance calculation, optimization of the nn_chain_from_middle algorithm Key implementation details: 1) Compressed index storage: Using SciPy's condensed_index format, the n×n distance matrix is ​​compressed into a one-dimensional array of n(n-1) / 2, saving 50% of memory. 2) Multi-threaded parallel computation: OpenMP is used to parallelize distance computation. The number of threads is controlled by setting the OMP_NUM_THREADS environment variable to avoid the GIL limitation. 3) The children array format uses the SciPy linkage matrix format, with each row containing four elements: [cluster_1, cluster_2, distance, sample_count]. 4) Thread safety mechanism: LLM API calls use thread pools and semaphores to control the number of concurrent calls, avoiding frequency limitations. 3. Optimization of LLM cluster number Sample pair generation: based on reverse access of the children array LLM Prediction: Domain Adaptive Prompt, Multi-Model Support F-beta score: β=0.92, selecting the optimal number of clusters. 4. Result Generation Multi-granularity output: fine-grained, coarse-grained, clustered tree structure Quality assessment: Profile coefficient, CH index, and other indicators Export formats: CSV, JSON, HDF5, etc. Based on the above, the technical advantages of the present invention include: (a) Advantages of intelligent clustering in determining the number of clusters Semantic understanding: Utilizing the deep semantic understanding capabilities of large language models, automatically determine the optimal number of clusters. Theoretical Foundation: A complete theoretical system based on the principles of information theory (MDL) and cognitive psychology. Adaptability: Intelligent optimization of cluster number through F-beta scoring (ii) Advantages of efficient hybrid clustering architecture Computational complexity: Reduced from O(n³) to O(n²), significantly improving processing speed. Memory efficiency: Utilizing compressed storage, memory utilization efficiency is improved by 50%. Parallel processing: Supports multi-threaded parallel computing, making full use of hardware resources. (III) Advantages of Multi-Source Information Fusion Information density adaptation: a weight allocation strategy based on information entropy Feature quality assurance: Integrating vector quality detection and anomaly handling mechanisms Dimensionality reduction optimization: PCA dimensionality reduction can be used to reduce complexity while retaining 95% of the information. (iv) Engineering advantages Modular architecture: easy to deploy, maintain and extend Fault tolerance mechanism: Automatic retry of failed API calls to ensure system stability. Supports multiple formats: CSV, JSON, HDF5 and other data formats. In the above embodiments, although the steps are numbered S1, S2, etc., they are only specific embodiments given by the present invention. Those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0078] This invention also provides a two-stage intelligent text clustering system based on a large language model, the specific technical solution of which is as follows: The standardization module is used to: convert the acquired text to be processed into a set of multi-source text vectors, and combine the information entropy of each text component to determine the standardized unified text representation vector set; The determination module is used to: determine the pre-clustering result, which includes cluster labels and cluster centers, corresponding to the standardized unified text representation vector set by dynamically calculating the initial cluster number; The calculation module is used to: calculate the Ward distance between clusters based on the pre-clustering results, and construct the clustering tree after introducing a cluster size weighting correction; The processing module is used to: determine the candidate cluster number set based on the MDL principle, determine the cluster pairs to be merged under each cluster number corresponding to the cluster tree by reverse access, and determine the optimal cluster number based on all cluster pairs to be merged; The clustering module is used to: extract target clustering results at different levels from the clustering tree based on the optimal number of clusters, extract target cluster labels corresponding to different levels, and generate clustering results corresponding to the text to be processed based on the target clustering results and target cluster labels.

[0079] This invention also provides a system for processing synchronous multi-source heterogeneous data, the specific technical solution of which is as follows: The extraction module is used to: extract the metadata configuration file of the target platform, and based on the metadata configuration file, start the Canal service program and create a Kafka messaging system. The Canal service program is used to listen to the change log in the source database, and the Kafka messaging system is used to create a message channel for each data table to be synchronized. The listening module is used to: when the Canal service program listens for a data change event, determine the bus message object corresponding to the data change time by combining the metadata configuration file. The bus message object includes: event type, SQL template and group identifier. The first processing module is used to: serialize the bus message object, generate a binary byte array Kafka message, and store the binary byte array Kafka message in the Kafka message system; The second processing module is used to: extract any target binary byte array Kafka message from the Kafka messaging system, and obtain the corresponding structured message object through deserialization processing; The generation module is used to: generate executable SQL statements corresponding to structured messages, and determine whether the target binary byte array Kafka messages are synchronized in the target database based on the executable SQL statements.

[0080] It should be noted that the beneficial effects of the two-stage intelligent text clustering system based on a large language model guided by the above embodiments are the same as those of the two-stage intelligent text clustering method based on a large language model guided by the above embodiments, and will not be repeated here. Furthermore, the system provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the system can be divided into different functional modules according to the actual situation to complete all or part of the functions described above. In addition, the system and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process is detailed in the method embodiments, and will not be repeated here.

[0081] like Figure 10 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320 coupled to a memory 310. The memory 310 stores at least one computer program 330, which is loaded and executed by the processor 320 to enable the electronic device 300 to implement any of the above-mentioned methods. Specifically: The electronic device 300 can vary considerably due to differences in configuration or performance. It may include one or more processors 320 (Central Processing Units, CPUs) and one or more memories 310. The memories 310 store at least one computer program 330, which is loaded and executed by the processors 320 to enable the electronic device 300 to implement the two-stage intelligent text clustering method based on a large language model provided in the above embodiments. Of course, the electronic device 300 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. It may also include other components for implementing device functions, which will not be elaborated upon here.

[0082] An embodiment of the present invention provides a computer-readable storage medium storing at least one computer program, which is loaded and executed by a processor to enable a computer to implement any of the above-described methods.

[0083] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0084] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform any of the methods described above.

[0085] It should be noted that the terms "first" and "second" in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0086] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0087] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0088] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A two-stage intelligent text clustering method based on a large language model, characterized in that, include: The acquired text to be processed is converted into a set of multi-source text vectors, and the information entropy of each text component is combined to determine a standardized unified text representation vector set. The pre-clustering result, which includes cluster labels and cluster centers, corresponding to the standardized unified text representation vector set is determined by dynamically calculating the initial cluster number. Based on the pre-clustering results, the Ward distance between clusters is calculated, and after introducing cluster size weighting correction, a clustering tree is constructed. Based on the MDL principle, a candidate cluster number set is determined, and the cluster pairs to be merged under each cluster number corresponding to the cluster tree are determined by reverse access. The optimal cluster number is determined based on all cluster pairs to be merged. Based on the optimal number of clusters, target clustering results at different levels are extracted from the clustering tree, and target cluster labels corresponding to different levels are extracted. Based on the target clustering results and the target cluster labels, clustering results corresponding to the text to be processed are generated.

2. The two-stage intelligent text clustering method based on a large language model as described in claim 1, characterized in that, The text to be processed is one of the following: patent document text, academic paper text, news article text, social media content text, and corporate document text.

3. The two-stage intelligent text clustering method based on a large language model as described in claim 1, characterized in that, The method for determining the multi-source text vector set is as follows: Based on the pre-set extraction criteria, the text to be processed is vectorized to generate a 768-dimensional vector. The 768-dimensional vector is stored in HDFS format to obtain the multi-source text vector set.

4. The two-stage intelligent text clustering method based on a large language model as described in claim 1, characterized in that, The process of determining the standardized unified text representation vector set by combining the information entropy of each text component is as follows: Obtain the information entropy of each text component and calculate the normalized weights based on all information entropies; A length correction factor is introduced to correct the normalized weights and obtain the corrected weights. Based on the corrected weights, the multi-source text vector set is processed to obtain a unified text representation vector set; The standardized unified text representation vector set is obtained by standardizing the unified text representation vector set.

5. The two-stage intelligent text clustering method based on a large language model as described in claim 1, characterized in that, The process of determining the pre-clustering results includes: The clustering model is initialized using the initial number of clusters, and the training result with the least inertia is retained. The cluster labels and cluster centers corresponding to the target clustering model corresponding to the training results are extracted, and the pre-clustering result is constructed based on the cluster labels and cluster centers.

6. The two-stage intelligent text clustering method based on a large language model as described in claim 1, characterized in that, The process of constructing the clustering tree specifically includes: Using the set of cluster centers and the set of cluster sample numbers in the pre-clustering results as input, combined with the Ward distance and introducing cluster size weighting correction, a compressed weighted distance matrix is ​​calculated. The weighted distance matrix is ​​processed using the nearest neighbor chain algorithm to generate the original link matrix arranged in the merging order; The original linkage matrix is ​​converted into SciPy linkage format to obtain a children array containing the cluster index, distance, and number of new cluster samples for each merge, thus forming the clustering tree.

7. The two-stage intelligent text clustering method based on a large language model as described in claim 1, characterized in that, The process of determining the optimal number of clusters is as follows: Based on the MDL principle, a candidate cluster set is generated. For each candidate value k in the candidate cluster set, the children array of the cluster tree is accessed in reverse order from bottom to top to extract the cluster pairs to be merged, and a sample is randomly selected from each cluster pair to form a sample pair set. The set of sample pairs is input into a large language model to determine whether the sample pairs belong to the same cluster, and the predicted labels are obtained. Based on the predicted labels and the true labels given by the current hierarchical structure, a weighted F-beta score is calculated, and the candidate value with the largest weighted F-beta score is taken as the optimal number of clusters.

8. A two-stage intelligent text clustering system guided by a large language model, characterized in that, include: The standardization module is used to: convert the acquired text to be processed into a set of multi-source text vectors, and combine the information entropy of each text component to determine the standardized unified text representation vector set; The determination module is used to: determine the pre-clustering result containing cluster labels and cluster centers corresponding to the standardized unified text representation vector set by dynamically calculating the initial cluster number; The calculation module is used to: calculate the Ward distance between clusters based on the pre-clustering results, and construct a clustering tree after introducing a cluster size weighting correction; The processing module is used to: determine the candidate cluster number set based on the MDL principle, and determine the cluster pairs to be merged under each cluster number corresponding to the clustering tree by means of reverse access, and determine the optimal cluster number based on all cluster pairs to be merged; The clustering module is used to: extract target clustering results at different levels from the clustering tree according to the optimal number of clusters, extract target cluster labels corresponding to different levels, and generate clustering results corresponding to the text to be processed based on the target clustering results and the target cluster labels.

9. An electronic device, characterized in that, The electronic device includes a processor coupled to a memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the computer to perform the method as described in any one of claims 1 to 7.

Images