LLM-based scientific literature topic discovery method and device

Abstract

The invention discloses a scientific literature topic discovery method and device based on LLM. The method comprises the following steps: 1) acquiring text representation of each scientific literature sample and encoding the text representation by using a text encoder to obtain a document representation matrix corresponding to the scientific literature sample; 2) clustering the scientific literature samples to obtain clustering results of different themes; calculating an entropy value of each scientific literature sample, and selecting a high-uncertainty sample; 3) calculating semantic similarity between each high-uncertainty sample and other scientific literature samples, and constructing a plurality of triple tasks; fine-tuning the text encoder by using each triple task through a comparative learning method; 4) encoding the text representation of each scientific literature sample by using a text encoder to obtain a document representation matrix corresponding to the scientific literature sample; and 5) performing topic clustering on each scientific literature sample by using the document representation matrix of each scientific literature sample, and generating a clustering tag and a topic division result of each scientific literature sample.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of text mining and information retrieval, and specifically relates to a scientific literature topic discovery method and device based on a large language model (LLM), which can accurately identify research topics in scientific literature and improve the accuracy and semantic consistency of topic discovery. BACKGROUND

[0002] Under the background of the continuous expansion of the frontiers of scientific research, scholars are faced with an increasingly large flow of scientific literature information, making efficient information retrieval and management crucial. In rapidly developing fields such as computer science, the large number of research papers produced each year makes it difficult for researchers to keep up with the latest research progress. Therefore, topic discovery has become a core method for scientific information retrieval, which can help researchers more easily understand the trends in different research fields and accurately locate relevant literature. However, traditional information retrieval methods, such as manual sorting or keyword search, are often difficult to capture the complex relationships between different research fields, and may even overlook emerging interdisciplinary connections. Therefore, automated scientific literature topic discovery methods are urgently needed to cope with the complexity and scale of modern scientific literature.

[0003] In recent years, the progress of machine learning, especially deep learning, has driven the development of automated topic discovery technology. Classic topic discovery methods, such as Latent Dirichlet Allocation (LDA), Non-negative Matrix Factorization (NMF), and Probabilistic Latent Semantic Analysis (PLSA), can be directly used to discover scientific topics. However, these Bag-of-Words (BoW) based methods ignore the contextual relationships between words and cannot effectively capture the complex semantics of scientific text. In addition, many topic discovery methods rely on dimensionality reduction techniques (such as PCA or UMAP), but this may result in information loss, affecting the integrity of document topics.

[0004] In recent years, Deep Embedding Methods have gradually become the mainstream approach for scientific literature analysis. These methods can map text data into high-dimensional space, thus capturing the semantic relationships between words and phrases more accurately. For example, Document Embedding methods can represent documents as continuous vectors and cluster similar research topics in semantic space. Neural Topic Models (NTMs), such as Embedded Topic Model (ETM) and Neural Variational Document Model (NVDM), can combine word embeddings to enhance the ability of topic representation.

[0005] However, despite the improvements in topic discovery brought by deep learning techniques, existing methods still face three major challenges:

[0006] 1) Lack of deep understanding of text content: Traditional topic discovery methods mostly rely on bag-of-words model or static word embeddings, which cannot fully capture the contextual semantic information of text.

[0007] 2) Insufficient handling of fuzzy topic boundaries: Existing unsupervised clustering methods often struggle to handle interdisciplinary topics in scientific literature, leading to difficulty in distinguishing similar topics.

[0008] 3) Limitations of data representation: Some methods require dimensionality reduction to improve computational efficiency, but this dimensionality reduction may result in information loss, affecting the integrity of topic discovery.

[0009] In this context, it is crucial to develop a semantic-guided topic discovery method based on LLM for comprehensive exploration of scientific literature topic discovery. SUMMARY

[0010] To address the limitations of existing technologies in scientific literature topic discovery, the purpose of the present invention is to provide a scientific topic discovery method and device based on large language model (LLM) (hereinafter referred to as SciTopic). This method uses a supervised representation learning framework, combining text embedding matrix, semantic prompt guidance, LLM-guided topic clustering, and entropy value sampling optimization to improve the accuracy of topic clustering and enhance the model's ability to distinguish fuzzy topics, thus improving the effectiveness of scientific literature topic discovery.

[0011] To achieve the above-mentioned purposes, the technical solutions of the present invention include the following contents.

[0012] A scientific literature topic discovery method based on LLM, the method comprising:

[0013] 1) Obtain the text representation of each literature sample, including the title, abstract and metadata of the paper, and generate a document representation matrix;

[0014] Wherein, the document representation matrix is obtained based on the deep encoding of the text content by the pre-trained language model;

[0015] Further, the document representation includes: using a text encoder as a pre-trained language model to encode the title, abstract and metadata respectively to generate independent representation vectors; concatenate the title representation vector h b , the abstract representation vector h b and the metadata representation vector h m to form the final document representation matrix h p ;

[0016] 2) Cluster each scientific literature sample by unsupervised clustering to obtain clustering results of different topics; calculate the entropy value of each scientific literature sample, and select the scientific literature sample with an entropy value greater than a set entropy value threshold as a high uncertainty sample; wherein, a self-selected sampling strategy based on entropy value is used to select literature samples with high clustering uncertainty, focus on optimizing the properties of fuzzy topics, and improve the robustness and computational efficiency of the model;

[0017] Further, the self-selected sampling includes:

[0018] 21) Use an unsupervised clustering algorithm to cluster the obtained document representation;

[0019] 22) Calculate the entropy value H(x) of each document and select high-uncertainty documents as optimization targets.

[0020] Select high-uncertainty documents as anchor points, randomly select two documents in each cluster, construct LLM prompt words, and construct a topic optimization task, wherein the anchor point document is used as a query document, and the selected two documents are used as selection documents, thereby constructing multiple semantic prompt words;

[0021] By controlling the entropy value, the proportion of sampled documents is dynamically adjusted to ensure that computing resources are concentrated on the most uncertain documents, reducing computing overhead.

[0022] 3) For each high-uncertainty sample, use the LLM-guided topic clustering method to calculate the semantic similarity between the high-uncertainty sample and other scientific literature samples, and construct multiple triple tasks (a, c + , c - ), wherein a is the high-uncertainty sample, c + is a scientific literature sample similar to the high-uncertainty sample, and c -a scientific literature sample unrelated to the high-uncertainty sample; and then fine-tuning the text encoder through a contrastive learning method using each triplet task (a, c + ,c - ); wherein based on the sampling results obtained in 2) (i.e., a number of literature samples with high uncertainty), a topic clustering method guided by the LLM is used to calculate the semantic similarity between the literature through a triplet task, and the quality of topic clustering is optimized;

[0023] Further, the LLM-guided topic clustering includes:

[0024] The semantic similarity is evaluated using the LLM, and the constructed semantic prompt words are input into the LLM. The LLM will give a selection result according to the prompt words. The result given by the LLM is used to construct a triplet task (a, c + ,c - ), wherein a is an anchor document, i.e., a high-uncertainty document, c + is a similar document, and c - is an unrelated document. The clustering boundary is optimized to make the literature with similar semantics more closely aggregated. The text encoder is fine-tuned through contrastive learning, thereby optimizing the representation space to make the literature of the same topic more closely aggregated and enhance the discrimination between different topics. Based on the document semantic relationship feedback by the LLM, the topic clustering is adjusted to make the fuzzy topic more clearly divided and improve the topic consistency.

[0025] 4) The text encoder after fine-tuning is used to encode the text representation of each scientific literature sample to obtain a document representation matrix of the corresponding scientific literature sample. Then, the document representation matrix of each scientific literature sample is used to perform topic clustering on each scientific literature sample to generate the clustering label and topic division result of each scientific literature sample. Based on the class-level TF-IDF (c-TF-IDF) method, the topic keywords are extracted from each cluster to improve the explainability of the topic semantics, and finally the optimized topic representation is generated.

[0026] A scientific literature topic discovery device based on an LLM, the device comprising:

[0027] a text encoder configured to obtain a text representation matrix of the literature to be analyzed based on a pre-trained model; wherein the text representation matrix is encoded based on the title, abstract, and metadata of the literature to be analyzed to obtain a semantic feature representation of the literature;

[0028] an entropy value sampling module configured to calculate the topic uncertainty of the literature and select high-entropy literature for optimization; wherein the module calculates the entropy value of each literature to filter the literature with fuzzy topic boundaries to improve the ability of the model to distinguish complex topics.

[0029] a clustering optimization module for performing LLM-guided topic clustering; wherein the module constructs triplets based on the entropy value sampling module and performs triplet tasks through the LLM to optimize topic division, so that similar documents are more closely aggregated and topic consistency is improved;

[0030] a topic optimization module for optimizing the representation matrix of the documents based on an LLM-guided learning process; wherein the module uses a contrastive loss to optimize the semantic representation of the documents, so that documents of the same topic are more concentrated, and the separation between different topics is enhanced;

[0031] a topic semanticization module for extracting topic keywords and generating topic names; wherein the module uses a class-level TF-IDF (c-TF-IDF) method to select high-weight keywords from each topic cluster to enhance the interpretability of the topics and improve the readability of topic discovery.

[0032] Further, the process of training the LLM-guided scientific literature topic discovery device includes:

[0033] using K-Means to calculate the document representation matrix to the distance of all class centers μ, and using t-distribution calculation for soft assignment, so as to calculate the probability of different categories

[0034] calculate the uncertainty (entropy) of the documents, filter high-uncertainty samples, and construct triplet tasks (a, c + ,c - ) based on the entropy value, and use cross-entropy loss with in-batch negative sampling in the training process where s(x, y) represents the similarity between documents x and y; D represents all positive and negative samples in the current batch; τ is the softmax temperature parameter, which controls the sharpness of the distribution; a smaller τ can enhance the ability to distinguish the most similar samples, while a larger τ is suitable for noisy data.

[0035] A computer device, characterized in that the computer device comprises a processor and a memory storing computer program instructions; the processor executes the computer program instructions to realize the scientific topic discovery method for LLM described in any of the above.

[0036] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program is executed by a processor to realize the above method.

[0037] Compared with the prior art, the LLM guided topic clustering method is first introduced in the scientific literature topic discovery task, which fully utilizes the advantages of LLM in text semantic understanding, combines the triple task and the entropy value sampling mechanism, and improves the clustering quality and interpretability of scientific literature. The proposed method SciTopic optimizes the embedding through LLM guidance, which makes the literature topic division more accurate, improves the applicability of interdisciplinary research, and has the following advantages:

[0038] (1) Compared with the traditional topic discovery method (such as LDA, NMF), the present application can more effectively capture the deep semantic relationship between documents, improve the topic consistency and document clustering quality;

[0039] (2) The present application shows stable topic discovery ability in different data scales and fields, which is better than all existing topic discovery methods;

[0040] (3) The present application adopts LLM guided triple optimization mechanism, which makes the topic clustering result more stable, avoids the high dependence of traditional clustering method on initial parameters, and improves the scalability of the model;

[0041] (4) The present application combines entropy value sampling strategy, which can adaptively adjust the clustering target, so that the model can effectively process interdisciplinary literature and improve the topic discovery ability of emerging research fields;

[0042] (5) In practical application scenarios, the method of the present application can improve the computing efficiency and reduce the dependence on manual annotation under the premise of ensuring the accuracy of the topic, so that the automatic analysis of large-scale scientific literature is more efficient and reliable. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 is the SciTopic model framework diagram.

[0044] Figure 2 is the c-TF-IDF schematic diagram

[0045] Figure 3 is the overall flowchart.

[0046] Figure 4 is the sensitivity analysis of hyperparameters to AI-DM topic coherence and topic diversity;

[0047] (a) Sensitivity analysis corresponding to the hyperparameter alpha, (b) Sensitivity analysis corresponding to the hyperparameter lambda. DETAILED DESCRIPTION

[0048] The present application will be further described in detail below in conjunction with the drawings, and the examples are only used to explain the present application, not to limit the scope of the present application.

[0049] The present application first introduces an LLM-guided topic clustering method in scientific literature topic discovery tasks, fully utilizes the advantages of LLM in text semantic understanding, combines the triple task and entropy sampling mechanism, and improves the clustering quality and interpretability of scientific literature. A scientific literature topic discovery method based on LLM guidance is developed, hereinafter referred to as SciTopic. SciTopic mainly includes five modules: text encoding module, entropy sampling module, clustering optimization module, topic optimization module and topic semanticization module. 1. The text encoding module uses a text representation model to encode the title, abstract and metadata of the literature to generate a representation of the document. This module ensures the richness of the text features and provides high-quality input for topic clustering. 2. In the entropy sampling module, the clustering uncertainty of each literature is calculated, and high-entropy literature is selected for optimization. The higher the entropy value, the greater the uncertainty of the literature's theme attribution, so it is more necessary to use LLM for semantic optimization. 3. The clustering optimization module uses the triples constructed in the entropy sampling module and generates semantic triples (a, c + ,c - ) through LLM. 4. The topic optimization module fine-tunes the representation model based on the cross-entropy loss L of intra-batch negative sampling to optimize the semantic representation of the document, so that the literature of the same topic is more concentrated, and the separation between different topics is enhanced. 5. The topic semanticization module uses the class-level TF-IDF (c-TF-IDF) method to select high-weight keywords from each topic cluster. SciTopic improves the accuracy and interpretability of topic modeling through the joint optimization of LLM feedback, entropy sampling and contrastive learning loss. Compared with traditional topic discovery methods (such as LDA, NMF), SciTopic can better adapt to interdisciplinary research, optimize topic boundaries, improve the robustness and generalization ability of topic clustering, and make the automatic topic analysis of scientific literature more efficient and reliable.

[0050] The method flow is as shown in Figure 3 , and the specific steps include:

[0051] 1. Problem definition

[0052] In this task, our goal is to extract meaningful research topics from a large set of conference papers. Each paper exists in the form of a document, which contains a title t, an abstract b and metadata m, which can be represented as a set of text features x1, x2, …, x n . Metadata includes key information such as paper author, publication year and conference name. Based on these text features, we cluster the papers into multiple topic categories C1, C2, …, C n . In each cluster, we further extract a set of core terms to describe the represented topic, thereby achieving efficient topic identification and research trend analysis.

[0053] 2. Text encoding module

[0054] To construct an effective representation of scientific literature for improved clustering and retrieval performance, we adopt a dynamic embedding framework. This framework can handle text at different granularities, from single sentences to complete documents, and is adaptable to different types of text, such as short titles, detailed abstracts, and complete papers. In addition, the framework supports context-aware and domain-adaptive fine-tuning to further improve the representation effect. Our method generates a document representation matrix by encoding the title, abstract, and metadata of each paper separately, and combines these embedding vectors into a unified representation. The metadata includes author lists, publication years, and conference names, providing more rich context information for each document.

[0055] Each document is divided into three core components: title, abstract, and metadata. Among them: the title and abstract are independently encoded using fine-tuned models to adapt to different granularity text structures; the metadata is combined into a single string and embedded using the same model to maintain consistency.

[0056] The document encoding process can be represented as:

[0057]

[0058] where f represents the text encoder, and represent the text content of the title, abstract, and metadata, respectively. Finally, the representation matrix vectors of the title, abstract, and metadata are concatenated (Concat) into the complete representation of the document:

[0059]

[0060] where is the final representation matrix of document j, which integrates content information (title and abstract) and context information (metadata), ensuring a more comprehensive document representation.

[0061] 3. Entropy sampling module

[0062] To maximize clustering efficiency, we use an entropy-based sampling method to select the most informative triplets, which includes the following two key steps:

[0063] The first step is to calculate the entropy value of the high uncertainty sample.

[0064] We compute the entropy of each document representation matrix to identify samples with high uncertainty in the clustering process. These high-entropy documents are considered as anchors, representing the most ambiguous clustering cases.

[0065] In the specific computation, we use the K-Means algorithm to assign each document to the nearest cluster center μ i . Meanwhile, we use the t-distribution to compute the soft assignment probability, to measure the probability of a document belonging to different clusters in a finer granularity:

[0066]

[0067] where: is the representation matrix vector of document j; μ i represents the cluster center; is the squared Euclidean distance between the document and the cluster center; α controls the degrees of freedom of the t-distribution, to adjust the flexibility of assignment. Since the cost of computing the entropy of all clusters is high, we only compute a subset of neighboring clusters to improve computational efficiency. The number of selected clusters is determined by the following formula:

[0068] φ = max(λK, 2) # (Formula 4)

[0069] where: λ is the scaling factor; K is the total number of clusters. The probability of these φ clusters is normalized:

[0070]

[0071] Finally, the entropy of document j is computed as follows:

[0072]

[0073] To balance the computational cost and the depth of analysis, we set upper and lower thresholds (σ low and σ high ) to ensure that the computation focuses on the most informative documents.

[0074] The second step is to sample from the nearest clusters

[0075] For high-entropy anchor documents, we sample candidate documents from their neighboring clusters, and select the closest candidate based on the cosine similarity of the document representation matrix. Specifically: 1. Compute the average representation matrix vector of all clusters; 2. Select the cluster closest to the anchor document (based on Euclidean distance or cosine distance); 3. According to the density of these neighboring clusters, randomly sample candidate documents in proportion to ensure the diversity of data coverage.

[0076] Finally, we construct informative triplets: (Anchor, Positive, Negative). Where: Anchor: selected from high-entropy samples; Positive: selected from the same cluster as the anchor; Negative: selected from a similar but different cluster.

[0077] Through this entropy-driven sampling method, we can: 1. Reduce LLM query frequency, reduce computing cost; 2. Improve the clarity of cluster boundaries, reduce semantic ambiguity; 3. Enhance the thematic consistency of clusters, ensure that semantically similar documents are more closely aggregated. Ultimately, this method greatly improves computational efficiency and cost-effectiveness while ensuring high-quality clustering results.

[0078] 4. Cluster optimization module

[0079] This module uses a large language model (LLM) guided triplet task to optimize text clustering. This method can reflect the user's thematic perspective and improve the semantic consistency of clustering results. Each triplet consists of an anchor and two candidate documents, represented as (a, c1, c2), and the goal is to determine which candidate document is more similar to the anchor in terms of theme. LLM, due to its strong contextual understanding and adaptive characteristics, has become an effective tool for optimizing text clustering.

[0080] First step: Triplet task to optimize clustering perspective

[0081] The core of this method is the triplet task, which allows the LLM to evaluate the thematic relevance between three elements, namely an anchor document (a) and two candidate documents (c1, c2). The LLM is provided with the following query instruction:

[0082] ρ=``Select the paper closest to~a:c1,c2,or Neither.”#(Formula 7)

[0083] The LLM processes this instruction and determines which candidate document is more similar to the anchor in terms of theme. If the LLM chooses "Neither", the triplet is excluded, ensuring that only high-quality comparisons are used to guide the clustering process. This mechanism can improve the accuracy of the context, remove ambiguous or invalid triplets, and provide cleaner data input to optimize the fine-tuning process of the text encoder.

[0084] Second step: Triplet selection mechanism to optimize clustering

[0085] Each valid triplet consists of three documents: Anchor (a): a document with high entropy or ambiguous boundaries; Positive (c + ): a document closer to the anchor in topic; Negative (c - ): a document further away in semantics. These triplets are used to fine-tune the document representation matrix model. By learning these structured contrastive relationships, the text encoder can better capture fine-grained topic differences, improving the coherence and accuracy of clustering results.

[0086] 5. Topic Optimization Module

[0087] To improve the discriminative power of the text encoder, we use triplets generated by LLM to guide the fine-tuning process. Each triplet consists of an anchor document, a topic-similar document (positive example), and a topic-irrelevant document (negative example), represented as: These triplets are generated by analyzing high-entropy anchor documents (i.e., documents with the highest uncertainty in cluster assignment). LLM ensures that the most informative positive and negative examples are selected by sampling in the nearest clusters, further optimizing topic boundaries to make similar documents more closely related and different topic documents more separated.

[0088] During fine-tuning, we use a cross-entropy loss function with in-batch negative sampling:

[0089]

[0090] where s(x, y) represents the similarity score between two document representation matrices (such as cosine similarity or dot product); represents the set of all positive and negative examples within the current batch; τ is the softmax temperature parameter, controlling the sharpness of the probability distribution: a lower τ makes the distribution more sharp, emphasizing the strongest positive-negative contrast; a higher τ makes the distribution smoother, suitable for environments with noise.

[0091] To further improve the robustness of the model, we introduce margin-based adjustment in similarity calculation:

[0092]

[0093] where f(x) is the encoding function of the text encoder; γ is the margin parameter, which is used to prevent the model from learning trivial solutions and ensure that there is a minimum semantic distinction between positive and negative samples.

[0094] The fine-tuning process iteratively updates the text encoder to learn a more compact and semantically consistent latent space. In particular, in high-entropy regions, due to unclear topic boundaries, we introduce a dynamic sampling strategy: increase the weight of high-uncertainty samples to ensure that the model prioritizes learning difficult-to-classify instances; reduce the impact of low-entropy samples to avoid overfitting to easily classified samples.

[0095] Through this strategy, the model can more efficiently learn challenging samples, ultimately improving the overall consistency and accuracy of the clustering results, especially in the processing ability of ambiguous topic areas.

[0096] The following verifies the present application on three real datasets: NeurIPS, DBLPV10, and AI-DM.

[0097] 1. Dataset Overview

[0098] Large-scale experiments were conducted on three real datasets: NeurIPS, DBLPV10, and AI-DM, which are commonly used datasets for scientific literature topic discovery to evaluate the performance of SciTopic. Table 1 summarizes the specific information of these three datasets.

[0099] Table 1 Summary of Zero-shot Hashing Datasets

[0100]

[0101] 2. Competitive Methods

[0102] (1) Probabilistic Topic Discovery Methods:

[0103] LDA, NMF, ProdLDA, DecTM

[0104] (2) Neural Network Topic Discovery Methods:

[0105] ETM, NSTM, TSCTM, BERTopic, ECRTM, FASTopic

[0106] 3. Experimental Environment and Parameter Configuration

[0107] All competitive methods are used with their default hyperparameter configurations. We adopt the BGE-M3 model to generate document representation matrices and fine-tune on domain-specific data to optimize its semantic representation. The clustering process adopts the K-Means algorithm and is optimized in combination with the Llama3.170B model. The alpha in formula 3 is set to 1, and the lambda in formula 4 is set to 0.5, and the test and evaluation are carried out on two NVIDIA A100-80GB GPUs.

[0108] 4. Evaluation index

[0109] This part describes an evaluation index system for evaluating the performance of the method. The evaluation index includes topic discovery evaluation index and clustering evaluation index.

[0110] (1) Topic discovery evaluation index

[0111] The topic discovery evaluation index is used to measure the quality of the method in topic extraction, including the following indexes:

[0112] 1) Topic Coherence (TC)

[0113] Topic coherence is used to measure the semantic similarity between the most important words within each topic. This method adopts the (C V ) index, which is calculated based on the normalized pointwise mutual information (NPMI), to measure the co-occurrence of topic word pairs in the reference corpus. Its calculation formula is as follows:

[0114]

[0115] where W = {w1, w2, …, w T} represents the top T high-frequency words of the topic, P(w i , w j ) represents the co-occurrence probability of the word pair (w i , w j ) in the reference corpus, and NPMI(w i , w j ) is the normalized pointwise mutual information score.

[0116] 2) Topic Diversity (TD)

[0117] Topic diversity is used to measure the uniqueness and coverage of topic words, which is defined as follows:

[0118]

[0119] where k is the number of top k topic words selected for each topic. A higher TD value indicates a lower repetition rate of extracted topic words and a stronger topic difference.

[0120] (2) Clustering evaluation indicators

[0121] The clustering evaluation indicators are used to measure the performance of the method in the clustering task, including the following indicators:

[0122] 1) Calinski-Harabasz index (CHI)

[0123] The Calinski-Harabasz index is used to measure the ratio of between-cluster dispersion to within-cluster dispersion of the clustering results, and its calculation formula is as follows:

[0124]

[0125] where B and W represent the between-cluster dispersion matrix and the within-cluster dispersion matrix, respectively, n is the total number of samples, and k is the number of clusters. A higher CHI value indicates better clustering quality. k k

[0126] 2) Silhouette Coefficient (SC)

[0127] The Silhouette Coefficient is used to measure the compactness and separation of the clusters, and is calculated for each data point i as follows:

[0128]

[0129] where a(i) is the average distance of data point i to other points within the same cluster (within-cluster distance), and b(i) is the average distance of data point i to the nearest other cluster (between-cluster distance). Finally, SC calculates the average value of all samples, with a value range of [-1, 1], and the closer the value is to 1, the better the clustering effect.

[0130] 3) Davies-Bouldin index (DBI)

[0131] The Davies-Bouldin index measures the ratio of the average distance within the cluster to the distance between the clusters, and is defined as follows:

[0132]

[0133] where N is the number of clusters, σ i and σ j represent the average distance within cluster i and cluster j, respectively, and d ij represents the center distance between cluster i and j. A lower DBI value indicates better clustering quality. ​​

[0134] In summary, the evaluation method verifies the effectiveness of the method by topic discovery evaluation indicators and clustering evaluation indicators to ensure the applicability and stability of the method in topic extraction and clustering tasks.

[0135] 5. Experimental result analysis

[0136] (1) Here, the model performance of SciTopic is compared with 10 competitive methods (LDA, NMF, ProdLDA, DecTM, ETM, NSTM, TSCTM, BERTopic, ECRTM and FASTopic), and experiments are performed on three data sets respectively, and the experimental results are as follows:

[0137] Table 2 Topic quality results on different data sets (number of topics K = 100)

[0138]

[0139] The above experimental results, Table 2 shows the results of SPRING and all competitive baselines on data sets AWA2, CUB and SUN. In terms of quantitative analysis, experiments are performed on three data sets NeurIPS, DBLPV10 and AI-DM, and compared with ten baseline methods (such as LDA, NMF traditional method, and BERTopic, FASTopic neural topic model) under the condition of K = 100. As shown in Table 2, the method is significantly better than the comparison method in terms of topic consistency (Topic Coherence, TC) and topic diversity (Topic Diversity, TD) indicators. In particular, compared with advanced models such as ECRTM and TSCTM, the method achieves a better balance between topic consistency and diversity, avoiding the instability of some methods in a certain indicator. Overall, the method improves the TC by 21.8%, the TD by 14.6%, and the Calinski-Harabasz index (CHI) by 5.61% compared with the suboptimal method, fully verifying the advantages of the method in modeling complex topic structures.

[0140] In terms of scalability analysis, to further evaluate the topic discovery ability of the method, the clustering quality of different number of topics (K = 10, 20, 30, 40, 50) is tested on the AI-DM data set. As shown in Table 3, as K increases, the method can still maintain good topic separation and clustering tightness. For example, when K = 50, the TC of the method reaches 0.709, and the TD reaches 0.996, which maintains a significant advantage compared with the baseline method, proving its stability and generalization ability in a multi-topic environment.

[0141] Table 3 Topic quality results on AI-DM dataset under different number of topics (K)

[0142]

[0143] (2) To verify the semantic performance of SciTopic model, by comparing the performance of LDA, BERTopic, FASTopic and the method in topic clarity, discriminability and keyword diversity, the effectiveness of the method is further verified. The experimental results show that the method can accurately extract semantic focused topics, such as reinforcement, policy, agent and graph, node, network, etc. Compared with other methods, the method realizes better topic separation effect, greatly reduces the overlap between topics, and can capture a wide range of field-specific terms while maintaining high consistency. In addition, BERTopic performs better in professional fields such as quantum computing, but still has keyword overlap in some cases; FASTopic can cover multiple concepts, but contains some low-relevance words, affecting the focus of the topic; LDA has limited generalization ability and is prone to keyword overlap and less specific topics.

[0144] Table 4 Performance results of SciTopic variants by component ablation

[0145]

[0146] (3) To better understand SciTopic, the present application uses it and its several variants to do component analysis on the AI-DM dataset. These variants are: a) SciTopic w / o Title (T): remove title information; b) SciTopic w / o Abstract (A): remove abstract information; c) SciTopic w / o Metadata (M): remove metadata information; d) SciTopic w / o TM: remove title and metadata at the same time; e) SciTopic w / o AM: remove abstract and metadata at the same time. As shown in Table 4, all components have an important impact on the overall model performance, and removing any component will cause a significant performance drop, indicating the key role of each component in improving model efficiency (Efficiency) and robustness (Robustness).

[0147] (4) The present application also studies the influence of two important hyperparameters, α and λ. Experiments are conducted on the AI-DM dataset. The trend of changes in Topic Coherence (TC) and Topic Diversity (TD) indicators is analyzed. As shown in Figure 2, the performance of SciTopic is stable and robust under different α and λ values.Figure 4 As shown in the table, the parameter a is adjusted from 1.0 to 0.01, and the parameter λ is adjusted in the same range. The experimental results show that the change range of TC and TD under different parameter configurations is small, indicating that the model has low sensitivity to parameter adjustment. When a = 0.1 and λ = 0.5, the model reaches the highest TC value of 0.7313. Although the parameters change, the model can still maintain high stability, indicating that the method is robust in a wide range of parameters and can ensure the reliability of the model in different operating environments.

[0148] In summary, the present application provides an advanced topic modeling framework SciTopic that enhances the identification of topic structures in scientific literature using large language models (LLM). The core technologies of the framework include an entropy-based sampling method and a prompt-based triple task to optimize the document representation matrix and improve the topic clustering effect. Unlike traditional methods, the method does not rely on dimensionality reduction techniques, thereby improving the accuracy and robustness of topic identification. Experimental results show that SciTopic outperforms existing baseline models in terms of topic consistency (TC) and topic diversity (TD) indicators, and significantly surpasses well-known models such as LDA, NMF, and BERTopic on multiple datasets. In addition, the introduction of the triple task in the representation matrix optimization process helps to deeply mine the relationships between topics, and the class-based TF-IDF method further enhances the topic representation capability. The present application is verified by top conference datasets in the fields of artificial intelligence and data mining, and the experimental results show that the method has excellent performance in handling complex topic evolution. Looking to the future, SciTopic has broad application potential and can be applied to multiple research fields, providing an extensible solution for managing the growing scientific literature. Future research can further integrate more heterogeneous datasets and improve the interpretability of clustering results to support more comprehensive trend analysis and more efficient knowledge discovery.

[0149] Although specific embodiments of the present application are disclosed for the purpose of illustrating the present application and enabling its practice by persons skilled in the art, it is understood that various substitutions, changes and modifications can be made by those skilled in the art without departing from the spirit and scope of the present application and the appended claims. Therefore, the present application should not be limited to the disclosed best mode, and the scope of the present application is defined by the scope of the claims.

Claims

1. A method for discovering scientific literature topics based on LLM, comprising the following steps: 1) Obtain the text representation of each scientific document sample and encode it using a text encoder to obtain a document representation matrix corresponding to the scientific document sample; 2) Clustering the scientific literature samples through unsupervised clustering to obtain clustering results of different topics; calculating the entropy value of each scientific literature sample, and considering the scientific literature samples with entropy values ​​greater than the set entropy threshold as high uncertainty samples; 3) For each high uncertainty sample, the topic clustering method guided by LLM is used to calculate the semantic similarity between the high uncertainty sample and other scientific literature samples, and multiple triple tasks (a, c + ,c - ), where a is the high uncertainty sample, c + is a scientific literature sample similar to the high uncertainty sample, c - is a scientific literature sample that is not related to the high uncertainty sample; then, each triple task (a, c + ,c - ) fine-tuning the text encoder by contrastive learning; 4) using the fine-tuned text encoder to encode the text representation of each scientific document sample to obtain a document representation matrix corresponding to the scientific document sample; 5) Using the document representation matrix of each scientific literature sample obtained in step 4), the scientific literature samples are subject-clustered to generate cluster labels and subject division results for each scientific literature sample.

2. The method according to claim 1, characterized in that The text representation includes the paper title, abstract and metadata.

3. The method according to claim 2, characterized in that Use the text encoder to encode the paper title, abstract and metadata in the text representation respectively to generate the corresponding title representation vector h b , summary representation vector h b and metadata representation vector h m And splice to get the document representation matrix h p .

4. The method according to claim 1, 2 or 3, characterized in that: Construct triplet task (a,c + ,c - ) method is as follows: first, a scientific literature sample is selected from the cluster result of the high uncertainty sample and the cluster result of the neighboring high uncertainty sample, a semantic prompt word is constructed according to the high uncertainty sample and the two selected scientific literature samples, the semantic prompt word and the two selected scientific literature samples of the high uncertainty sample are input into the LLM, and the correlation between the high uncertainty sample and the two selected scientific literature samples is detected. If one of the selected scientific literature samples is similar to the high uncertainty sample and the other selected scientific literature sample is not related to the high uncertainty, a triple task (a, c + ,c - ), where a is the high uncertainty sample, c + is a scientific literature sample similar to the high uncertainty sample, c - is a scientific literature sample that is not related to the high uncertainty sample.

5. The method according to claim 4, characterized in that The method for determining the clustering results that are closest to the clustering result where the high uncertainty sample is located is as follows: calculating the average value of the document representation matrix of each scientific literature sample in the clustering result as the average embedding vector of the corresponding clustering result; The distance between the document representation matrix of the high-uncertainty sample and the average embedding vector of each clustering result is calculated, and the clustering result corresponding to the minimum distance is taken as the clustering result that is closest to the clustering result where the high-uncertainty sample is located.

6. The method according to claim 1, 2 or 3, characterized in that: The loss function used when fine-tuning the text encoder through the contrastive learning method is the cross entropy loss function s(x,y) represents the similarity score between the document representation matrix of scientific literature sample x and the document representation matrix of scientific literature sample y, where x is the high uncertainty sample a and y is or c j , is a scientific literature sample similar to the high uncertainty sample a, c j is a scientific literature sample that is unrelated to the high uncertainty sample a; represents the sample set consisting of all scientific literature samples similar to the high-uncertainty sample a and scientific literature samples unrelated to the high-uncertainty sample a in the current batch; τ is the softmax temperature parameter used to control the sharpness of the probability distribution.

7. The method according to claim 6, characterized in that Where f(x) is the document representation matrix of scientific literature sample x, and f(y) is the document representation matrix of scientific literature sample y. γ is a boundary parameter used to ensure the minimum semantic distinction between positive and negative samples. Positive samples are scientific literature samples similar to high-uncertainty samples, and negative samples are scientific literature samples unrelated to high-uncertainty samples.

8. A scientific literature topic discovery device based on LLM, characterized in that: It includes text encoder, entropy sampling module, clustering optimization module, topic optimization module, and topic semanticization module; The text encoder is used to encode the text representation of the scientific document sample to obtain a document representation matrix corresponding to the scientific document sample; The entropy sampling module is used to cluster the scientific literature samples by unsupervised clustering to obtain clustering results of different topics; then calculate the entropy value of each scientific literature sample, and regard the scientific literature samples with entropy values ​​greater than the set entropy value threshold as high uncertainty samples; The clustering optimization module is used to calculate the semantic similarity between each high uncertainty sample and other scientific literature samples using the LLM-guided topic clustering method, and construct multiple triple tasks (a, c + ,c - ), where a is the high uncertainty sample, c + is a sample of scientific literature similar to the high uncertainty sample, c - A sample of scientific literature that is not related to the high-uncertainty sample; The topic optimization module is used to utilize each triple task (a,c + ,c - ) fine-tuning the text encoder by contrastive learning; The topic semanticization module is used to enable the fine-tuned text encoder to encode the text representation of each scientific document sample to obtain a document representation matrix corresponding to the scientific document sample; then use the document representation matrix of each scientific document sample to perform topic clustering on each scientific document sample to generate cluster labels and topic division results for each scientific document sample.

9. A server, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images