Method for generating controllable abstract of scientific and technological literature based on large language model and term clustering

Abstract

The present application belongs to the field of natural language processing and text summary operation and maintenance technology, and is a controllable abstract generation method for scientific and technical literature based on large language model and term clustering, comprising the following steps: S1, term extraction and representation: extracting field-specific terms from the original abstract and citation text of the target scientific and technical literature to form a term set; S2, using a pre-trained language model to obtain a high-dimensional semantic vector for each term, and mapping it to a low-dimensional semantic space through a UMAP dimension reduction algorithm; S3, adaptive term clustering: using a density-based hierarchical clustering algorithm to perform unsupervised adaptive clustering on the dimension-reduced term vectors, automatically capturing the potential research aspects in the literature composed of key terms, and forming multiple semantically independent term clusters. The controllable abstract generation method for scientific and technical literature based on large language model and term clustering solves the problems of key point deviation and insufficient controllability that exist when large language models process scientific and technical literature.

Description

Technical Field

[0001] This invention relates to the field of natural language processing and text summarization technology, specifically to an automatic summarization method for scientific and technological documents that combines adaptive aspect clustering with the conditional generation capabilities of large language models. Background Technology

[0002] As scientific and technological literature becomes increasingly detailed and complex, its length and information volume continue to grow, making it increasingly difficult for readers to quickly grasp its novelty and core methods. Automatic summarization technology for scientific and technological literature is an effective way to address this "information overload" challenge. It can overcome reading and comprehension obstacles caused by the high complexity of the literature. Scientific and technological literature summaries aim to help readers quickly grasp the methods and contributions of the literature. Given limited time, readers need to use such summaries to quickly capture the core ideas, methodologies, and main contributions of an article, thereby efficiently filtering literature and acquiring knowledge.

[0003] Compared to general texts, scientific and technological literature has distinct domain-specific characteristics, manifested in the following ways: First, the text contains a large number of domain-specific terms, which relate to the research methods, mathematical tools, and key concepts in the literature and play a decisive role in guiding the content of the abstract. However, most existing models treat them as ordinary lexical units and fail to fully explore their semantic value. Second, the evaluation of a scientific and technological article is highly dependent on the public understanding of its academic field. If the abstract is based solely on the content of the original text, its reference value may be overlooked.

[0004] Existing document summarization methods are mainly divided into two categories: one is extractive summarization, which directly extracts representative sentences from the original text; the other is generative summarization, which uses input text to understand sentences and generate new content. Although these methods have achieved significant results in general text summarization tasks, they still have obvious shortcomings in the field of scientific and technological document summarization.

[0005] Key bias and information omission: When large language models process long texts, there is a "beginning and end bias", which easily leads to the omission of core methods and experimental results in the middle of the literature; Insufficient controllability and illusion problem: The model is prone to generating false content that does not match the original text, and it cannot accurately control the specific research aspects covered by the abstract; Strong domain dependence: Existing professional summarization models need to be fine-tuned on large-scale domain datasets, have poor generalization ability, and are difficult to quickly adapt to emerging disciplines.

[0006] Therefore, an automatic summarization method for scientific and technological documents that combines adaptive aspect clustering with the conditional generation capabilities of a large language model is proposed to address the above problems. Summary of the Invention

[0007] The purpose of this invention is to provide a controllable abstract generation method for scientific and technological documents based on large language models and term clustering, so as to solve the problems of focus deviation, insufficient use of terminology, poor controllability and illusion in the generation of scientific and technological documents in the prior art mentioned in the background.

[0008] To achieve the above objectives, this invention provides the following technical solution: a method for generating controllable summaries of scientific and technological documents based on large language models and terminology clustering, comprising the following steps: S1. Terminology Extraction and Representation: Extracting domain-specific terms from the original abstract and cited text of the target scientific and technological literature to form a terminology set; S2. Use a pre-trained language model to obtain the high-dimensional semantic vector of each term, and then use the UMAP dimensionality reduction algorithm to map it to a low-dimensional semantic space. S3. Adaptive term clustering: The density-based hierarchical clustering algorithm is used to perform unsupervised adaptive clustering on the dimensionality-reduced term vectors, automatically capturing potential research aspects composed of key terms in the literature and forming multiple semantically independent term clusters. S4. Construction of structured cue words: Based on semantic similarity, representative terms are sampled from each term cluster and transformed into structured cue words containing research constraints; S5. Large Language Model Conditional Generation: The structured prompt words, the original text summary, and the cited context fragments are input into the large language model. The prompt words constrain the model to focus on key semantic fragments and generate the target summary.

[0009] Preferably, the specific steps for term extraction in S1 include: S1.1. By calling the interface through the large language model, the three-part prompt word template of "system prompt word + instance prompt word + main prompt word" is used to identify scientific and technical terms in the original abstract and cited text. S1.2. Use natural language processing tools to perform named entity recognition on the same text and filter out noisy entities such as date, ordinal number, percentage, and quantity. S1.3 Combine the large language model prompt word extraction and named entity recognition tool filtering, take the union of the two results, and remove duplicates; Named entity recognition filters remove noisy entities that do not refer to the research object, mathematical tool, or technical terminology.

[0010] Preferably, the pre-trained language model in S2 is RoBERTa-Large. The RoBERTa-Large pre-trained model is used to encode each term, and the vector corresponding to the [CLS] label is taken as the primary semantic representation of the term with a dimension of 1024. The hyperparameters n_neighbors and n_components of UMAP dimensionality reduction are set to 10.

[0011] Preferably, the primary semantic representation ,in d is the vector dimension.

[0012] Preferably, the adaptive term clustering in S3 uses the HDBSCAN algorithm, and the hyperparameters are set as follows: min_cluster_size=2; min_samples=2; cluster_selection_method="leaf"; The number of clusters is adaptively determined by the algorithm based on the data density distribution.

[0013] Preferably, the structured prompt word construction in step S4 specifically includes: S3.1 Calculate the centroid vector of each term cluster, which is the mean of all term vectors within the cluster; S3.2 Calculate the cosine similarity between each term vector and the centroid vector within the cluster, and sort them in descending order of similarity; S3.3, Set the sampling ratio ρ, and select the previous... A set of terms is used as the representative term set for this cluster, among which For the first The number of elements in each term cluster, and the sampling ratio ρ for the overall summary are set to 0.1-0.7; S3.4. Sample all term clusters and incorporate the union of the representative term sets of each cluster into the structured prompt words.

[0014] Preferably, the formula for calculating the mean of the term vector is:

[0015] in for UMAP vector representation; The sampling formula is:

[0016] in The cosine similarity between vectors is... Sampling based on semantic similarity ensures the conciseness of prompts while guaranteeing that the terms input to the LLM are the most typical and core elements of its respective area.

[0017] Preferably, the large language model conditional generation described in S5 supports two modes: Holistic Abstract Mode: The prompt words contain representative terms from all term clusters, and the constraint model generates a comprehensive abstract that fully covers the core contributions of the literature; Aspect-Specific Summary Mode: The prompt words contain only representative terms from the target term cluster and explicitly instruct the model to ignore other aspects, generating a summary that focuses on a specific research topic.

[0018] Preferably, the generation parameters of the large language model in S4 are set as follows: Temperature value: Temperature = 0.2, repetition penalty: repetition_penalty = 1.3, presence penalty: presence_penalty = 0.8, frequency penalty: frequency_penalty = 0.8, maximum generation length: max_tokens = 160.

[0019] A controllable abstract generation system for scientific and technological documents based on large language models and terminology clustering includes: The terminology extraction module is used to extract domain-specific terms from the original abstracts and citation texts of target scientific and technological documents. The adaptive clustering module is used to perform unsupervised clustering of the dimensionality-reduced term vectors using the HDBSCAN algorithm to generate term clusters that represent the potential aspects of the documents. The prompt word construction module is used to sample representative terms from a term cluster and generate structured prompt words. Conditional generation module: used to input prompt words and original text information into a large language model to generate a controllable target summary; The output control module is used to set the generation parameters, including temperature parameters, repetition penalty, frequency penalty, and maximum output length.

[0020] Compared with the prior art, the beneficial effects of the present invention are: the method for generating controllable summaries of scientific and technological documents based on large language models and term clustering; 1. It solves the common problems of focus deviation and insufficient controllability in existing large language models when processing scientific and technological literature. By integrating adaptive aspect clustering technology, it can automatically capture potential research aspects composed of key terms from the literature, and elevate scientific and technological terms from ordinary word units to multi-dimensional features that reflect the core research focus of the literature. It fully explores the deep semantic value of professional terms, guides the large model to accurately anchor the main line of literature research from the input representation level, and avoids the abstract content from deviating from the core of the literature from the root. 2. It improves the factual consistency and controllable flexibility of scientific literature abstract generation. The structured prompt word construction strategy based on semantic similarity sampling selects representative terms from each term cluster to form knowledge anchors. Combined with the original text abstract and cited context fragments, it constrains the large model generation process, effectively reducing the "illusion" phenomenon of large models and the problem of missing key information. At the same time, it overcomes the inherent "first and last bias" of large models. This strategy also supports two controllable generation modes: "overall abstract" and "specific aspect abstract", which can flexibly meet the differentiated abstract needs under different application scenarios. 3. This method significantly reduces the application cost and threshold for generating high-quality scientific and technological literature abstracts, while achieving excellent generation performance. It does not require any domain-annotated data for fine-tuning. Through zero-shot inference and term-level prompting engineering, it can achieve results comparable to or even surpass those of professional domain-specific fine-tuned models such as SciBERTSum and SKT5SciSumm on public benchmark datasets such as ScisummNet and CL-SciSumm. It also performs outstandingly in automatic evaluation metrics such as ROUGE and BERTScore as well as in human evaluation, significantly improving the information coverage and factual consistency of scientific and technological literature abstracts, and possessing stronger generalization ability and engineering practical value. Attached Figure Description

[0021] Figure 1 This is an example diagram of a context fragment referenced in this invention; Figure 2 This is a schematic diagram of the prompt word template for the present invention; Figure 3 This is a schematic diagram of a UMAP example of the present invention; Figure 4 This is a schematic diagram of the overall summary keywords of the present invention; Figure 5 This is a schematic diagram of a summary of a specific aspect of the present invention; Figure 6 This is a schematic diagram of the TOLLM working framework of the present invention; Figure 7 For the purposes of this invention The following is a schematic diagram of ROUGE2 fractions; Figure 8 For the purposes of this invention The following is a diagram illustrating the BERTScore scores; Figure 9 This is a reference chart showing the average profile coefficients under different combinations of hyperparameters in this invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Example 1: This invention provides a technical solution: a method for generating controllable summaries of scientific and technological documents based on large language models and terminology clustering, comprising the following steps: S1. Terminology Extraction and Representation: Extracting domain-specific terms from the original abstract and cited text of the target scientific and technological literature to form a terminology set; S1.1. By calling the interface of the large language model (such as GPT-3.5), a three-part prompt word template of "system prompt word + instance prompt word + main prompt word" is used to identify and extract scientific and technical terms in the original abstract and cited text. This three-part design can effectively overcome the problem that simple prompt words are prone to omitting compound terms (such as "GraphAttentionNetwork"); S1.2. Use natural language processing tools to perform named entity recognition on the same text and filter out noisy entities such as date, ordinal number, percentage, and quantity. S1.3. Take the union of the extraction results of GPT-3.5 and the results filtered by spaCy and remove duplicates to obtain the final set of scientific and technological terms for the document. Named entity recognition filters remove noisy entities that do not refer to the research object, mathematical tool, or technical terminology.

[0024] S2. Term semantic vectorization and dimensionality reduction: High-dimensional semantic vectors of each term are obtained using a pre-trained language model, and then mapped to a low-dimensional semantic space using the UMAP dimensionality reduction algorithm. The pre-trained language model in S2 is RoBERTa-Large. Each term is encoded using the RoBERTa-Large pre-trained model, and the vector corresponding to the [CLS] tag is taken as the primary semantic representation of the term. ,in d is the vector dimension, for RoberTa-Large The value is 1024; the UMAP (Uniform Manifold Approximation and Projection) algorithm is used to reduce the high-dimensional vector to 10 dimensions to alleviate the "curse of dimensionality" problem in subsequent clustering algorithms in high-dimensional spaces. The hyperparameters of UMAP are preferably set as follows: n_neighbors=10, n_components=10.

[0025] The optimal low-dimensional UMAP vector for this study is determined by controlling its hyperparameters: “n_neighbors” and “n_components”. The following formula:

[0026] in for The low-dimensional vector representation, This represents the UMAP vector dimension.

[0027] S3. Adaptive term clustering: The density-based hierarchical clustering algorithm is used to perform unsupervised adaptive clustering on the dimensionality-reduced term vectors, automatically capturing potential research aspects composed of key terms in the literature and forming multiple semantically independent term clusters. Adaptive term clustering uses the HDBSCAN algorithm, with hyperparameters set as follows: min_cluster_size=2; min_samples=2; cluster_selection_method="leaf"; The number of clusters is adaptively determined by the algorithm based on the data density distribution.

[0028] S4. Construction of structured cue words: Based on semantic similarity, representative terms are sampled from each term cluster and transformed into structured cue words containing research constraints; The construction of structured prompt words specifically includes: S3.1 Calculate each term cluster centroid vector That is, the mean of all term vectors within the cluster;

[0029] in for UMAP vector representation; S3.2 Calculate the cosine similarity between each term vector and the centroid vector within the cluster, and sort them in descending order of similarity; S3.3, Set the sampling ratio ρ, and select the previous... A set of terms is used as the representative term set for this cluster, among which For the first The number of elements in each term cluster, and the sampling ratio ρ for the overall summary are set to 0.1-0.7;

[0030] in The cosine similarity between vectors is... Sampling based on semantic similarity ensures the conciseness of prompts while guaranteeing that the terms input to the LLM are the most typical and core elements of its respective aspect. S3.4. Sample all term clusters and incorporate the union of the representative term sets of each cluster into the structured prompt words.

[0031] S5. Large Language Model Conditional Generation: The structured prompt words, the original text summary, and the cited context fragments are input into the large language model. The prompt words constrain the model to focus on key semantic fragments and generate the target summary. Temperature value: Temperature = 0.2, repetition penalty: repetition_penalty = 1.3, presence penalty: presence_penalty = 0.8, frequency penalty: frequency_penalty = 0.8, maximum generation length: max_tokens = 160; The holistic summary generation mode is achieved by integrating k pre-built term clusters into the prompts input to the TOLLM model. Representative term set The TOLLM model can respond to constraints from multiple technical dimensions simultaneously during text generation, and dynamically switch attention focus between different technical dimensions corresponding to different term clusters by leveraging its built-in cross-attention mechanism.

[0032] For example, the TOLLM model can first output the complete content about the "data preprocessing" technical dimension, then naturally switch to the "core model architecture" technical dimension for detailed explanation, and finally focus on the "evaluation index" technical dimension for systematic summary.

[0033] The multi-cluster joint prompting method employed in this invention effectively solves the "beginning-end bias" defect commonly found in existing large language models during long text processing. This "beginning-end bias" refers to the tendency of large language models to overemphasize the beginning and end of the input context when processing long texts, easily neglecting key technical information contained in the middle paragraphs. With the multi-cluster joint prompting method of this invention, even if important technical information is located in the middle paragraphs of the document being processed, as long as the corresponding technical term is included in the corresponding term cluster in the prompt words, the TOLLM model's attention will be directed to the text region containing that term, thereby ensuring that all key technical information can be accurately extracted and incorporated into the generated overall summary.

[0034] Example 2: This example provides another mode for conditional generation of large language models; The implementation of aspect-specific summary generation is as follows: when the prompt words input to the TOLLM model contain only a single target term cluster. sampling set Furthermore, when the prompt contains an explicit instruction to require the model to ignore all non-target technical dimensions, the TOLLM model will automatically assign extremely low attention weights to the original text regions unrelated to the target keywords during the self-attention calculation process. Through the aforementioned targeted attention control mechanism, this invention enables the TOLLM model to accurately output customized summary content related to the user-specified technology.

[0035] Furthermore, all scientific and technological terms used to construct the aforementioned prompts in this invention are directly extracted from the original documents to be processed, and all terms have undergone rigorous aspect clustering and semantic similarity screening beforehand. These verified key terms constitute a rigid constraint on the factual content of the original documents, significantly biasing the vocabulary output distribution of the TOLLM model during text generation and forcing the model to generate content consistent with the facts of the original documents. Compared to general-purpose large language models that do not employ the aforementioned term-level constraint mechanism, the summaries generated by TOLLM in this invention show significant performance improvements in all factual consistency-related metrics.

[0036] Given a science and technology document D; The original abstract is a collection of... The combination of sentences ; Its cited text is a collection of The combination of sentences ; The referenced context fragment is obtained through a positioning algorithm. Include One sentence.

[0037] Terminology Collection .

[0038] Define an unsupervised adaptive clustering algorithm ,set up The set of aspects is ,but ,in Determined by the adaptive clustering algorithm and .

[0039] Define a sampling function based on semantic similarity For any term cluster Let its center of mass be Based on the elements within each cluster With the centroid vector Semantic similarity between them in proportion Extract the elements with the highest similarity to form the sampling set of this cluster. .

[0040] Let the large language model used by TOLLM be... Based on the different objectives of abstract generation, the following two mappings are defined: Overall summary: Define the prompts for the overall summarization task Then the mapping can be defined as:

[0041] Specific aspect summary: Definition for specific aspects Summary task prompts Then the mapping can be defined as: .

[0042] A controllable abstract generation system for scientific and technological documents based on large language models and terminology clustering includes: The terminology extraction module is used to extract domain-specific terms from the original abstracts and citation texts of target scientific and technological documents. The adaptive clustering module is used to perform unsupervised clustering of the dimensionality-reduced term vectors using the HDBSCAN algorithm to generate term clusters that represent the potential aspects of the documents. The prompt word construction module is used to sample representative terms from a term cluster and generate structured prompt words. Conditional generation module: used to input prompt words and original text information into a large language model to generate a controllable target summary; The output control module is used to set the generation parameters, including temperature parameters, repetition penalty, frequency penalty, and maximum output length.

[0043] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0044] Furthermore, various different embodiments of the present invention can be combined in any way, as long as they do not violate the spirit of the present invention, they should also be regarded as the content disclosed by the present invention.

[0045] In summary, the above description of the disclosed embodiments enables those skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating controllable abstracts of scientific and technological documents based on large language models and terminology clustering, characterized in that, Includes the following steps: S1. Terminology Extraction and Representation: Extracting domain-specific terms from the original abstract and cited text of the target scientific and technological literature to form a terminology set; S2. Use a pre-trained language model to obtain the high-dimensional semantic vector of each term, and then use the UMAP dimensionality reduction algorithm to map it to a low-dimensional semantic space. S3. Adaptive term clustering: The density-based hierarchical clustering algorithm is used to perform unsupervised adaptive clustering on the dimensionality-reduced term vectors, automatically capturing potential research aspects composed of key terms in the literature and forming multiple semantically independent term clusters. S4. Construction of structured cue words: Based on semantic similarity, representative terms are sampled from each term cluster and transformed into structured cue words containing research constraints; S5. Large Language Model Conditional Generation: The structured prompt words, the original text summary, and the cited context fragments are input into the large language model. The prompt words constrain the model to focus on key semantic fragments and generate the target summary.

2. The method for generating controllable summaries of scientific and technological documents based on large language models and terminology clustering according to claim 1, characterized in that, The specific steps for term extraction in S1 include: S1.

1. By calling the interface through the large language model, the three-part prompt word template of "system prompt word + instance prompt word + main prompt word" is used to identify scientific and technological terms in the original abstract and cited text. S1.

2. Use natural language processing tools to perform named entity recognition on the same text and filter out noisy entities such as date, ordinal number, percentage, and quantity. S1.3 Combine the large language model prompt word extraction and named entity recognition tool filtering, take the union of the two results, and remove duplicates; Named entity recognition filters remove noisy entities that do not refer to the research object, mathematical tool, or technical terminology.

3. The method for generating controllable abstracts of scientific and technological documents based on large language models and terminology clustering according to claim 1, characterized in that, The pre-trained language model in S2 is RoBERTa-Large. Each term is encoded using the RoBERTa-Large pre-trained model, and the vector corresponding to the [CLS] label is taken as the primary semantic representation of the term with a dimension of 1024. The hyperparameters n_neighbors and n_components of UMAP dimensionality reduction are set to 10.

4. The method for generating controllable abstracts of scientific and technological documents based on large language models and terminology clustering according to claim 3, characterized in that, The primary semantic representation ,in d is the vector dimension.

5. The method for generating controllable abstracts of scientific and technological documents based on large language models and terminology clustering according to claim 1, characterized in that, The adaptive term clustering in S3 uses the HDBSCAN algorithm, with hyperparameters set as follows: min_cluster_size=2; min_samples=2; cluster_selection_method="leaf"; The number of clusters is adaptively determined by the algorithm based on the data density distribution.

6. The method for generating controllable abstracts of scientific and technological documents based on large language models and terminology clustering according to claim 1, characterized in that, The structured prompt word construction described in S4 specifically includes: S3.1 Calculate the centroid vector of each term cluster, which is the mean of all term vectors within the cluster; S3.2 Calculate the cosine similarity between each term vector and the centroid vector within the cluster, and sort them in descending order of similarity; S3.3, Set the sampling ratio ρ, and select the previous... A set of terms is used as the representative term set for this cluster, among which For the first The number of elements in each term cluster, and the sampling ratio ρ for the overall summary are set to 0.1-0.7; S3.

4. Sample all term clusters and incorporate the union of the representative term sets of each cluster into the structured prompt words.

7. The method for generating controllable summaries of scientific and technological documents based on large language models and terminology clustering according to claim 1, characterized in that, The formula for calculating the term vector mean is: ； in for UMAP vector representation; The sampling formula is: ； in The cosine similarity between vectors is... Sampling based on semantic similarity ensures the conciseness of prompts while guaranteeing that the terms input to the LLM are the most typical and core elements of its respective area.

8. The method for generating controllable summaries of scientific and technological documents based on large language models and terminology clustering according to claim 1, characterized in that, S5's large language model conditional generation supports two modes: Holistic Abstract Mode: The prompt words contain representative terms from all term clusters, and the constraint model generates a comprehensive abstract that fully covers the core contributions of the literature; Aspect-Specific Summary Mode: The prompt words contain only representative terms from the target term cluster and explicitly instruct the model to ignore other aspects, generating a summary that focuses on a specific research topic.

9. The method for generating controllable abstracts of scientific and technological documents based on large language models and terminology clustering according to claim 8, characterized in that, The generation parameters of the large language model in S4 are set as follows: Temperature value: Temperature = 0.2, repetition penalty: repetition_penalty = 1.3, presence penalty: presence_penalty = 0.8, frequency penalty: frequency_penalty = 0.8, maximum generation length: max_tokens = 160.

10. A controllable abstract generation system for scientific and technological documents based on large language models and terminology clustering, characterized in that, include: The terminology extraction module is used to extract domain-specific terms from the original abstracts and citation texts of target scientific and technological documents. The adaptive clustering module is used to perform unsupervised clustering of the dimensionality-reduced term vectors using the HDBSCAN algorithm to generate term clusters that represent the potential aspects of the documents. The prompt word construction module is used to sample representative terms from a term cluster and generate structured prompt words. Conditional generation module: used to input prompt words and original text information into a large language model to generate a controllable target summary; The output control module is used to set the generation parameters, including temperature parameters, repetition penalty, frequency penalty, and maximum output length.

Images