A multi-document intelligent classification and automatic summary generation method based on a large language model

Abstract

The application discloses a kind of based on large language model's multiple literature intelligent classification and automatic abstract generation method, its characteristics are the method includes: searching document or uploading local document, extract text content;Key words are identified using algorithm or large language model;Key words are converted into vector form using vectorization model;The research direction of clustering result is summarized using large language model;Single document and the literature of different research directions are summarized using large language model;Through large language model and vectorization semantic matching technology, steps such as abstract polishing and citation insertion are executed.The application has significantly improved the efficiency and quality of document classification and abstract generation compared with prior art, better fusion text vectorization, subject clustering and large language model generation and other technologies, realize efficient, accurate multiple literature abstract automatic generation, suitable for interdisciplinary academic literature analysis and information integration application, show broad application potential.

Description

Technical Field

[0001] This invention relates to the fields of natural language processing and automatic review technology, specifically a method for intelligent classification of multiple documents and automatic review generation based on a large language model. Background Technology

[0002] In the field of scientific research, the importance of literature is self-evident, especially within the broad scope of literature processing and knowledge management, where literature classification and automated review technologies play a crucial role. Traditional literature classification methods, such as manual classification, are limited by both human resources and time costs, making it difficult to adapt to the continuous growth in the number of documents. Therefore, machine learning-based literature classification methods have emerged, providing innovative solutions for the efficient organization and management of literature.

[0003] In the field of document classification, researchers have conducted extensive explorations into different types of documents. Since the concept of automatic document classification was first proposed in the 1950s, the research history in this field can be roughly divided into three stages: feasibility studies of automatic classification, experimental studies, and the practical application stage. Significant progress has been made in the practical application stage over the past two decades. Regarding document classification methods, researchers have employed various techniques, including: classifiers based on Support Vector Machines (SVM), automated Dewey Decimal Classification combining machine learning and text mining, multi-level library classification models using classifiers such as Naive Bayes, Support Vector Machines, and Decision Trees (C4.5), and the Bidirectional Encoding Representation Model (BERT) based on deep learning. Meanwhile, automatic review technology, as a high-level form of automatic text summarization, has also received widespread attention. Early automatic review research used data processing machines to scan documents, calculating relative measures of saliency through statistical word frequency and distribution information, and then extracting and printing the sentences with the highest importance scores to form the summary text. With technological advancements, automatic review technology has gradually evolved from simple extractive summarization to complex generative summarization. Extractive summarization primarily uses methods such as cluster analysis, supervised SVM machine learning models, and neural networks to identify and extract core sentences from documents, while generative summarization utilizes deep learning models to generate entirely new text as a summary.

[0004] Despite advancements in automated review technology, numerous challenges remain regarding comprehensiveness and accuracy. The introduction of Large Language Models (LLMs) has brought a new breakthrough to automated review technology. LLMs, with their powerful language understanding capabilities, make automated reviews more efficient when handling complex texts. For example, through large language models, systems can understand and integrate implicit information and contextual relationships within documents, generating more coherent and accurate summaries. However, these models also have shortcomings in practical applications. For instance, the generated review content may lack systematicity and fail to comprehensively reflect the relationships between documents; when dealing with complex topics, the models may not accurately capture subtle differences between different studies; and current tools lack flexibility in reflecting the latest research developments when updating data. These problems limit the development and widespread application of automated review technology. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing a multi-document intelligent classification and automatic review generation method based on a large-scale language model. This method employs keyword extraction and vectorization for multi-document intelligent classification, and utilizes a large language model to generate automatic reviews, enabling rapid classification and comprehensive summarization of a large number of documents. The method is simple and efficient, significantly improving the efficiency and quality of automatic multi-document reviews, providing strong support for academic research and practical applications. It effectively solves the problems of low efficiency and insufficient accuracy in document classification and review generation, and is applicable to academic literature analysis and information integration tasks in various fields. It achieves efficient and accurate automatic generation of multi-document reviews, and has broad application scenarios and value.

[0006] The specific technical solution to achieve the purpose of this invention is: a method for intelligent classification and automatic review generation of multiple documents based on a large language model, characterized by the following steps:

[0007] 1) Obtain a collection of literature on the target research topic from the literature database, and perform preliminary filtering and text preprocessing on the literature;

[0008] 2) Extract keywords from the literature;

[0009] 3) Use a vectorization model to represent keywords using semantic vectors;

[0010] 4) Based on the vectorization results, keywords are classified by using a variety of clustering algorithms, and corresponding research perspectives are generated using a large language model according to the semantic features of each keyword category.

[0011] 5) Utilize large language models to generate hierarchical and categorized review texts for individual articles and groups of articles from different research perspectives; finally, by integrating the summary texts from various research perspectives, an optimized and complete review text is formed to achieve a comprehensive and structured summary result.

[0012] 6) Utilize large-scale language models to implement comprehensive expansion, polishing, and logical optimization. By leveraging the analytical and judgment capabilities of large-scale models and combining them with the semantic matching function of the Embedding model, relevant literature can be efficiently located and citations can be inserted in appropriate positions.

[0013] Compared with the prior art, the present invention has the following beneficial technical advancements and significant technical effects:

[0014] 1) This invention supports dynamic adaptation and expansion across different disciplinary topics, emphasizing "dynamic adaptation" and "multi-level semantic analysis," enabling flexible access to knowledge resources in multi-domain research. Simultaneously, by introducing multi-level semantic analysis technology, it provides in-depth analysis and broad coverage in literature topic classification and multi-angle review generation, thereby meeting users' needs for analysis at different levels.

[0015] 2) This invention integrates modules such as literature screening, keyword extraction, clustering and classification, and review generation, which have significant advantages in performance optimization and module scalability. It not only has the ability to call up functions efficiently, but also can dynamically expand functions through modular design to adapt to data processing tasks of different scales. It can process a large number of literature resources in a short time, greatly shorten the time cost of traditional methods, and avoid errors caused by manual operation.

[0016] 3) In terms of document classification and data association, this invention supports intelligent classification and topic-focused analysis of massive amounts of documents by combining keyword semantic vectorization with clustering algorithms. Combined with the natural language generation capabilities of a large language model, it can generate structured and logically sound review texts, providing researchers with a reliable research foundation and analytical tools.

[0017] 4) This invention addresses the illusion problem that may occur when large language models generate reviews. Through a multi-round generation and aggregation strategy, it effectively reduces the uncertainty of the generated results. Multiple tests and consistency screenings significantly improve the accuracy and stability of the generated reviews, ensuring the credibility and usability of the final output. Finally, through standardized templates and intelligent optimization techniques, it not only guarantees the richness and logical rigor of the generated reviews but also meets the normative requirements of academic writing, providing an efficient and reliable solution for academic research and information integration.

[0018] 5) For the entire research field, this invention employs cluster analysis to divide all relevant literature into different research perspectives, thereby revealing the research focus and trends of each subfield. Through preliminary summarization and in-depth analysis of these research perspectives, the final review report can be ensured to be more comprehensive and accurate in content. Furthermore, these summary texts from different perspectives not only serve as part of the review report, providing users with richer reference information, but also help researchers quickly grasp the core viewpoints and research progress of each subfield.

[0019] 6) In terms of citation annotation technology, this invention adopts a large-scale language model to determine whether citation annotation is necessary, and uses cosine similarity calculation to vectorize the document text, thereby achieving more precise location of the required cited documents. Furthermore, given that the article already possesses a complete logical structure and core viewpoints, inappropriate citation insertion is avoided. Attached Figure Description

[0020] Figure 1 This is a flowchart of the present invention;

[0021] Figure 2 This is a flowchart of the document preprocessing process;

[0022] Figure 3 A flowchart for classifying literature research directions;

[0023] Figure 4 A flowchart summarizing all literature and research directions;

[0024] Figure 5 A flowchart for the logical optimization and citation insertion of the final review; Detailed Implementation

[0025] See Figure 1 The intelligent classification and automatic review generation of multiple documents based on a large language model specifically includes the following steps:

[0026] 1) Literature acquisition, screening, and text preprocessing (i.e., step 1)

[0027] See Figure 2 Literature related to the review topic is retrieved from literature databases or uploaded by users, forming a topic-related literature cluster. Subsequently, the literature undergoes initial screening by examining the abstracts of the web pages linked to the search results to determine their relevance to the review topic. For relevant literature, the content of the web pages is extracted and saved; unrelated literature is considered irrelevant and not saved. Finally, the text content of the literature is parsed to extract the title, abstract, and main text, and then organized into a structured dataset for subsequent analysis and use.

[0028] 2) Keyword extraction and vectorization (i.e., steps 2 and 3)

[0029] See Figure 3 Based on user needs, explicit keywords in documents can be extracted using regularization rules. For documents without keyword annotations or those required by the user, keyword extraction or generation can be performed using TF-IDF, TextRank algorithms, or large language models. All keywords are semantically represented using embedding models such as SimBERT, BERT, or text2vec-large. Through keyword extraction and vectorization, document topics can be efficiently mapped to the semantic space, enabling accurate analysis of the core content of the document. During the vectorization process, different optimization strategies can be selected according to the application scenario; for example, for multilingual documents, multilingual versions of the vectorization model can be used.

[0030] 3) Cluster analysis and research perspective construction (i.e., step 4)

[0031] See Figure 3 This invention calculates the cosine distance of semantic similarity between keywords and uses clustering algorithms such as hierarchical clustering, K-means, or DBSCAN to classify keywords and form research perspectives. Each document is matched with its corresponding keywords and perspectives to construct document sets under each research perspective. In practical applications, hierarchical clustering can intuitively present the hierarchical relationship between documents and is suitable for small-scale datasets; the K-means algorithm is more suitable for large-scale document sets; DBSCAN performs better in identifying sparse or heterogeneous data. This invention selects the optimal clustering algorithm based on the specific characteristics of the document data. In addition, this invention adopts a dynamic clustering threshold adjustment strategy in cluster analysis to adapt to document sets with different data distributions. This flexible adjustment mechanism makes the clustering results more accurate and helps in the construction of subsequent research perspectives.

[0032] 4) Automatic review generation (i.e., step 5)

[0033] See Figure 4 This invention automatically generates single-article summaries, comprehensive reviews, and research perspective reviews. The single-article summaries utilize a large language model to initially summarize the content of each article and extract core information. The research perspective reviews categorize and summarize the literature collections for each research perspective, identifying research hotspots and trends. The comprehensive reviews integrate the content of each perspective review and combine it with the large language model to generate the final structured review text. In the comprehensive review generation stage, this invention incorporates a multi-round optimization strategy to iteratively improve the initially generated review text. Through quality evaluation of the generated results, the model input prompts are further adjusted to ensure the logical consistency and coherence of the review text.

[0034] 5) Citation recommendation and optimization (i.e., step 6)

[0035] See Figure 5 This approach uses an embedding model to vectorize documents, combines it with RAG (Enhanced Retrieval) methods to match the most relevant documents, and sets matching thresholds to ensure the relevance and accuracy of citations. Finally, a large language model completes citation integration, logical optimization, and content polishing. In practical applications, the choice of embedding model can be adjusted according to the document type; for example, a domain-optimized model can be used for domain-specific documents. Furthermore, a manual citation modification function is provided through a user interface to meet personalized needs.

[0036] The present invention will now be described in detail with reference to the embodiments and accompanying drawings. The large language model used refers to a natural language processing model that can complete basic dialogue. No specific requirements are made regarding the size of the parameters, the language to be adapted, the deployment platform, or the implementation scheme. Example

[0037] See Figure 1 This embodiment provides a review generation method based on a large language model, which includes six steps: literature content extraction, keyword extraction, keyword classification and direction summary, literature hierarchical summary, comprehensive literature review summary, review polishing and citation insertion. The large language model involved is the Qwen2.5:7b model.

[0038] See Figure 2 The literature is preprocessed (i.e., step 1).

[0039] After determining the review topic, such as "digital transformation in education," a search was conducted using the CNKI (China National Knowledge Infrastructure) full-text database, with the keyword "digital transformation in education." To ensure the relevance of the research, high-quality research papers belonging to disciplines such as "educational theory and management," "higher education," "vocational education," "secondary education," "adult education and special education," and "primary education," and indexed by CSSCI, were selected. Based on the relevant definitions of digital transformation in education, the acquired literature data was denoised to obtain valid data. After saving all relevant literature, multiple libraries in Python (PyPDF2, pdfminer, etc.) can be used to parse the text content in PDF documents. Additionally, the API interfaces of relevant databases can be used to obtain the required literature, read the complete content of the documents, and perform structured processing through programming to compile a structured list of literature related to the review topic.

[0040] See Figure 3 Keyword extraction and vectorization (i.e., steps 2 and 3)

[0041] Keywords in literature typically begin with "keywords:" or "[keywords]", and are usually separated by spaces, commas, or semicolons. Keywords are extracted from the literature through regular expression matching, or by using algorithms such as TF-IDF, TextRank, and large-scale language models to summarize the entire text, forming a complete keyword set. Simultaneously, the correspondence between the literature and the keywords is established, and the extracted keywords are processed using models such as SIMBERT to obtain a vector representation for each keyword.

[0042] See Figure 3 The classification of literature research directions is achieved through clustering algorithms (i.e., steps 2 and 3).

[0043] Based on the characteristics of the literature data and its classification requirements, an appropriate clustering algorithm is selected. Common clustering algorithms include K-means, hierarchical clustering, DBSCAN, and spectral clustering. After selecting the clustering algorithm, the corresponding parameters are configured. The vectorized keyword data is input into the selected clustering algorithm, and the clustering operation is performed. After clustering is completed, a large language model is used to extract research directions, and the correlation between research directions, keywords, and literature is established by combining the correspondence table between literature and keywords.

[0044] See Figure 4 The literature review process (i.e., step 5)

[0045] Each document is summarized and refined using a large language model (such as Qwen2.5 7b or related APIs). Then, the summary texts of documents corresponding to different research topics are categorized and summarized according to those topics. Based on the review texts generated for each topic, a comprehensive analysis of the overall research objectives is conducted, resulting in a final review of the entire collection of documents.

[0046] See Figure 5 The final steps include expanding, polishing, logically optimizing, and inserting citations for the review (i.e., step 6).

[0047] After logical optimization and polishing of the review text, citations are inserted using a large language model. First, the literature collection is vectorized using embedding. Then, the large language model is applied to each sentence of the review text to determine if citations are needed for argumentation. If argumentation is necessary, the sentence is vectorized and matched semantically with the literature collection. A threshold is set to determine which literature can be inserted, and the format is adjusted accordingly. This process continues until all literature has been properly inserted.

[0048] The above is merely a further explanation of the present invention and is not intended to limit the scope of the patent. Any equivalent implementations of the present invention should be included within the scope of the claims of the patent.

Claims

1. A method for intelligent classification and automatic review generation of multiple documents based on a large language model, characterized in that, The method specifically includes the following steps: 1) Obtain a collection of literature on the target research topic from the literature database, and perform preliminary filtering and text preprocessing on the literature; 2) Extract keywords from the literature; 3) Use the Embedding vectorization model to represent keywords using semantic vectors; 4) Based on the vectorization results, multiple clustering algorithms are used to classify the keywords, and according to the semantic features of each keyword, a large language model is used to generate the corresponding research perspective. 5) Utilize large language models to generate hierarchical and categorized review texts for single articles and groups of articles from different research perspectives. By integrating the summary texts from various research perspectives, an optimized and complete review text is formed, achieving a comprehensive and structured summary result. 6) Utilize large language models to implement comprehensive expansion, polishing, and logical optimization. Relying on the analytical and judgment capabilities of large language models combined with the semantic matching function of Embedding models, locate relevant literature and insert citations in appropriate positions. Step 1) specifically includes: 1.1: Obtaining Document Clusters Literature related to the review topic is obtained from literature databases or uploaded by users to form a literature cluster related to the review topic. This literature collection is a collection of literature related to a specific research topic, a specific research institution, or an individual researcher. 1.2: Preliminary screening of literature Each search result that points to a webpage containing a literature article is examined to determine whether it is related to the selected review topic. If it is related, the literature content from the webpage is extracted and saved; if it is not related, it is considered an irrelevant article and is not saved. 1.3: Text Extraction The text content of the documents is parsed to extract the document title, abstract and main text, and then organized into a structured dataset. The keyword extraction in step 2) adopts one or more of the following schemes: Option 1: Extracting keywords using regularization Keywords are extracted by regularizing code rules, and similar patterns are found by using regular expressions in the code to accurately extract keywords that have been explicitly listed in the literature. Option 2: Keyword Extraction Using Algorithms Keywords are extracted from the text using algorithms including but not limited to TF-IDF, TextRank, or LDA topic models. Option 3: Keyword Extraction by Model The keywords are obtained by summarizing the literature or abstract three times using a large language model, and then input into the large language model to further summarize the keywords. Step 3) specifically includes: 3.1: Integrate all keywords into a keyword list and establish a correspondence between documents and keywords, with one document corresponding to multiple keywords and the same keyword corresponding to multiple documents; 3.2: Vectorize all keywords using an Embedding vectorization model, which includes, but is not limited to, SimBERT, BERT, or text2vec-large and related language variants; Step 4) specifically includes: 4.1: Calculate and determine the cosine similarity distance between each keyword; 4.2: Based on the calculated distance between keywords, perform cluster analysis. The cluster analysis operation uses clustering algorithms including but not limited to hierarchical clustering, K-means clustering, or density clustering algorithms. 4.3: Based on the clustering algorithm, the analysis results of each class are composed of multiple keywords. The large language model is used to construct the corresponding research perspective based on the keywords of each class. 4.4: Match each document with its corresponding keywords and research perspectives to form a collection of documents from each research perspective; Step 5) uses a large language model to perform the following three-dimensional literature review: 5.1: Use a large language model to summarize and extract the full text of each document, and generate a review of each document. 5.2: The literature summary texts from various research perspectives are classified and summarized a second time using a large language model, focusing on the relevant research perspectives. The research perspectives should be clearly stated in the prompt words during the second classification and summary, guiding the model to accurately extract the literature under the specific research topic, so as to ensure the accuracy and completeness of the extraction results. The literature summary texts from different perspectives are provided as part of the output results for users to refer to. 5.3: Based on the review texts generated from various research perspectives, a comprehensive analysis of the overall research theme is conducted to obtain the final review text; Step 6) specifically includes: 6.1: Expand the article content using a large language model and optimize it logically by adding transitional sentences that connect the context and strengthening the expression of logical thinking to improve the overall quality of the article; 6.2: The system allows for manual intervention through an interactive interface to adjust and polish specific paragraphs or sentences. It supports undoing, regenerating text, and custom editing. 6.3: Vectorize the full text of each document using the Embedding model. If the total word count exceeds the maximum length that the model can input, it needs to be vectorized in segments. 6.4: Evaluate the final review text sentence by sentence using a large language model. If the evaluation results indicate that citations need to be added, proceed to the next step; if the evaluation results indicate that no citations need to be added, perform the same evaluation on the next sentence. 6.5: If a specific statement needs to be cited, it should be vectorized and a search enhancement method similar to RAG should be used to identify the most relevant references by calculating cosine similarity and selecting the reference with the highest similarity. 6.6: Set a threshold to determine whether to include the most similar document in the citation. If the document with the highest similarity does not meet the threshold requirement, give the user the option to manually add the document.

2. The method for intelligent classification and automatic review generation of multiple documents based on a large language model according to claim 1, characterized in that, The large-scale language model employs manual intervention during each literature review to re-summarize those literatures deemed insufficiently summarized, ensuring that the review content meets user needs.

3. The method for intelligent classification and automatic review generation of multiple documents based on a large language model according to claim 1, characterized in that, The implementation of the large language model includes, but is not limited to, API interfaces for commercial models, localized models, or customized models.

Images