Multi-literature intelligent classification and automatic review generation method based on large language model

Abstract

The invention discloses a multi-literature intelligent classification and automatic review generation method based on a large language model, which is characterized by comprising the following steps of: retrieving literatures or uploading local literatures, and extracting text contents; identifying keywords by utilizing an algorithm or a large language model; converting the keyword into a vector form by using a vectorization model; summarizing the research direction of the clustering result by using a large language model; comprehensively summarizing the single literature and literatures in different research directions by using a large language model; and executing the steps of reviewing, quotation insertion and the like through a large language model and a vectorization semantic matching technology. Compared with the prior art, the method has the advantages that the efficiency and quality of document classification and review generation are remarkably improved, technologies such as text vectorization, topic clustering and large language model generation are well fused, efficient and accurate multi-document review automatic generation is achieved, the method is suitable for interdisciplinary academic document analysis and information integration application, and the method is suitable for popularization and application. Wide application potential is shown.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing and automatic review technology, and in particular to a multi-document intelligent classification and automatic review generation method based on a large language model. Background Art

[0002] In the field of scientific research, the importance of literature is self-evident, especially in the broad scope of literature processing and knowledge management, where literature classification and automatic review technology play a vital role. Traditional literature classification methods, such as manual classification, are limited by human resources and time costs, and are difficult to adapt to the continuous growth of the number of literature. Therefore, the literature classification method based on machine learning has emerged, providing an innovative solution for the efficient organization and management of literature.

[0003] In the field of document classification, researchers have conducted extensive explorations on different types of documents. Since the concept of automatic document classification was first proposed in the 1950s, the research process in this field can be roughly divided into three stages: feasibility study of automatic classification, experimental research, and practical application. In the past two decades, the practical application stage has achieved remarkable results. In terms of document classification methods, researchers have adopted a variety of technologies, including: classifiers based on support vector machines (SVMs), automated Dewey decimal classification combined with machine learning and text mining, multi-layer library document classification models using classifiers such as naive Bayes, support vector machines, and decision trees (C4.5), and bidirectional encoding representation models (BERT) based on deep learning. At the same time, automatic review technology, as an advanced form of automatic text summarization, has also received widespread attention. The initial automatic review research used data processing machines to scan documents, calculated the relative measure of significance by counting word frequency and distribution information, and then extracted and printed out the sentences with the highest importance scores to form summary text. With the advancement of technology, automatic review technology has gradually developed from simple extractive summarization to complex generative summarization. Extractive summarization mainly uses cluster analysis, supervised SVM machine learning models, and neural networks to identify and extract core sentences from documents, while generative summarization uses deep learning models to generate completely new text as summaries.

[0004] Although automatic review technology has made some progress, it still faces many challenges in terms of comprehensiveness and accuracy. The introduction of large language models (LLMs) has brought new breakthroughs to automatic review technology. With its powerful language understanding ability, LLMs make automatic review more efficient when processing complex texts. For example, through large language models, the system can understand and integrate implicit information and contextual relationships in the literature to generate more coherent and accurate summaries. However, these models also have shortcomings in practical applications. For example, the generated review content may lack systematicity and it is difficult to fully reflect the relationship between the literature; when dealing with complex topics, the model may not be able to accurately capture the subtle differences between different studies; at the same time, when updating data, current tools are less flexible and cannot reflect the latest research trends in a timely manner. These problems limit the development and widespread application of automatic review technology. Summary of the invention

[0005] The purpose of the present invention is to provide a multi-document intelligent classification and automatic review generation method based on a large language model to address the deficiencies in the prior art. The multi-document intelligent document classification method using keyword extraction and vectorization is adopted, and automatic reviews are generated using a large language model to achieve rapid classification and comprehensive summary of a large number of documents. The method is simple and efficient, and can significantly improve the efficiency and quality of automatic reviews of multiple documents, providing strong support for academic research and practical applications, and better solving the problems of low efficiency and insufficient precision in document classification and review generation. It is suitable for academic document analysis and information integration tasks in various fields, and can achieve efficient and accurate automatic generation of multi-document reviews, and has a wide range of application scenarios and application values.

[0006] The specific technical solution for achieving the purpose of the present invention is: a method for intelligent classification and automatic review generation of multiple documents based on a large language model, which is characterized in that the method specifically includes the following steps: 1) Obtain a collection of documents on the target research topic from the literature database, and perform preliminary filtering and text preprocessing on the documents; 2) Extract keywords from the literature; 3) Use vectorization models to represent keywords in semantic vectors; 4) Based on the vectorization results, the keywords are classified by using a variety of clustering algorithms, and the corresponding research angles are generated using a large language model according to the semantic characteristics of each type of keyword; 5) Use the large language model to generate hierarchical and classified review texts for single articles and groups of articles from different research perspectives; finally, by integrating the summary texts from various research perspectives, an optimized complete review text is formed to achieve a comprehensive and structured summary result; 6) With the help of large-scale language models, comprehensive expansion, polishing and logic optimization are implemented, and relying on the analysis and judgment capabilities of the large model combined with the semantic matching function of the Embedding model, relevant literature can be efficiently located and citations can be inserted in appropriate locations.

[0007] Compared with the prior art, the present invention has the following beneficial technical advances and significant technical effects: 1) This invention supports dynamic adaptation and expansion of different subject topics, highlights "dynamic adaptation" and "multi-level semantic analysis", and can realize the flexible use of knowledge resources in multi-field research. At the same time, by introducing multi-level semantic analysis technology, it can provide in-depth analysis and breadth coverage in document subject classification and multi-angle review generation, thereby meeting users' needs for analysis at different levels.

[0008] 2) The present invention has significant advantages in performance optimization and module extensibility by integrating modules such as literature screening, keyword extraction, clustering classification and review generation. It not only has the ability of efficient calling, but also can realize dynamic expansion of functions through modular design, adapt to data processing tasks of different scales, and can process a large amount of literature resources in a short time, greatly shortening the time cost of traditional methods and avoiding errors caused by manual operations.

[0009] 3) In terms of document classification and data association, the present invention supports intelligent classification and topic-focused analysis of massive documents by combining keyword semantic vectorization with clustering algorithms. Combined with the natural language generation capability of the large language model, it can generate structured and logical review texts, providing researchers with a reliable research foundation and analysis tools.

[0010] 4) This invention aims at the hallucination problem that may occur when a large language model generates a review. Through multiple rounds of generation and aggregation strategies, the uncertainty of the generated results is effectively reduced. Through multiple tests and consistency screening, the accuracy and stability of the review generation can be significantly improved, ensuring the credibility and availability of the final output results. Finally, through standardized templates and intelligent optimization technology, not only the content richness and logical rigor of the generated review are guaranteed, but also the normative requirements of academic writing are met, providing an efficient and reliable solution for academic research and information integration.

[0011] 5) For the entire research field, the present invention uses cluster analysis methods to divide all relevant literature into different research perspectives, thereby revealing the research focus and trends of each sub-field. By making a preliminary summary and in-depth analysis of these research perspectives, it can ensure that the final review report is more comprehensive and accurate in content. In addition, these summary texts from different perspectives can not only serve as part of the review report to provide users with more abundant reference information, but also help researchers quickly grasp the core ideas and research progress of each sub-field.

[0012] 6) In terms of citation annotation technology, the present invention adopts a large language model to determine whether citation annotation is needed, and by vectorizing the document text and using the cosine similarity calculation method, it can achieve a more accurate positioning of the required cited documents. In addition, since the article already has a complete logical structure and core ideas, inappropriate citation insertion is avoided. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is a flow chart of the present invention; Figure 2 This is the flowchart of literature preprocessing; Figure 3 A flow chart for classifying the research directions of the literature; Figure 4 A flowchart summarizing all the literature and research directions; Figure 5 Flowchart for logical refinement and citation insertion of the final review; DETAILED DESCRIPTION

[0014] See also Figure 1 ,Multi-document intelligent classification and automatic review generation based on large language model specifically include the following steps: 1) Literature acquisition, screening and text preprocessing (i.e. step 1) See also Figure 2 , obtain documents related to the review topic from the literature database or upload them by users themselves, and form a cluster of documents related to the topic. Then, the documents are preliminarily screened, and the abstracts pointing to the document web pages in the search results are checked one by one to determine whether they are related to the review topic. For related documents, the content of the web pages is extracted and saved; for unrelated documents, they are regarded as irrelevant documents and not saved. Finally, the text content of the documents is parsed, the title, abstract and text are extracted, and organized into a structured data set for subsequent analysis and use.

[0015] 2) Keyword extraction and vectorization (i.e. steps 2 and 3) See also Figure 3According to user needs, regularization rules can be used to extract keywords clearly marked in the document. For documents without keyword annotations or user needs, keywords can also be extracted or generated through TF-IDF, TextRank algorithm or large language model. All keywords are semantically represented by embedding model vectorization models such as SimBERT, BERT or text2vec-large. Through keyword extraction and vectorization, the document topic can be efficiently mapped to the semantic space to achieve accurate analysis of the core content of the document. During the vectorization process, different optimization strategies can be selected according to the application scenario. For example, for multilingual documents, a multilingual version of the vectorization model can be used.

[0016] 3) Cluster analysis and research perspective construction (i.e. step 4) See also Figure 3 , calculate the cosine distance of the semantic similarity between keywords, and use clustering algorithms such as hierarchical clustering, K-means or DBSCAN to classify keywords to form a research perspective. Each document is matched with its corresponding keyword and perspective to construct a document collection under each research perspective. In practical applications, hierarchical clustering can intuitively present the hierarchical relationship between documents and is suitable for small-scale data sets; the K-means algorithm is more suitable for large-scale document collections; and DBSCAN performs better in identifying sparse or heterogeneous data. The present invention selects the optimal clustering algorithm based on the characteristics of specific document data. In addition, the present invention adopts a dynamic clustering threshold adjustment strategy in cluster analysis to adapt to document collections with different data distributions. This flexible adjustment mechanism makes the clustering results more accurate and helps to build subsequent research perspectives.

[0017] 4) Automatic review generation (i.e. step 5) See also Figure 4 , the automatic review generates single document summary, comprehensive review and research perspective review. The single document summary uses a large language model to make a preliminary summary of the content of each document and extract the core information; the research perspective review classifies and summarizes the document collection of each research perspective, and summarizes the research hotspots and trends of each perspective; the comprehensive review integrates the review content of each perspective and combines the large language model to generate the final structured review text. In the comprehensive review generation stage, the present invention combines multiple rounds of optimization strategies to iteratively improve the initially generated review text, and further adjusts the model input prompts through quality evaluation of the generated results to ensure the logic and coherence of the review text.

[0018] 5) Citation recommendation and optimization (i.e. step 6) See also Figure 5, vectorize the documents through the Embedding model, match the most relevant documents with the RAG (enhanced retrieval) method, set the matching threshold to ensure the relevance and accuracy of the citation, and finally complete the citation integration, logic optimization and content polishing through the large language model. In practical applications, the choice of Embedding model can be adjusted according to the document type. For example, for field-specific documents, a field-optimized model can be used. At the same time, a manual citation modification function is provided through the user interface to meet personalized needs.

[0019] The present invention is described in detail below in conjunction with the embodiments and drawings, wherein the large language model used refers to a natural language processing model that can complete basic conversations, and no specific requirements are made on the size of parameters, adaptive language, deployment platform, and implementation solution. Example

[0020] See also Figure 1 This embodiment provides a review generation method based on a large language model, which includes six steps: document content extraction, keyword extraction, keyword classification and direction summary, document classification summary, comprehensive document review summary, review polishing and citation insertion. The large language model involved uses the Qwen2.5:7b model.

[0021] See also Figure 2 , pre-process the literature (i.e. step 1) After determining the review topic, such as "digital transformation of education", search from the "CNKI Full-text Database" and set the keyword "digital transformation of education". To ensure the relevance of the research, screen the high-quality research papers in the fields of education such as "education theory and education management", "higher education", "vocational education", "secondary education", "adult education and special education", and "primary education" and included in CSSCI. According to the relevant definition of digital transformation of education, the obtained literature data is denoised to obtain valid data. After saving all relevant documents, you can use multiple libraries in Python (PyPDF2, pdfminer, etc.) to parse the text content in the PDF document. In addition, you can also use the API interface of the relevant database to obtain the required literature, read the complete content of the literature, and implement structured processing through programming, and then sort out a list of structured literature related to the review topic.

[0022] See also Figure 3 , keyword extraction and vectorization (i.e. steps 2 and 3) The format of keywords in the document usually starts with "Keyword:" or "[Keyword]", and keywords are usually separated by spaces, commas or semicolons. By matching the documents with regular expressions, extracting keywords from the documents, or using algorithms such as TF-IDF, TextRank, and large language models to summarize the full text, a complete keyword set is formed. At the same time, the correspondence between documents and keywords is established, and the extracted keywords are processed using models such as SIMBERT to obtain the vector representation of each keyword.

[0023] See also Figure 3 , the step of classifying the research direction of the literature (i.e. steps 2 and 3) is realized through clustering algorithm Select an appropriate clustering algorithm based on the characteristics of the literature data and its classification requirements. Common clustering algorithms include K-means, hierarchical clustering, DBSCAN, spectral clustering, etc. After selecting a clustering algorithm, configure the corresponding parameters. Input the vectorized keyword data into the selected clustering algorithm to perform clustering operations. After clustering is completed, use the large language model to extract the research direction, and combine the correspondence table between literature and keywords to establish the relationship between research direction, keywords and literature.

[0024] See also Figure 4 , Literature summary step (i.e. step 5) Use a large language model (such as Qwen2.5 7b or related APIs) to summarize and refine each document, and then classify and summarize the summary texts of the documents corresponding to different research topics on related research topics. Based on the review texts generated for each topic, conduct a comprehensive analysis of the overall research objectives and obtain a final review of all document collections.

[0025] See also Figure 5 , final review expansion and polishing, logic optimization and citation insertion steps (i.e. step 6) After logical optimization and polishing of the review text, citations are inserted using a large language model. First, embedding vectorization is performed on the document collection, and then the large language model is used to determine whether citation argumentation is needed for the review text sentence by sentence. If argumentation is necessary, the sentence is vectorized and matched with the semantics of the document collection. The judgment is made by setting a threshold, determining the documents that can be inserted, and adjusting the format to insert the documents. This process continues until all documents are properly inserted.

[0026] The above is only a further explanation of the present invention and is not intended to limit the patent of the present invention. Any implementation that is equivalent to the present invention should be included in the scope of the claims of the patent of the present invention.

Claims

1. A method for intelligent classification and automatic review generation of multiple documents based on a large language model, characterized in that: The method specifically comprises the following steps: 1) Obtain a collection of documents on the target research topic from the literature database, and perform preliminary filtering and text preprocessing on the documents; 2) Extract keywords from the literature; 3) Use the Embedding vectorization model to represent the semantic vectors of keywords; 4) Based on the vectorization results, a variety of clustering algorithms are used to classify keywords, and according to the semantic characteristics of each type of keyword, a large language model is used to generate corresponding research angles; 5) Use the large language model to generate graded and classified review texts for single articles and groups of articles from different research perspectives. By integrating the summary texts from various research perspectives, an optimized complete review text is formed to achieve a comprehensive and structured summary result; 6) Use the large language model to implement comprehensive expansion, polishing and logic optimization, and rely on the analysis and judgment capabilities of the large language model combined with the semantic matching function of the Embedding model to locate relevant literature and insert citations at appropriate locations.

2. The method for intelligent classification and automatic review generation of multiple documents based on a large language model according to claim 1 is characterized in that: The step 1) specifically includes: 1.1: Obtaining document clusters Obtain documents related to the review topic from the literature database or upload documents by users to form a document cluster related to the review topic. This document cluster is a document cluster related to a specific research topic, a specific research institution or an individual researcher. 1.2: Preliminary screening of literature Check the abstracts of the literature web pages in the search results one by one to determine whether they are related to the selected review topic. If so, extract the literature content in the web page and save it; if not, treat it as an irrelevant literature and do not save it; 1.3: Text Extraction Parse the text content of the document, extract the document title, document abstract and document text, and organize them into a structured data set.

3. The method for intelligent classification and automatic review generation of multiple documents based on a large language model according to claim 1, characterized in that: The keyword extraction in step 2) adopts one or more of the following schemes: Solution 1: Regularization to extract keywords Extract keywords through regularized code rules, use regular expressions in the code to find similar patterns, and accurately extract keywords that have been clearly listed in the literature; Solution 2: Algorithm to extract keywords Extract keywords from the text using algorithms including but not limited to: TF-IDF, TextRank or LDA topic model; Solution 3: Model Extraction Keywords Use the large language model to summarize the literature or abstract three times to obtain keywords, and input them into the large language model to further summarize the keywords.

4. The method for intelligent classification and automatic review generation of multiple documents based on a large language model according to claim 1 is characterized in that: The step 3) specifically includes: 3.1: Integrate all keywords into a keyword table, and form a corresponding relationship between documents and keywords. The same document corresponds to multiple keywords, and the same keyword corresponds to multiple documents; 3.2: All keywords are vectorized using the Embedding vectorization model. The vectorization model includes but is not limited to SimBERT, BERT or text2vec-large and its related language variants.

5. The method for intelligent classification and automatic review generation of multiple documents based on a large language model according to claim 1 is characterized in that: The step 4) specifically includes: 4.1: Calculate and determine the cosine similarity distance between each keyword; 4.2: Performing a clustering analysis operation according to the calculated distance between keywords, wherein the clustering algorithm adopted by the clustering analysis operation includes but is not limited to hierarchical clustering, K-means clustering or density clustering algorithm; 4.3: Based on the clustering algorithm, we obtain the analysis results of each category composed of multiple keywords, and use the large language model to build the corresponding research perspective according to the keywords of each category; 4.4: Match each document with its corresponding keywords and research perspectives to form a collection of documents under each research perspective.

6. The method for intelligent classification and automatic review generation of multiple documents based on a large language model according to claim 1 is characterized in that: Step 5) uses the large language model to perform the following three literature summaries: 5.1: Use the large language model to summarize and refine the full text content of each document and generate a review of each document; 5.2: The literature summary texts from various research perspectives are classified and summarized for the second time with the help of a large language model, focusing on the relevant research perspectives. The research perspectives need to be clearly stated in the prompt words during the second classification and summary, guiding the model to accurately extract the literature under this specific research topic to ensure the accuracy and completeness of the extraction results, and the literature summary texts from different perspectives are used as part of the output results for user reference; 5.3: Based on the review texts generated from each research perspective, conduct a comprehensive analysis of the overall research topic to obtain the final review text.

7. The method for intelligent classification and automatic review generation of multiple documents based on a large language model according to claim 1 is characterized in that: The step 6) specifically includes: 6.1: Expand the content of the article with the help of a large language model and optimize its logic by adding transitional sentences that connect the context and strengthening the expression of logical thinking to improve the overall quality of the article; 6.2: Manual intervention is implemented with the help of an interactive interface to adjust and polish specific paragraphs or sentences. The system supports undo operations, regenerating text, and custom editing; 6.3: Vectorize the full text of each document through the Embedding model. If the number of words in the full text exceeds the maximum length that the model can input, it needs to be vectorized in segments; 6.4: Evaluate the final review text sentence by sentence through the large language model. If the evaluation result shows that a citation is needed, proceed to the next step. If the evaluation result shows that a citation is not needed, perform the same evaluation on the next sentence. 6.5: If a specific statement needs to be cited, it should be vectorized and a search enhancement method similar to RAG should be used to identify the most relevant references by calculating cosine similarity and select the reference with the highest similarity; 6.6: Set a threshold to decide whether to include the most similar document in the citation. If the most similar document does not meet the threshold requirement, give the user the option to manually add the document.

8. The method for intelligent classification and automatic review generation of multiple documents based on a large language model according to claim 2 is characterized in that: The large language model uses manual intervention every time a document is summarized, and re-summarizes those documents that are considered to be insufficiently summarized to ensure that the content of the review meets user needs.

9. The method for intelligent classification and automatic review generation of multiple documents based on a large language model according to claim 1 or 2, characterized in that: The implementation of the large language model includes but is not limited to an API interface of a commercial model, a localized model or a customized model.

Images