Report auto-generation search enhancement optimization method and system based on local knowledge base

Abstract

The application discloses a retrieval enhancement optimization method and system based on local knowledge base report automatic generation, and proposes a reconstruction retrieval enhancement controllable generation architecture, which comprises the following steps: document preprocessing, extracting document structure, eliminating invalid content, dividing the document into blocks, vectorization and storage into a database; a problem reconstruction module, which decomposes a user problem into four parts, i.e., a retrieval keyword, a user intention, a problem rewriting and a problem expansion; a retrieval enhancement module, which applies a keyword search method to the retrieval keyword, applies a vector semantic retrieval method to the problem rewriting and the problem expansion, combines, filters, reorders, compresses and aligns the retrieval results; and finally, the retrieval results are combined with the user intention, and a specific prompt template is applied to generate text content based on the local knowledge base. The application standardizes the structure of unstructured files, extracts chart and picture information, decomposes, reconstructs and expands the user input, and generates a specific format document based on the local knowledge.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, specifically to a method and system for automatically generating and optimizing reports based on a local knowledge base, and enhancing search capabilities. Background Technology

[0002] With the rapid development of natural language processing technology, especially breakthroughs in deep learning and large language models, it has become possible to understand and generate human language. Nowadays, large language models have excellent text classification, question answering, and summary generation capabilities. However, in practical use, due to the black-box nature of large language models, they are prone to producing illusions and are not accurate enough in answering questions about private domain knowledge and real-time messages. This makes them difficult for enterprises with data privacy concerns to apply. Retrieval Augmentation (RAG) is a method aimed at improving the quality and efficiency of information retrieval. Retrieval results are usually used as supplementary knowledge for large language models to enhance their capabilities.

[0003] However, existing search enhancement generation has the following problems in use: (1) Search enhancement generation is not perfect in extracting information such as tables and pictures in unstructured documents; (2) There is a difference between the user's question and the actual search question; (3) The search accuracy and relevance are insufficient, and the search results are too redundant; (4) There are flaws in the output of documents in specific formats. Therefore, this application proposes a search enhancement optimization method and system based on the automatic generation of reports from a local knowledge base, and proposes to reconstruct the search enhancement generation architecture. Summary of the Invention

[0004] The purpose of this invention is to provide a retrieval enhancement and optimization method and system based on automatically generated local knowledge base reports. Specifically, it standardizes the structure of unstructured documents, extracts chart and image information, deconstructs, reconstructs, and expands user input, and generates documents of specific formats based on local knowledge. This effectively solves problems such as incomplete extraction of information from unstructured documents (tables, images, etc.); discrepancies between user queries and actual search queries; insufficient retrieval accuracy and relevance; excessively redundant search results; and defects in outputting documents of specific formats.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a retrieval enhancement method for automatic report generation based on a local knowledge base, comprising:

[0006] A new architecture for reconstructing and enhancing retrieval generation is proposed. Knowledge base document preprocessing involves extracting table and image information and standardizing document formatting. A question reconstruction method decomposes, reconstructs, and expands the original user input question into four parts: retrieval keywords, user intent, question rewriting, and question expansion. Keyword retrieval algorithms are applied to retrieval keywords, while vector semantic retrieval is applied to question rewriting and expansion. The two retrieval results are filtered, reordered, compressed, and merged to form a logical context. This context is then used as external knowledge, and text with a specific format is generated based on this knowledge and following a known template.

[0007] Furthermore, the document preprocessing uses OpenCV to detect and extract the location of incomplete wireframe tables, including:

[0008] First, convert the pages you need to extract from the PDF into images;

[0009] The image is processed by binarizing the grayscale image;

[0010] Use the binarized image to obtain the horizontal and vertical lines in the table, using the morphological functions Erosion and Dilation in OpenCV;

[0011] The obtained horizontal and vertical lines are intersected to obtain the intersection points. The outline of the intersection points is then judged to determine whether it is a table. Finally, the outline is approximated and converted into a rectangle.

[0012] Furthermore, the keyword extraction in the problem reconstruction method utilizes an improved TextRank algorithm, including:

[0013] Using keyword part-of-speech matching and the sliding window method to construct the initial undirected graph can accurately capture keyword relationships and improve the semantic expressiveness of the graph, reducing the computational complexity of subsequent algorithms.

[0014] Introducing a dynamic damping factor to satisfy the nonperiodicity and irreducibility of the state transition matrix can balance the scores between different types of nodes, thereby improving the accuracy and robustness of the results.

[0015] Introducing cosine similarity as the weight of the edge between two nodes can accelerate Markov convergence, represent the relationship between nodes more richly, and handle more complex text relationships, thus significantly improving the accuracy of keyword extraction.

[0016] Adding an L1 regularization term causes some parameter weights to become zero, which helps to build a sparse graph structure, allowing the algorithm to focus more on important nodes and edges, extract more representative keywords, and also improves computational efficiency.

[0017] Furthermore, the extraction of problem intent in the problem reconstruction module includes:

[0018] Use the Neo4j graph database to pre-build a database for classifying a problem intent;

[0019] By using the Cypher query language, various graph operations can be performed efficiently to retrieve intents, sub-intents, and related intents;

[0020] Using semantic similarity as the criterion, the cosine similarity between the retrieved intent and the original question is calculated, and the intent of the node with the highest similarity is taken as the final question intent;

[0021] By manipulating the question intent node, its sub-intents and related intents can be retrieved;

[0022] Furthermore, the process of filtering the search results, taking into full account various text characteristics, proposes a comprehensive text information index, including:

[0023] Text repetition rate is calculated by forming a text vector based on the frequency of each word, and then using cosine similarity to calculate the repetition rate between two texts.

[0024] The unit text information entropy calculation represents the amount of information contained in the text;

[0025] Calculate text similarity using the embedded text vectors;

[0026] The text information comprehensive index can be represented as a linear combination of unit text information entropy and text similarity. First, the text repetition rate of the search results is calculated, and texts with high repetition rates are initially filtered. Then, the information comprehensive index of each text is calculated. Texts with too low information comprehensive index can be discarded or compressed.

[0027] A retrieval enhancement system for automatically generating local knowledge base reports includes:

[0028] The question restructuring module determines whether local knowledge base content is needed. If local knowledge base retrieval is required, it narrows the gap between user input and retrieval, and performs question restructuring. The four parts of the question restructuring are stored in key-value pair format, including:

[0029] Search for keywords and extract the search keywords provided by the user, as well as the most representative keywords from the entire input.

[0030] User intent: Extract the purpose and requirements of user input, and transform redundant user input into simple semantic instructions;

[0031] The problem is rewritten to enrich and specify the user input, complete incomplete user input, and remove redundant parts;

[0032] Problem expansion involves expanding the original problem into multiple problems, including three aspects: word recombination, grammatical structure transformation, and intention expansion.

[0033] The retrieval processing module is used to retrieve documents related to the question, including:

[0034] The document is segmented, and the segmented document and question are vectorized using an embedding model;

[0035] The k most similar sentence blocks are found by calculating the cosine similarity.

[0036] The post-retrieval processing module is used to perform preliminary filtering and compression of the retrieval results, including:

[0037] The repetition rate between statement blocks is calculated, and statement blocks with high repetition rates are initially filtered out. Then, the comprehensive text information index of the remaining statement blocks is calculated, and statement blocks with low indices are discarded or compressed.

[0038] The text generation module obtains the knowledge base context from the retrieval processing module and generates user intent from the question reconstruction module. Through organic combination and specific format templates, it utilizes the text generation capabilities of the large language model to generate the final text.

[0039] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0040] 1. This invention provides a retrieval enhancement and optimization method and system based on automatically generated local knowledge base reports, improving upon previous methods that suffered from inaccurate local document retrieval, low relevance, and the illusion of large models. The innovative feature is a question reconstruction module that reconstructs user input, improving the clarity and understandability of the question, as well as the accuracy and relevance of the answer, thus enhancing the user experience. Keyword extraction utilizes an improved TextRank algorithm, which offers faster convergence, more accurate keyword extraction, and higher computational efficiency. Simultaneously, the retrieval results are filtered and compressed, significantly shortening the context of the model input, improving the output efficiency of large models, and applying specific prompt templates to ensure controllable output while unlocking the potential of large model text output.

[0041] 2. The method designed in this invention first extracts and vectorizes the local document structure. In the past, user input was usually directly retrieved in the RAG structure. This technical solution adds a question reconstruction module, which generalizes the original question to improve the accuracy and breadth of retrieval. It can identify whether a retrieval is needed and what to retrieve without fine-tuning the model, thus adding intelligent performance to the RAG structure. In the output stage, controllable output is achieved through customized templates and local knowledge. This method has superior performance in both retrieval results and final generated text. Attached Figure Description

[0042] Figure 1 This is a schematic diagram of the overall process of the present invention;

[0043] Figure 2 for Figure 1 Flowchart of the problem refactoring module;

[0044] Figure 3 for Figure 1 Flowchart of the retrieval processing module;

[0045] Figure 4 for Figure 1 Flowchart of the post-retrieval processing module;

[0046] Figure 5 Flowchart for the text generation module. Detailed Implementation

[0047] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0048] Please see Figures 1-5 This invention provides a technical solution: a retrieval enhancement and optimization method and system based on automatically generated local knowledge base reports, referring to... Figure 1 This is a flowchart of the search enhancement and optimization process automatically generated based on local knowledge base reports. It includes a question reconstruction module, a search processing module and a post-search processing module, and a text generation module. The user input is directly connected to the question reconstruction module. After the user inputs the question, it enters the question reconstruction module. Part of the reconstructed result enters the search processing module. The search processing module generates relevant context fragments and combines them with another part of the reconstructed result to enter the text generation module. In the text generation module, different prompt templates are selected and applied according to different content to generate text content with different characteristics based on the local knowledge base.

[0049] Document preprocessing includes extracting the structure and content of unstructured documents such as Word and PDF documents, and finally storing them in JSON data format. This includes the following steps:

[0050] For Word documents, the python-docx library is used to preprocess the Word documents, such as extracting the document page number, header, footer, images, tables, styles and other basic information as the document, and extracting the text content of each chapter as the JSON data body;

[0051] The PDFPlumber library is used to extract text, images, and regular tables from PDF documents. Then, the PDF is traversed a second time, and OCR technology is used to extract text and locate and extract irregular tables from the PDF file. The obtained information is stored in a JSON file in the form of link paths.

[0052] The image data is stored in an external folder, and the JSON data is stored in the form of an index.

[0053] The JSON data is divided into two parts. The first part is the basic information of the document, including the number of pages, headers, footers, and styles. The second part is the main content, which is stored in units of paragraphs, with each paragraph marked with the chapter it belongs to.

[0054] Reference Figure 2 The question reconstruction module is the first module that users enter after inputting their questions. This module reconstructs the original question into four parts: search keywords, user intent, question rewriting, and question expansion.

[0055] The first step in problem reconstruction is to extract the keywords from the original problem, which includes the following steps:

[0056] First, the jieba library is used to perform word segmentation on the original problem. After word segmentation, the Harbin Institute of Technology stop word list is loaded, which can be filtered and expanded according to needs. Then, the stop words are removed from the segmented results.

[0057] Iterate through the remaining words, determine the part of speech for each word and mark it, and use a dictionary to store each word and its part of speech;

[0058] At this point, taking "What will the weather be like tomorrow?" as an example, the specific structure of the resulting vocabulary list is as follows:

[0059] {“tomorrow”:“t”,“weather”:“n”,“how”:“r”}.

[0060] Where t is a time word, n is a noun, and r is a pronoun.

[0061] Based on part-of-speech consistency and combined with the sliding window method, an initial undirected graph is constructed, where two word nodes with the same part of speech are connected by a line, or two words within the same sliding window are connected by a line. The weight of each line is determined by the formula:

[0062]

[0063] The calculation is as follows, among which This represents the vector of the word after the embedding model.

[0064] Calculate the score for each word node using the formula.

[0065] Perform iterative calculations.

[0066] The introduction of dynamic damping factor, two-node text similarity, and L1 regularization term greatly improves the accuracy of keyword extraction.

[0067] After the iteration converges, the k words with the highest scores are selected as search keywords.

[0068] For user intent extraction in question reconstruction, an intent classification graph approach is adopted. Specifically, firstly, an intent classification graph database is constructed, which contains commonly used question intents. Each node represents an intent, each edge represents the relationship between intents, and each edge or node has a category attribute. The relationship between intents includes, but is not limited to, sub-intents, similar intents, and extended intents. Question intent extraction uses the Cypher query language to retrieve multiple candidate intents through fuzzy search. The candidate intents are then compared with the original question using a similarity algorithm. The intent node with the highest similarity is retained, and this intent is selected as the final question intent.

[0069] Problem expansion in problem reconstruction involves three aspects: 1. Lexical restructuring; 2. Grammatical transformation; 3. Intent expansion. Lexical restructuring includes synonym replacement, order swapping, and word removal. Grammatical transformation mainly involves grammatical structure transformation. Intent expansion can be achieved by querying similar or extended intents through retrieved intent nodes.

[0070] The original problem was refactored into four parts, and the refactored text is in JSON dataset format, as follows:

[0071] {"rag_keywords":"content","user_purpose":"content","query_rewrite":"content","expand_input":"content"}

[0072] After refactoring, rag_keywords, user_purpose, query_rewrite, and expand_input represent search keywords, user intent, query rewriting, and query expansion, respectively.

[0073] Reference Figure 3 This is a flowchart of the retrieval processing module. This module receives the reconstructed rag_keywords, query_rewrite, and expand_input parts, and outputs the relevant fragments retrieved from the local document. This module includes the following steps:

[0074] Step S310: Load local knowledge base documents, including various types such as pdf, word, txt, csv, and html, and read the local documents into memory.

[0075] Step S320: The loaded document is divided into blocks using a recursive segmentation method. This involves using a set of delimiters to layer and iterate, dividing the input text into smaller blocks. If the initial segmentation does not produce blocks of the required size or structure, the method will recursively call the resulting blocks using different delimiters or criteria until the required block size or structure is produced.

[0076] Step S330: After the block division is completed, the block-divided content is vectorized using a finely tuned BGE-M3-Embedding model.

[0077] Step S340: Create an index for the Faiss database, using the IndexFlatl2 index, and then store the vectorized data into the database.

[0078] Step S350: Retrieve fragments and perform a hybrid search based on BM25 and Embedding, i.e., rag_keywords applies keyword search, query_rewrite and expand_input apply vector search, and finally merge the two search results.

[0079] Reference Figure 4 This is the post-retrieval processing module that processes the search results. The steps are as follows:

[0080] Step S410: First, the search results are initially filtered by calculating the repetition rate between sentence blocks to remove sentence blocks with high repetition rates. The repetition rate is calculated using the cosine similarity formula, where the vector consists of the word frequencies of the sentence block.

[0081] Step S420: Re-rank the initially filtered search results. Here, the comprehensive text information index of the invention is used, and the formula is as follows.

[0082]

[0083] Where H(x) represents the information entropy of a unit character. This indicates the degree of relevance between the statement block and the original problem.

[0084] Text information with higher comprehensive indices is placed at the top and bottom, while sentence blocks with lower indices are placed in the middle.

[0085] Step S430: Compress the sorted statement blocks to remove stop words from each statement block.

[0086] Step S440: Summarize the processed statement blocks and use the large language model to summarize multiple statement blocks into a logical statement as supplementary knowledge to the large model.

[0087] Reference Figure 5 The text generation module receives the output from the retrieval post-processing module and uses it as external knowledge for the large model. It also proofreads the output of the large model to achieve controllable generation.

[0088] The processed search text is combined with the question intent from the reconstruction module to form the model input. For different types of text generation, such as paper reviews, company weekly reports, and marketing copy, which have specific formats and require domain knowledge, the final model input needs to be formed by combining the templates of that type.

[0089] The text generation module determines whether a template should be used and, if so, which type, based on the question intent. If no template is used, the main model simply combines the retrieved sentence blocks and the question to generate a question and answer. If a template is required, the text is output according to the template format. Taking an excellent paper evaluation as an example, the paper evaluation template format is shown below:

[0090] {

[0091] {<#system>: You are a senior paper review expert. You are able to output relevant content based on known information and strictly follow the prompt format.}

[0092] {<#Known Information>: Content generated by the post-retrieval processing module}

[0093] {<#prompt>: "Under the diligent and meticulous guidance of the <supervisor>, student [Name] completed the graduation project with the following title: [Thesis Title]. This thesis employed [Technical Methods] and utilized [Technologies and Tools], including [Technical Implementation Details], and completed / implemented / built the [System / Task]. Throughout the graduation project process, the student demonstrated a proactive attitude, in-depth thinking, and correctly fulfilled the content and requirements outlined in the graduation project assignment. The student is proficient in using relevant tools, and the guiding principles and design schemes were correct. The student demonstrates strong independent work ability, effectively connecting theory with practice and applying learned knowledge to solve practical problems. The thesis content is correct, the structure is clear, the concepts are well-defined, the organization is logical, and the writing is fluent. This thesis is rated as excellent."}

[0094] {<#Question Intent>: "Generate a paper evaluation for me based on the paper's content."}

[0095] }

[0096] The final model input consists of four parts. The first part, <#system>, defines the roles of the large model and specifies the task requirements. The second part, <#known information>, is the corpus obtained from the retrieval module, which serves as the external knowledge source for the large model. The third part, <#prompt>, is the format template for the output text. The template for evaluating excellent papers is provided above, and the large model will output the text in the corresponding format according to the given template. The fourth part, <#question intent>, is the user's purpose and task, generated by the question reconstruction module. Different question intents can lead to changes in <#system>, <#prompt>, and <#known information>.

[0097] Here, a locally deployed large language model is applied, using the final template as input. The model generates a paper evaluation according to the requirements. The paper evaluation output from the large model then enters a proofreading model. The proofreading model checks for errors or sensitive content in the output and verifies its consistency with the template, achieving controlled generation. Compared to other AI-generated evaluations that are stiff and empty, R-RAG+LLMs demonstrates its powerful text generation and summarization capabilities. It not only avoids clichés but also identifies the technical tools and implementation details used by the authors, analyzes the article, points out its shortcomings and deficiencies, and provides directions for improvement.

[0098] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A method for automatically generating and optimizing reports based on a local knowledge base, characterized in that: include: S11 knowledge base document processing: extract document structure and content, parse the information presented by tables and images, and format the documents uniformly; The S12 question reconstruction module breaks down the original question into four parts: search keywords, user intent, question rewriting, and question expansion. S13 applies keyword search to the search keywords, applies vector semantic retrieval to question rewriting and question expansion, calculates the semantic richness and question relevance of the two search results respectively, filters the search results, and finally merges the two results; S14 reorders and compresses the final search results, and forms a logical context by supplementing the syntactic structure. S15 organically combines the user intent with the contextual content and applies a specific template to generate the required text content based on the local knowledge base; Step S12, the problem reconstruction module, where keyword extraction uses an improved TextRank algorithm, includes: ； ； ； ； ； In the formula: This represents the score of the i-th keyword node; This represents the dynamic damping factor of the node; This represents the cosine similarity between node i and node j; This represents the set of nodes that point to node i. This represents the set of nodes pointed to by node j; Indicates the number of nodes in the set; This represents the L1 regularization term. This represents the regularization strength parameter; The improved TextRank algorithm first uses the jieba library for word segmentation when constructing the undirected graph. It constructs the initial undirected graph by analyzing whether the part-of-speech tags of the keywords are consistent and by using the sliding window method. Considering the consistency of part-of-speech tags can avoid some meaningless associations, and the sliding window method can enhance the connection strength between related words and improve the semantic accuracy of the undirected graph. Improve the damping factor to a dynamic damping factor. The dynamic damping factor is dynamically adjusted based on the importance of the nodes to prevent the algorithm from getting stuck in an infinite loop or isolated nodes from having too much influence. It balances the scores between different types of nodes. The dynamic damping factor is related to the similarity between nodes and the score of that node. The logistic function is used to map the damping factor to the [0,1] interval. The text similarity between two nodes is introduced when calculating the score. Here, the embedding vector of the pre-trained embedding model is used to calculate the similarity between nodes, thereby introducing contextual information and accelerating the convergence speed of the Markov model. L1 regularization is introduced to prevent scores from being overly concentrated on a few nodes. By penalizing the sum of the absolute values ​​of the scores, sparsity is encouraged, which can automatically filter out important features, making the scoring more effective and reasonable.

2. The method according to claim 1, characterized in that, Step S11 involves locating tables in a PDF document, including: converting the PDF to an image; processing the image and identifying the horizontal and vertical lines of the table; extracting the intersection points of the intersecting horizontal and vertical lines; approximating a rectangle using the outline of the intersection points; and extracting data from the rectangle.

3. The method according to claim 1, characterized in that, Step S12: Based on the retrieval-enhanced generation RAG architecture, a reconstructed retrieval-enhanced controllable generation R-RACG architecture is proposed. In the R-RACG architecture, the controllable generation employs a trained, pluggable proofreading model to detect and label potential false or erroneous information in the content generated by the large language model, filtering it before content output and gradually approaching the expected target through multiple rounds of feedback.

4. The method according to claim 1, characterized in that, Step S12, the question reconstruction module, uses the Neo4j graph database to extract question intents, constructs an intent classification graph, where each node represents an intent, establishes hierarchical relationships between intents, clarifies the relationships between main intents, sub-intents, and subdivided intents, models the cross-relationships of intents, identifies the cross-relationships and mutual influences between different intents, and uses the Cypher query language to perform intent queries on intent nodes, calculates the semantic similarity between the original question and the intent node, and uses a similarity of 0.9 or higher as the final question intent.

5. The method according to claim 1, characterized in that, Step S13: Filtering the search results. First, calculate the repetition rate between each statement block and the remaining statement blocks. If two statement blocks have a high degree of repetition, the statement block with the highest comprehensive text information index is retained. The comprehensive text information index is defined as follows: ； in, ; ； ； ； In the formula: Represents a block of statements The comprehensive index of textual information; This represents the information entropy of a unit of text; This indicates the degree of relevance of the statement block to the original problem, where , It is a vector after word embedding; , Indicates weight; Representing text and Repetition rate, of which , It is a vector composed of the frequencies of each word in the statement block; The repetition rate between statement blocks is calculated using the cosine similarity formula, where the vectors of the two statement blocks are composed of word frequencies.

6. The method according to claim 1, characterized in that, In step S14, the statement blocks are sorted according to the comprehensive index of text information as defined in claim 5. The sorting is bidirectional, that is, statement blocks with high comprehensive indices are located at the beginning and end, and statement blocks with low comprehensive indices are stored in the middle.

7. The method according to claim 1, characterized in that, Step S15: The large model prompts that multiple candidate templates are used. Different templates are intelligently selected or no template is used depending on the different intents of the question, so as to generate text content in different formats.

8. A system for implementing the method of any one of claims 1-7, comprising: The system includes a document preprocessing module, a problem reconstruction module, a retrieval processing module, a retrieval postprocessing module, and a text generation module.

Images