Method for generating picture-text report by fusing multi-modal large language model and RAG mechanism

Abstract

This invention discloses a method for generating image-text reports by integrating a multimodal large language model and a RAG mechanism, belonging to the field of text processing technology. The method first converts PDF documents into images, then uses a multimodal model to identify and extract text, tables, charts, and other content to construct a searchable knowledge fragment library. Next, based on user queries, a hybrid retrieval strategy is used to obtain relevant evidence, and a Markdown report containing image placeholders is generated using the large language model. Simultaneously, a text-to-image module is invoked to generate accompanying images, ultimately achieving a visually integrated image-text report output. This invention supports multimodal content understanding, cross-modal retrieval, and collaborative generation, possessing good generalization, accuracy, and practicality, and is suitable for complex document processing and report generation across multiple fields and languages.

Description

Technical Field

[0001] This invention belongs to the field of text processing technology, specifically relating to a method for generating graphic and text reports that integrates a multimodal large language model and the RAG mechanism. Background Technology

[0002] In recent years, the combination of multimodal large language models (MLLMs) and retrieval-enhanced generation (RAG) technology has provided new ideas for intelligent document processing and image-text report generation. Traditional methods, when processing complex and multimodal PDF documents, usually rely heavily on manual extraction or predefined templates, resulting in low efficiency, poor scalability, and weak cross-modal semantic associations.

[0003] In existing technologies, rule-based template methods can quickly generate reports, but they struggle to adapt to complex layouts such as multi-column and multi-page tables, and template customization and maintenance are costly. Machine learning-based methods utilize pre-trained models for text and image feature extraction and achieve information fusion through cross-modal attention, but their performance is highly dependent on the scale and quality of training data, lacks generalization ability in specialized fields or data-scarce scenarios, and suffers from high computational resource consumption and difficulty in fine-tuning. Retrieval-enhanced generation (RAG) methods improve the accuracy and credibility of generated content by introducing external knowledge retrieval, but still face limitations such as high retrieval latency, weak multimodal retrieval support, and lagging knowledge updates. While multimodal large language models (MLLM) can process mixed text and image content end-to-end and support flexible generation, they still have shortcomings in generation control, factual accuracy, and process interpretability, and require extremely high computational power.

[0004] Especially in application scenarios with high requirements for accuracy, timeliness and visualization (such as policy analysis and academic research report writing), existing methods are unable to achieve high-confidence semantic alignment of text and graphics, fine-grained source citation and dynamic knowledge integration, which limits their widespread application in actual business. Summary of the Invention

[0005] To address the aforementioned technical issues, this invention provides a method for generating graphic and textual reports that integrates a multimodal large language model and the RAG mechanism. This method supports multimodal content understanding, cross-modal retrieval, and collaborative generation, and possesses good generalization, accuracy, and practicality. It is suitable for complex document processing and report generation in multiple fields and languages.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A method for generating graphic reports that integrates a multimodal large language model and a RAG mechanism, the method comprising:

[0008] Step 1, Preprocessing and Knowledge Construction: Convert the input PDF document into a sequence of page images, use a multimodal model to perform layout analysis and feature extraction, generate knowledge fragments in the form of text, tables and charts, and record the document source and location information for each knowledge fragment;

[0009] Step 2: Construct a hybrid retrieval system: The knowledge fragments are vectorized using a text embedding model to construct a hybrid retrieval system that simultaneously supports vector similarity retrieval and keyword matching.

[0010] Step 3, Retrieval and Rearrangement: Parse the user's query intent, use the hybrid retrieval tool to obtain a set of relevant evidence fragments, and rearrange the set of evidence fragments based on the maximum marginal relevance criterion;

[0011] Step 4, Image and Text Collaborative Generation: Based on the rearranged set of evidence fragments, generate accompanying image descriptions and image placeholders, and use a large language model to generate a draft report in Markdown format containing the image placeholders and a statement of evidence source;

[0012] Step 5, Reference Binding and Rendering Output: Perform vector similarity calculation and threshold determination on the report paragraph units in the draft report and the knowledge fragments in the knowledge base. After consistency verification and terminology unification, render and output the final graphic report.

[0013] Secondly, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for generating graphic reports by integrating a multimodal large language model and a RAG mechanism.

[0014] Thirdly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned method for generating graphic reports that integrates a multimodal large language model and a RAG mechanism.

[0015] The beneficial effects of this invention are as follows:

[0016] Automation and efficiency: It can automatically parse unstructured PDF documents without relying on manual rules and pre-annotated templates, and achieve unified understanding and extraction of multimodal content such as text, tables and charts, which greatly improves report generation efficiency and reduces labor costs.

[0017] Strong cross-modal semantic fusion capability: By combining a multimodal large model with a hybrid retrieval mechanism (vector + keyword), it effectively captures the deep semantic relationship between text and images, ensuring that the textual arguments and visual illustrations in the generated report are highly consistent in content and avoiding the disconnect between text and images.

[0018] The generated content is accurate and reliable: The introduction of Search Enhanced Generation (RAG) and fine-grained citation binding mechanism links source evidence to each unit of discussion in the report, enhancing the traceability and factual accuracy of the results, and meeting the high reliability requirements of academic, policy, and industry analysis applications.

[0019] The system is flexible and practical: it supports customizing report content and style through natural language interaction, adapts to complex document processing in multiple fields and languages; the output supports multiple formats such as Markdown, HTML, and PDF, which facilitates integration and subsequent typesetting applications, and has good versatility and scalability. Attached Figure Description

[0020] Figure 1 This is a schematic diagram illustrating the principle of the graphic report generation method that integrates a multimodal large language model and the RAG mechanism according to the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] Complex PDF documents possess multiple modalities, including text, tables, charts, formulas, and images. Their structure is loose, with frequent cross-page connections, dense terminology, and domain specificity. Traditional rule-based or unimodal solutions struggle to achieve high-confidence automatic extraction and recombination while maintaining semantic integrity. Even with the introduction of a multimodal large language model (MLLM), without controllable retrieval enhancement and citation binding, factual biases or semantic misalignments between text and images may occur. Therefore, this invention focuses on the following core issues:

[0023] How to normalize different modalities to the "knowledge fragment" level with a unified preprocessing and representation learning pipeline to support interpretable retrieval;

[0024] How to achieve a balance between semantic matching and exact term matching using a hybrid retrieval strategy, and reduce omissions and redundancy;

[0025] How to establish a fine-grained two-way binding between generated text and source evidence to improve traceability and academic compliance;

[0026] How to achieve collaborative generation and consistent rendering of "text, image, and layout" to ensure that the final report is consistent in structure, evidence, and visual expression.

[0027] like Figure 1 As shown, this invention proposes an end-to-end pipeline consisting of "preprocessing and knowledge construction—hybrid retrieval—image and text collaborative generation—reference binding—layout rendering". The specific steps are as follows:

[0028] Step 1, Preprocessing and Knowledge Construction: The input PDF document is converted page by page into a high-resolution image sequence. A multimodal layout analysis model (such as LayoutLM or DocFormer) is used to perform structural recognition on each page image, extracting elements such as text blocks, table cells, and chart descriptions to form structured "knowledge fragments." Each knowledge fragment is accompanied by source metadata, including document ID, page number, and location information, forming a triple (document ID, page number, location information).

[0029] In particular, to address the common issue of cross-page tables in complex PDF documents, the system detects visual cues (including repeated table headers, continued borders, and sequential numbering) and semantic continuity between adjacent pages to merge table portions scattered across different pages.

[0030] Candidate detection: Locate the table area in adjacent pages and extract the header text, border lines, and cell coordinates.

[0031] Visual matching: Calculate the similarity of the header string simhead(hi,hj) and the continuation of the border contbox to obtain the visual score: V(Ti,Tj)=γ1·simhead+γ2·contbox.

[0032] Semantic matching: Vectorize the content of adjacent rows of cells in the table, calculate the cosine similarity, and sum them to obtain a semantic score.

[0033] = ,

[0034] The function for recognizing cross-page tables is:

[0035] R(Ti,Tj)=α·V(Ti,Tj)+β·S(Ti,Tj)

[0036] Where Ti and Tj are the candidate tables for adjacent pages, respectively; α and β are weighting coefficients.

[0037] If the overall score R(Ti,Tj) exceeds the threshold θ, then Ti and Tj are merged into a complete table Tij, and stored in the knowledge base with a unified table description structure.

[0038] Subsequently, a text embedding model (such as Sentence-BERT) is used to vectorize the text content of each knowledge fragment, resulting in a vector representation e(c), and a vector index library is constructed. Simultaneously, a keyword inverted index is built to support precise term-level matching. At this point, the unstructured PDF document is transformed into a structured, searchable knowledge base.

[0039] Step 2: Construct a hybrid search engine: Construct a hybrid search engine that combines support vector similarity and keyword matching.

[0040] The hybrid search engine combines the advantages of vector search and keyword search (BM25) to balance semantic relevance and terminology accuracy.

[0041] Vector retrieval: Encode the user query q into a vector e(q), and calculate its cosine similarity with each knowledge fragment vector e(c):

[0042] ,

[0043] Keyword retrieval: If the query contains key terms, relevant fragments are retrieved from the inverted index using the BM25 algorithm, and the BM25 score BM25(q,c) is calculated.

[0044] Hybrid Score: The two calculation results are weighted and combined to obtain the final search score.

[0045] Score(q,c)=α· +(1-α)·BM25(q,c),

[0046] Here, α∈[0,1] is an adjustable weight parameter used to balance the importance of semantics and term matching.

[0047] Step 3, Retrieval and Reordering: The system performs semantic parsing and intent recognition on user queries, breaking them down into subqueries if necessary. A hybrid retrieval tool is used to initially retrieve relevant evidence fragments, which are then rearranged using the Maximum Marginal Relevance (MMR) criterion to obtain the evidence set, balancing relevance and diversity.

[0048] λ·Score(q,c)-(1-λ)· ,

[0049] in, For the selected evidence set, λ controls the trade-off between relevance and redundancy removal.

[0050] The system calls a lightweight large language model (such as GPT-3.5-turbo) as the evaluator. The user query q and the evidence set Ccet after MMR rearrangement are input into the evaluator. The evaluator makes a comprehensive judgment based on two dimensions: relevance (whether the evidence matches the query intent) and coverage (whether the evidence covers the key elements of the user query).

[0051] The evaluator outputs a comprehensive quality score Sq between 0 and 1, and a diagnostic code Dcode to identify the main problems that may exist in the search results, such as: terminology_mismatch: indicating that there is a difference between the key terms in the query and the descriptions in the evidence set; query_ambiguity: indicating that the original query is too short or ambiguous, resulting in a scattered search intent.

[0052] insufficient_coverage: indicates that the evidence set does not fully cover all sub-intents of the query;

[0053] Sufficient (good quality): This indicates that the evidence set does not require optimization.

[0054] The system presets a quality threshold τ. If the overall score Sq ≥ τ, the process continues directly to step 4 for collaborative image and text generation. If the overall score Sq < τ, the system triggers the corresponding optimization strategy based on the diagnostic code Dcode.

[0055] When the diagnostic code is terminology_mismatch: the system determines that the semantic vector space is inconsistent with the keyword representation. The optimizer will automatically adjust the weight parameter α of the hybrid search engine to favor keyword matching, and perform a second round of retrieval using the original query q. The new evidence set Cnew will be merged with the initial evidence set Ccet and re-ranked through MMR to enrich the evidence related to the key terms.

[0056] When the diagnostic result is query_ambiguity or insufficient_coverage: the system determines that the query itself needs optimization. The optimizer will call the same lightweight LLM to refactor and expand the original query.

[0057] Subsequently, the system uses the reconstructed query q' and the original weight parameter α to perform a second round of retrieval, and merges and rearranges the results of the two rounds. The final evidence set Cfinal is obtained after the optimization process.

[0058] Step 4: Collaborative generation of text and images;

[0059] Based on the structured information in the evidence set (such as indicators, comparison dimensions, time / region, etc.), generate accompanying illustrations and image placeholders. Call the text generation module (such as Stable Diffusion or DALL·E) to generate corresponding schematic diagrams (such as trend charts, distribution maps, thematic maps, etc.).

[0060] Simultaneously, a large language model (such as GPT-4 or Claude) is invoked to generate a draft report in Markdown format. The model declares the source of evidence at the beginning of each paragraph and organizes the content internally at the granular level of "report paragraph units" (sentences or paragraphs) to ensure logical integrity. Domain terminology is standardized according to a preset terminology library during the generation process to ensure consistency.

[0061] Next, the generated content is subject to fact-checking: key fact statements are automatically extracted from the text content of the Markdown report draft. These key fact statements include quantitative data (such as "25% growth"), time points (such as "in 2022"), entity relationships (such as "Company A acquired Company B"), and core assertions (such as "Technology C is the optimal solution").

[0062] Subsequently, the system treats each extracted statement s as a new query intent, invokes the hybrid retrieval tool constructed in step 2, and retrieves the original evidence fragment Es most semantically relevant to that statement from the knowledge base. It calculates the cosine similarity between the vector representation of each statement and the vector representations of all its related evidence fragments, and sets two decision thresholds: a support threshold τsup and a risk threshold τrisk.

[0063] The risk assessment logic is as follows: If there exists evidence e∈Es with a similarity cos(s, e) > τsup with statement s, then the statement is considered to have strong evidence support and is low risk; if the similarity cos(s,e) of all evidence e∈Es with s is < τrisk, then the statement is considered high risk; if max(cos(s,e)) is between τrisk and τsup, then the statement is considered to have medium risk, with weak evidence support, and may be model inference or generalization.

[0064] For high-risk statements, the system directly visually marks them in the Markdown draft, for example, using HTML tags. <mark style="background-color：#ffe6a1;">Highlight the yellow background and insert a detailed Markdown comment in the adjacent line, with the format: <!--Fact-checking warning: This statement may conflict with the records in the source [document ID, page number]. Please review.-->

[0065] Step 5: Citation binding and rendering output: Calculate the vector similarity between the report paragraph units and knowledge fragments in the draft graphic report, and bind the threshold, and render and output the final graphic report.

[0066] For each "report paragraph unit" p in the draft report, calculate its cosine similarity with each knowledge fragment c in the knowledge base . If the maximum similarity exceeds the preset threshold τ, bind the report paragraph unit to the corresponding knowledge fragment and record the source information.

[0067] If a paragraph unit contains multiple sentences, calculate the sentence-level similarity and merge the sources to form a paragraph-level citation list. The system also performs a consistency check: if there are conflicts for the same term in different sources, weighted selection is performed according to the similarity and source authority, and manual review is prompted.

[0068] Finally, perform unified standardization processing on the terms in the full text, replace the image placeholders with the actual image paths, automatically generate cross-references and reference lists, and render and output the final report in Markdown, HTML or PDF format.

[0069] Example:

[0070] Taking the macro trend research of the global smart wearable device market as an example, 300 relevant PDF reports were collected, including corporate financial reports, industry white papers and academic papers. The document languages cover Chinese and English, with complex structures and huge amounts of data. According to the foregoing method of the present invention, the steps for generating a statistical report include:

[0071] Step 1: Automatic knowledge base construction;

[0072] The system batch-converts 300 PDF documents into images and calls a multi-modal large model for unified parsing. Millions of "knowledge fragments" are successfully extracted and vectorized, such as "Company A's headset shipments in 2023: 20 million pairs", "Summary of B subsidy rules", "Line chart of the patent growth rate of Technology C", forming a giant retrievable knowledge base across documents and modalities.

[0073] Step 2: Complex query and hybrid retrieval;

[0074] User query: "Analyze the technological innovation, changes in the market share of major manufacturers and their driving factors in the health monitoring field of smart wearable devices in the Asia-Pacific region in the past three years."

[0075] System response: The system identified three sub-intents: "technological innovation", "market share change", and "driving factors".

[0076] Hybrid search engine startup:

[0077] Semantic retrieval: Find fragments that are semantically related to "technological innovation" and "driving factors".

[0078] Keyword search: Exactly matches terms such as "Asia-Pacific", "health monitoring", "market share", and "Company A, B, C".

[0079] Through weighted formula After initial screening and subsequent MMR rearrangement to remove redundancy, a set of cross-document, highly relevant, and diverse "evidence sets" was finally selected from 300 documents.

[0080] Search quality assessment initiated:

[0081] The system concatenates the user query with the evidence set and inputs it into a lightweight LLM evaluator. The evaluator generates the following diagnostic report:

[0082] {

[0083] "quality_score": 0.85,

[0084] "diagnosis_code": "sufficient"

[0085] }

[0086] Assuming the preset threshold is 0.7, since the quality score of 0.85 is higher than the threshold and the diagnosis reason is "good quality", the system proceeds to step 3.

[0087] Step 3: Image and text collaborative generation and reference binding;

[0088] The Wensheng Graph module automatically generates an "Asia-Pacific Market Share Trend Chart" and a "Health Monitoring Technology Comparison Radar Chart" based on data from the evidence set regarding market share changes and technology comparisons.

[0089] Large Language Model: Based on the evidence set, write a structured draft report in fluent Chinese and English, including "Analysis of Technological Innovation", "Evolution of Competitive Landscape" and "Policy and Market Drivers", and insert image placeholders and source statements in each argument unit (paragraph).

[0090] Citation binding: The system calculates the vector similarity (cos(e(p), e(c))) between each conclusion in the report and the source knowledge fragment, and automatically binds it to the original page number and document to generate detailed citation annotations.

[0091] Step 4: Multi-format rendering output;

[0092] The system ultimately outputs a detailed market research report of over 50 pages (in Markdown and PDF formats), featuring data visualizations, cross-references, and a list of references. All terminology is consistent, and all data conclusions can be traced back to specific page numbers in the original report.

[0093] This embodiment demonstrates the powerful ability of the present invention to process massive amounts of heterogeneous documents. What traditionally requires weeks of manual work to sort through literature and write reports can be completed in hours by this system. This not only greatly improves efficiency but also ensures the accuracy, traceability, and visualization of the content, providing efficient and reliable data support for decision-making.

[0094] Secondly, the present invention provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the aforementioned method for generating graphic reports by integrating a multimodal large language model and a RAG mechanism.

[0095] Thirdly, the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enable the processor to implement the aforementioned method for generating graphic reports that integrates a multimodal large language model and a RAG mechanism.

[0096] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.< / mark>

Claims

1. A method for generating graphic reports by integrating a multimodal large language model and a RAG mechanism, characterized in that, The method includes: Step 1, Preprocessing and Knowledge Construction: Convert the input PDF document into a sequence of page images, use a multimodal model to perform layout analysis and feature extraction, and generate knowledge fragments in the form of text, tables and charts. For each knowledge fragment, record the source and location information of the knowledge fragment in the form of a (document identifier, page number, location coordinates) triple. Step 2: Construct a hybrid retrieval system: The knowledge fragments are vectorized using a text embedding model to construct a hybrid retrieval system that simultaneously supports vector similarity retrieval and keyword matching. Step 3, Retrieval and Rearrangement: Analyze the user's query intent, use the hybrid retrieval tool to obtain a set of relevant evidence fragments, and rearrange the evidence fragment set based on the maximum marginal relevance criterion; including: iteratively selecting the evidence fragment with the lowest similarity to the selected evidence set while ensuring high relevance to the query; using a lightweight large language model to score the quality of the rearranged evidence fragment set; if the quality score is lower than a preset threshold, perform a second round of retrieval; merge and rearrange the results of the two rounds of retrieval to form the final evidence set; Step 4, Image and Text Collaborative Generation: Based on the rearranged set of evidence fragments, generate accompanying image descriptions and image placeholders. Use a large language model to generate a draft report in Markdown format containing the image placeholders and evidence source statements. This includes: the system first generates detailed accompanying image description text based on the structured data points in the evidence fragment set, then inputs the accompanying image description text into the text-to-image module to generate a visual chart that is semantically consistent with the report content; extracts key factual statements from the draft Markdown format report, performs semantic similarity matching and risk assessment with the original evidence in the knowledge base, and marks or revises statements with deviation risks. The key factual statements include numerical data, time information, entity relationship assertions, and core conclusive descriptions. Step 5, Reference Binding and Rendering Output: Perform vector similarity calculation and threshold determination on the report paragraph units in the report draft and the knowledge fragments in the knowledge base. If the cosine similarity value exceeds the preset threshold, bind the corresponding knowledge fragment as the reference source for the report paragraph unit. After consistency verification and terminology unification, render and output the final graphic report.

2. The method for generating graphic reports by integrating a multimodal large language model and a RAG mechanism according to claim 1, characterized in that, In step 1, when there are cross-page tables, the table parts scattered across different pages are merged across pages by detecting visual cues and semantic continuity of adjacent pages, and a complete table representation is formed in the knowledge fragment library.

3. The method for generating graphic reports by integrating a multimodal large language model and RAG mechanism according to claim 1, characterized in that, In step 2, the retrieval process of the hybrid retrieval system is as follows: the user query is encoded into a vector, and its cosine similarity score with the knowledge fragment vector is calculated; at the same time, the matching score between the query and the knowledge fragment is calculated through a keyword matching algorithm; finally, the cosine similarity score and the matching score are weighted and fused to obtain a comprehensive retrieval score.

4. The method for generating graphic reports by integrating a multimodal large language model and a RAG mechanism according to claim 3, characterized in that, The evaluation dimensions of the lightweight large language model include: the relevance of evidence fragments to query intent, and the coverage of the evidence set to the key elements of the user query; finally, a quality score and the reason for the judgment are output; if the quality score is greater than or equal to the threshold, the process proceeds directly to step 4; if the quality score is less than the threshold, the query keywords are automatically reconstructed or the weight parameters of the mixed retrieval are adjusted according to the reason for the judgment, and a second round of retrieval is performed.

5. The method for generating graphic reports by integrating a multimodal large language model and a RAG mechanism according to claim 1, characterized in that, In step 5, the consistency verification includes: when the same term is found to have definition or numerical conflicts in different knowledge segments, the system makes a comprehensive judgment based on the authority of the source of the knowledge segment and the similarity with the report paragraph unit, selects the preferred source, and prompts the conflict location for the user to review.

6. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When one or more programs are executed by the one or more processors, the one or more processors implement the graphic report generation method that integrates a multimodal large language model and a RAG mechanism as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, It stores executable instructions that, when executed by a processor, enable the processor to implement the graphic report generation method that integrates a multimodal large language model and a RAG mechanism as described in any one of claims 1-5.

Images