Multi-modal report automatic generation method and system based on small-scale LLM and RAG technologies

Abstract

The invention provides a multi-modal report automatic generation method and system based on small-scale LLM and RAG technologies, aiming at solving the problems that technical report compilation in the fields of tunnel engineering and the like depends on manpower, a general model is insufficient in specialty, the multi-format processing capacity is weak, the deployment cost is high and the like. The method comprises the steps of multi-format text extraction and path reservation, domain knowledge driven outline generation, RAG-based content optimization and recombination and report assembly and output. According to the method, full-process automation of the multi-modal report is realized, the generation efficiency and specialty are improved, and the deployment cost and the data security risk are reduced.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and engineering knowledge management technology, and relates to a method and system for automatically generating multimodal reports based on small-scale LLM and RAG technologies. Background Technology

[0002] In fields such as tunnel engineering, compiling large-scale technical reports (such as construction organization designs, completion summaries, and technical achievement reports) requires integrating massive amounts of text materials from various formats, including Word, PDF, and PPT, such as construction logs, technical specifications, meeting minutes, research reports, and patent documents. Currently, this process is highly dependent on manual labor, resulting in problems such as low efficiency, poor content consistency, inconsistent use of professional terminology, and susceptibility to errors.

[0003] Existing automation solutions that use a general large language model (LLM) have significant drawbacks: (1) Lack of domain knowledge: The general model lacks professional knowledge of tunnel engineering, has a superficial understanding of construction methods, parameters and specifications, and generates low technical accuracy of content. It requires a lot of manual correction and cannot be used directly.

[0004] (2) Context length limitation: It is not possible to process and generate a complete report of tens of thousands of words at once.

[0005] (3) Insufficient multi-format text processing capability: Although it can process some formats, it is not effective for complex layouts (such as accurately extracting formatted text and tables from PDF), nested tables, and extracting technical points from engineering PPTs, resulting in serious information loss.

[0006] (4) Lack of processing strategies for non-text elements: Non-text elements such as pictures and CAD drawings are common in engineering documents. Common solutions usually ignore or cannot process them, resulting in the generated report lacking key visual information.

[0007] (5) Deployment cost and data security: Large-scale general-purpose models have high computing power requirements and cannot be deployed on ordinary project site computers. Using cloud APIs poses a risk of leakage of core engineering data.

[0008] Therefore, there is an urgent need in this field for an automated report generation solution that possesses deep domain knowledge, can accurately process multi-format text, has appropriate strategies for handling non-text elements, and can be deployed locally at low cost. Summary of the Invention

[0009] To address the problems existing in the background technology, this invention proposes a method and system for automatically generating multimodal reports based on small-scale LLM and RAG technologies.

[0010] To achieve the above objectives, the technical solution adopted by the present invention is as follows: On the one hand, this invention provides an automated method for generating multimodal reports based on small-scale LLM and RAG technologies, including: Multi-format text extraction and path preservation steps: Perform format-specific text extraction on engineering documents of various formats, record paths for non-text elements, and output a standardized intermediate format file containing text content and path reference tags; Domain knowledge-driven outline generation steps: call the vertical large model of the tunnel domain to analyze the intermediate format file and generate a hierarchical report outline that conforms to engineering logic; RAG-based content optimization and reorganization steps: Perform semantic retrieval based on the outline, find relevant text fragments from the document library, and call the vertical large model for content optimization and reorganization; Report assembly and output steps: Convert the optimized content into the final report format, and embed non-text elements into the corresponding positions in the report according to the path information.

[0011] Furthermore, the multi-format text extraction and path preservation steps include: Use an enhanced parser to extract text and table data from PDF documents; Extract slide text and notes from a PowerPoint presentation; For images and CAD drawing attachments embedded in the document, record their path information in the original document or save a copy of the file.

[0012] Furthermore, the standardized intermediate format is Markdown format, and non-text elements are represented by quotation marks.

[0013] Furthermore, the vertical large model in the tunnel domain is a model obtained by fine-tuning existing models on the tunnel engineering corpus, with fewer than 10 bytes of parameters, and can be deployed locally on a personal computer.

[0014] Furthermore, the RAG-based content optimization and reorganization steps include: Perform semantic retrieval based on the requirements of each chapter in the outline to find relevant text fragments; The vertical large model is invoked to integrate, polish, correct errors, and standardize professional terminology in the retrieved text; During processing, retain reference tags for non-text elements such as images and CAD files.

[0015] Furthermore, the report assembly and output steps include: Convert Markdown content containing text and image reference tags to DOCX format; Based on the path information, physically embed the corresponding images and CAD files into the DOCX document; Automatically apply preset document style templates to generate a table of contents and page numbers.

[0016] Furthermore, the method supports at least three of the following input document formats: Word, PDF, PPT, Excel, and TXT.

[0017] Furthermore, the semantic retrieval is based on the BM25 algorithm or a vector similarity matching algorithm.

[0018] On the other hand, the present invention also provides a multimodal report automated generation system for implementing the above method, comprising: The multi-format text extraction and path preservation module is configured to extract text and record paths from engineering documents in multiple formats. The domain knowledge-driven outline generation module is configured to call the vertical large model of the tunnel domain to generate a report outline; The content optimization and reorganization module based on RAG is configured to perform semantic retrieval and content optimization. The report assembly and output module is configured to generate a final format report and embed non-text elements.

[0019] Compared with the prior art, the present invention has the following beneficial effects: (1) Excellent domain text processing capabilities: The vertical large model has a much deeper understanding of tunnel professional texts and generates more accurate reports than the general model. The technical credibility of the generated reports exceeds 95%, and they can even be directly used in engineering practice.

[0020] (2) Accurate extraction of text in multiple formats: Optimized for extracting text in complex formats such as PDF and PPT, improving information completeness by more than 40%, and effectively preserving the tables and data of the original document.

[0021] (3) Practical handling of non-text elements: Through the strategy of "path preservation and physical embedding", the required images, drawings and other elements in the report were 100% preserved without the ability to recognize multimodal data, thus ensuring the integrity of the report.

[0022] (4) Extremely low deployment and usage costs: Small-scale vertical models can run smoothly on personal computers, reducing computing power costs by more than 80%, achieving complete localization of engineering data processing, and ensuring security.

[0023] (5) Full-process automation and high efficiency: The entire process from document parsing to report generation is automated, reducing the manual compilation work that originally took several weeks to several hours, improving efficiency by more than 10 times. Attached Figure Description

[0024] Figure 1 This is a flowchart of the method of the present invention; Figure 2 This is a flowchart of the multi-format text extraction and path preservation steps of the present invention; Figure 3 This is a flowchart of the outline generation steps driven by knowledge in the field of this invention; Figure 4 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] On the one hand, such as Figure 1 As shown, this invention provides an automated method for generating multimodal reports based on small-scale LLM and RAG technologies, including: Multi-format text extraction and path preservation steps: Perform format-specific text extraction on engineering documents of various formats, record paths for non-text elements, and output a standardized intermediate format file containing text content and path reference tags; Domain knowledge-driven outline generation steps: call the vertical large model of the tunnel domain to analyze the intermediate format file and generate a hierarchical report outline that conforms to engineering logic; RAG-based content optimization and reorganization steps: Perform semantic retrieval based on the outline, find relevant text fragments from the document library, and call the vertical large model for content optimization and reorganization; Report assembly and output steps: Convert the optimized content into the final report format, and embed non-text elements into the corresponding positions in the report according to the path information.

[0027] The implementation environment of this embodiment is a personal computer equipped with an Intel i7-12600k processor and 32GB of memory. The system is deployed without using a dedicated GPU.

[0028] We collected over 500 documents for a tunnel project, including technical specifications (PDF), construction weekly reports (Word), technical briefing PPTs, and material inspection reports (Excel).

[0029] like Figure 2As shown, the system parses files one by one, performing format-specific text extraction on engineering documents of various formats. For PDF documents, an enhanced parser is used to extract text and table data. For PPT documents, slide text and notes are extracted, along with the title and body text of each page.

[0030] It records the paths of non-text elements and outputs a standardized intermediate format file containing text content and path reference tags. For images in all documents, it only records their location and filename. For images and CAD drawing attachments embedded in documents, it records their path information in the original document or saves a file copy and centrally saves the copies. Finally, it generates a unified Markdown knowledge containing text and image reference tags. Non-text elements in Markdown are represented as reference tags in the form of a library.

[0031] The intermediate format file is analyzed using a large-scale vertical model for the tunnel engineering field, generating a hierarchical report outline that conforms to engineering logic. A locally deployed large-scale vertical model for the tunnel engineering field, based on knowledge base content, automatically generates a professional report outline including chapters on project overview, construction methods, machinery and equipment, quality control, and safety measures. This large-scale vertical model for the tunnel engineering field is a model finely tuned from an existing model on a tunnel engineering corpus, with fewer than 10 bytes of parameters, and is deployed locally on a personal computer.

[0032] like Figure 3 As shown, semantic retrieval is performed based on the outline to find relevant text fragments in the document library, and the vertical large-scale model is called for content optimization and reorganization. Semantic retrieval is performed according to the requirements of each chapter of the outline to find relevant text fragments. Based on the requirements of the "Construction Methods" chapter, the system uses semantic retrieval to find relevant technical briefing PPT text and construction weekly report descriptions. The semantic retrieval is based on the BM25 algorithm or vector similarity matching algorithm. The vertical large-scale model is called to integrate, polish, correct errors, and unify professional terminology in the retrieved text, generating professional and accurate narrative text. During processing, reference tags for non-text elements such as images and CAD files are retained, as well as the tags for referenced process flow diagram images.

[0033] The optimized content is converted into the final report format, and non-text elements are embedded into the corresponding positions in the report based on the path information. The Markdown content containing text and image reference tags in each chapter is converted into DOCX. Based on the path of the image tags, the system automatically inserts the corresponding process flow diagrams, on-site photos, etc. into the specified positions in the document, automatically applies the preset document style template, and generates a table of contents and page numbers to produce a professional tunnel construction report of more than 100 pages with corresponding text and images.

[0034] Results Verification: Compared to traditional methods, the professionalism and accuracy of the reports generated by this solution have been highly recognized by domain experts, and all reference images have been fully preserved. The entire process requires no manual intervention and takes only 4 hours, demonstrating a significant improvement in efficiency.

[0035] On the other hand, such as Figure 4 As shown, the present invention also provides a multimodal report automated generation system for implementing the above method, comprising: The multi-format text extraction and path preservation module is configured to extract text and record paths from engineering documents in multiple formats. The domain knowledge-driven outline generation module is configured to call the vertical large model of the tunnel domain to generate a report outline; The content optimization and reorganization module based on RAG is configured to perform semantic retrieval and content optimization. The report assembly and output module is configured to generate a final format report and embed non-text elements.

[0036] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for automatically generating multimodal reports based on small-scale LLM and RAG technologies, characterized in that, include: Multi-format text extraction and path preservation steps: Perform format-specific text extraction on engineering documents of various formats, record paths for non-text elements, and output a standardized intermediate format file containing text content and path reference tags; Domain knowledge-driven outline generation steps: call the vertical large model of the tunnel domain to analyze the intermediate format file and generate a hierarchical report outline that conforms to engineering logic; RAG-based content optimization and reorganization steps: Perform semantic retrieval based on the outline, find relevant text fragments from the document library, and call the vertical large model for content optimization and reorganization; Report assembly and output steps: Convert the optimized content into the final report format, and embed non-text elements into the corresponding positions in the report according to the path information.

2. The method according to claim 1, characterized in that, The multi-format text extraction and path preservation steps include: Use an enhanced parser to extract text and table data from PDF documents; Extract slide text and notes from a PowerPoint presentation; For images and CAD drawing attachments embedded in the document, record their path information in the original document or save a copy of the file.

3. The method according to claim 1, characterized in that, The standardized intermediate format is Markdown, and non-text elements are represented by quotation tags.

4. The method according to claim 1, characterized in that, The vertical large model in the tunnel domain is a model obtained by fine-tuning an existing model on a tunnel engineering corpus. It has fewer than 10 bytes of parameters and can be deployed locally on a personal computer.

5. The method according to claim 1, characterized in that, The RAG-based content optimization and reorganization steps include: Perform semantic retrieval based on the requirements of each chapter in the outline to find relevant text fragments; The vertical large model is invoked to integrate, polish, correct errors, and standardize professional terminology in the retrieved text; During processing, retain reference tags for non-text elements such as images and CAD files.

6. The method according to claim 1, characterized in that, The report assembly and output steps include: Convert Markdown content containing text and image reference tags to DOCX format; Based on the path information, physically embed the corresponding images and CAD files into the DOCX document; Automatically apply preset document style templates to generate a table of contents and page numbers.

7. The method according to claim 1, characterized in that, The method supports input document formats including at least three of the following: Word, PDF, PPT, Excel, and TXT.

8. The method according to claim 1, characterized in that, The semantic retrieval is based on the BM25 algorithm or a vector similarity matching algorithm.

9. A multimodal report automated generation system for implementing the method of any one of claims 1-8, characterized in that, include: The multi-format text extraction and path preservation module is configured to extract text and record paths from engineering documents in multiple formats. The domain knowledge-driven outline generation module is configured to call the vertical large model of the tunnel domain to generate a report outline; The content optimization and reorganization module based on RAG is configured to perform semantic retrieval and content optimization. The report assembly and output module is configured to generate a final format report and embed non-text elements.

Images