Energy field intelligent knowledge base question-answering system based on multi-model cooperation

Abstract

The invention discloses an energy field intelligent knowledge base question answering system based on multi-model cooperation, and aims to solve the problems of data type identification, image structured extraction and cross-modal indexing in energy field multi-modal document question answering. The system comprises 13 core modules, dynamic cooperation of a multi-mode large model and a large language model is achieved through a collaborative scheduling module, data types involved in user problems can be recognized, and corresponding modules can be called; structured information of images such as charts and flow charts can be extracted from documents and coded; and cross-modal retrieval is realized through a unified semantic vector. When answers are integrated, an attached source is quoted, and traceability is ensured. The system improves the accuracy and efficiency of complex document question answering in the energy field, and is suitable for professional document question answering scenes containing multiple types of images.

Description

Technical Field

[0001] This invention relates to the field of intelligent question answering and knowledge management technology, and in particular to an intelligent knowledge base question answering system for the energy field based on multi-model collaboration. Background Technology

[0002] As the complexity and diversity of knowledge in the energy sector continue to increase, traditional knowledge base question-answering systems are gradually revealing many limitations when processing multimodal data. Among existing technologies, question-answering systems based on Large Language Models (LLMs) are widely used, but they lack the ability to process unstructured data such as charts, flowcharts, and operation diagrams—data that often contains core structured knowledge in the energy sector, while traditional language models can only process plain text, resulting in low efficiency in cross-modal information processing and knowledge extraction.

[0003] The shortcomings of existing technologies are mainly reflected in the following three aspects: Limitations of a single model: Existing systems often rely on a single model (such as a pure text understanding model or an image recognition model), which cannot integrate related information from different data types. For example, when faced with multimodal problems containing images, tables, and text, it is difficult to fully understand and integrate diverse data, which can easily lead to information loss or misinterpretation.

[0004] Insufficient image structure extraction: Existing multimodal data extraction methods (such as OCR technology) can only recognize text in images and cannot capture the hierarchical structure and semantic relationships of charts and flowcharts. For example, key information such as the "node-connection" logic in flowcharts and data relationships in charts in the energy field cannot be effectively extracted, resulting in incomplete or inaccurate knowledge extraction.

[0005] Weak cross-modal indexing capabilities: Existing technologies mostly use simple text indexes, which cannot support joint retrieval of text and charts. When users query flowcharts or operation diagrams, the system cannot flexibly associate relevant text information, resulting in slow response speed and low accuracy. Summary of the Invention

[0006] To address the aforementioned issues, this invention aims to propose an intelligent knowledge base question-answering system for the energy field based on multi-model collaboration. Through the collaborative scheduling of multi-modal large models and large language models, it achieves dynamic processing and cross-modal indexing of multiple types of data, thereby improving the accuracy and response speed of question answering.

[0007] To achieve the above objectives, the technical solution of the present invention is implemented as follows: A multi-model collaborative intelligent knowledge base question-answering system for the energy field comprises 13 core modules: a document parsing and preprocessing module, a text slicing and multimodal vectorization module, a multi-domain energy knowledge base input module, an intent recognition and task classification module, a collaborative scheduling module, a multimodal structured processing module, a multimodal content vectorization module, a text content vectorization module, a cross-modal knowledge base retrieval module, a memory module, a multimodal large model answering module, a text model answering module, and an answer integration and structure referencing module. The collaborative scheduling module receives the output results from the intent recognition and task classification module through a preset data interface, enabling collaborative scheduling of the multimodal large model and the large language model. It dynamically calls corresponding modules to process multiple data types based on the data types involved in the question, and supports cross-modal indexing through unified semantic encoding of the multimodal content vectorization module and the text content vectorization module. All modules are interconnected through preset data interfaces and control signaling, forming a complete closed loop of "document parsing - knowledge input - intent recognition - collaborative scheduling - cross-modal retrieval - intelligent answering," adaptable to distributed system deployment scenarios.

[0008] Furthermore, the intent recognition and task classification module performs semantic analysis on user questions using BERT or LLM classifiers to identify the data types and task requirements involved in the questions, and uses the recognition results as the scheduling basis for the collaborative scheduling module.

[0009] Furthermore, the collaborative scheduling logic of the collaborative scheduling module is as follows: When the intent recognition result is a text-based task, the text content vectorization module, the cross-modal knowledge base retrieval module, and the text model answering module are scheduled, and the large language model is responsible for text understanding and answer generation. When the intent recognition result is an image-based task, the following modules are scheduled: multimodal structuring processing module, multimodal content vectorization module, cross-modal knowledge base retrieval module, multimodal large model answering module, and text model answering module. The text model answering module is responsible for image-structured text answering, and the multimodal large model is responsible for image-text question answering. When the intent recognition result is a cross-modal task, the text model response module and the multimodal large model response module are called in tandem to achieve parallel processing and result fusion of the multimodal large model and the large language model.

[0010] Furthermore, the file parsing and preprocessing module parses energy-related documents in PDF, Word, PPT, and Excel formats, extracting text, tables, images, and formulas. The image processing includes classifying images into six categories using an image classification model: general images, charts, flowcharts, operation diagrams, circuit diagrams, and schematic diagrams. After performing OCR text extraction on each type of image, the multimodal large model is called to output structured information.

[0011] Furthermore, the multimodal content vectorization module fuses and encodes the image ontology, image OCR text, image structured JSON, and image category tags to generate an embedding vector in a unified semantic space; the text content vectorization module encodes text paragraphs and user questions into semantic vectors; both types of vectors jointly support the cross-modal indexing and semantic matching of the cross-modal knowledge base retrieval module.

[0012] Furthermore, the cross-modal knowledge base retrieval module performs cross-modal semantic retrieval based on text vectors and image embedding vectors, supporting the following strategies: When a question is asked in text, relevant images and text information are retrieved based on text vectors. When querying images, images with similar structures and associated text are retrieved based on the image embedding vector; When asking questions involving both text and images, the system retrieves and matches mixed text and image information based on a hybrid vector; the retrieved results support rank sorting, multiple rounds of re-detection, and conditional filtering.

[0013] Furthermore, the multi-domain energy knowledge base ingestion module constructs a multimodal vector knowledge base, storing text vectors, image embedding vectors, and metadata. It supports indexing at the page, paragraph, and image block granularity, providing data support for cross-modal indexing.

[0014] Furthermore, the multimodal large model answering module calls the GPT-4V and Qwen-VL multimodal models to parse the images and structured information returned by the cross-modal knowledge base retrieval module and generate image-related question-and-answer results; the text model answering module calls the large language model to integrate the retrieved text and structured JSON and generate text-based question-and-answer results.

[0015] Furthermore, the answer integration and structure reference module receives the outputs of the multimodal large model answer module and the text model answer module, and generates the final answer by combining the conversation context stored in the memory module; when the answer involves an image, it includes the corresponding page number and image number; when it involves text, it includes the paragraph number and document identifier to ensure that the answer is traceable.

[0016] Furthermore, the memory module stores the user's historical questions, intermediate search results, and generated answers, providing contextual information only to the large language model to achieve semantic continuity and historical reference in multi-turn dialogues.

[0017] Beneficial effects: 1. Enhance multimodal data processing capabilities: Through image classification and targeted structured parsing, structured information such as node-edge format and Netlist format can be extracted from complex images such as charts, flowcharts, and circuit diagrams in energy field documents. This solves the problem that traditional technologies can only extract text from images and lose structural semantics, enabling deep parsing of multiple types of images.

[0018] 2. Achieve accurate cross-modal collaboration and question answering: Relying on the collaborative scheduling module, multimodal large models and large language models are dynamically scheduled according to user questions. Processing logic is adapted for image, text and cross-modal tasks respectively. Combined with cross-modal retrieval, it supports image and text joint question answering, which solves the limitation that a single model cannot take into account multiple types of data.

[0019] 3. Supports efficient cross-modal indexing and recall: By using unified semantic encoding to map text and images to the same vector space, cross-modal retrieval such as text query recall of images and image query recall of related text is realized, solving the problem that traditional indexes cannot associate information of different modalities and improving retrieval efficiency and accuracy.

[0020] 4. Ensure answers are traceable and verifiable: When integrating answers, include metadata such as image number and page number, and combine them with source identifiers stored in the knowledge base to ensure that answers can be traced back to the original document. This solves the problem of unclear sources of traditional question-and-answer results and enhances reliability in professional scenarios.

[0021] 5. Adapt to the professional needs of the energy sector: In response to the characteristics of the energy sector, which contains a large number of complex documents such as flowcharts and operation diagrams, the modular design supports distributed deployment, which can efficiently process batch document import and complex Q&A, and significantly improve the efficiency of knowledge query and decision support in the energy sector. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram illustrating the structure and function of the intelligent knowledge base question-answering system for the energy field based on multi-model collaboration, as described in an embodiment of the present invention. Figure 2 This is a PDF parsing flowchart of the intelligent knowledge base question-answering system for the energy field based on multi-model collaboration, as described in an embodiment of the present invention. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] Example 1 See Figure 1-2 A multi-model collaborative intelligent knowledge base question-answering system for the energy field comprises 13 core modules: a document parsing and preprocessing module, a text slicing and multimodal vectorization module, a multi-domain energy knowledge base input module, an intent recognition and task classification module, a collaborative scheduling module, a multimodal structured processing module, a multimodal content vectorization module, a text content vectorization module, a cross-modal knowledge base retrieval module, a memory module, a multimodal large-model answering module, a text model answering module, and an answer integration and structure referencing module. The collaborative scheduling module receives intent recognition data through a preset data interface. The system integrates the output of the task classification module to achieve collaborative scheduling of the multimodal large model and the large language model. Based on the data types involved in the question, it dynamically calls the corresponding modules to process multiple types of data (text, charts, flowcharts, operation diagrams, circuit diagrams, schematic diagrams). It also supports cross-modal indexing through unified semantic encoding of the multimodal content vectorization module and the text content vectorization module. All modules are interconnected through preset data interfaces and control signaling to form a complete closed loop of "document parsing - knowledge storage - intent recognition - collaborative scheduling - cross-modal retrieval - intelligent answering", which is suitable for distributed system deployment scenarios.

[0025] It should be noted that the text slicing and multimodal vectorization module in this embodiment can slice the text content in a document by paragraph or semantic unit, and call models such as CLIP and BLIP-Embedding to vectorize the text, images, image-text pairs and their structured information. Image vectorization not only includes the image content itself, but also combines image category, image OCR results, image structured JSON text, etc. for multi-dimensional encoding.

[0026] The multimodal structuring processing module in this embodiment: When the problem involves image content, this module performs image classification and content structure recognition on charts, flowcharts, operation diagrams, etc. Combining OCR, graph neural networks, and multimodal understanding models, it outputs standardized JSON structure results. The output structure for different graph types is as follows: Flowchart / Operation Diagram: Flow structure in node-edge format, including node type, text, and location information; Circuit diagram: Netlist format, consisting of components, pins, connecting wires, and electrical network structure; Diagram: A semantic structure consisting of components, directions, annotations, and spatial layout.

[0027] This embodiment constructs a complete closed-loop system that supports multimodal collaboration and distributed deployment.

[0028] Multi-module collaboration: Through 13 core modules, it forms a full-process processing capability of "document parsing → knowledge storage → intent recognition → collaborative scheduling → cross-modal retrieval → intelligent answering".

[0029] Unified semantic space: The multimodal content vectorization and text content vectorization modules enable cross-modal indexing and support mixed retrieval of text and images.

[0030] Distributed adaptation: Modules interact with each other through preset interfaces and signaling, naturally supporting distributed system deployment and strong scalability.

[0031] This embodiment implements an intelligent question-and-answer mode of "file-driven + multi-model collaboration" for energy technology documents. It supports automatic parsing and accurate answers to mixed text and graphics knowledge, which greatly improves the practicality, scalability and accuracy of complex document understanding and question-and-answer systems. It is particularly suitable for querying and assisting decision-making of engineering data in the energy field that includes flowcharts, charts and structural diagrams.

[0032] In a specific example, the intent recognition and task classification module performs semantic analysis on user questions using BERT or LLM classifiers to identify the data types (text, charts, flowcharts, operation diagrams, circuit diagrams, schematic diagrams) and task requirements (explanation, location, comparison, structural path extraction) involved in the question, and uses the recognition results as the scheduling basis for the collaborative scheduling module.

[0033] This embodiment accurately classifies tasks, improving system response efficiency.

[0034] Semantic analysis: Use BERT or LLM classifiers to perform deep semantic understanding of user questions, accurately identify data types (text / image / cross-modal) and task requirements (such as interpretation, comparison, prediction, etc.).

[0035] Scheduling basis: Provide clear decision input for the collaborative scheduling module, avoid resource waste, and optimize processing paths.

[0036] In a specific example, the collaborative scheduling logic of the collaborative scheduling module is as follows: When the intent recognition result is a text-based task, the text content vectorization module, the cross-modal knowledge base retrieval module, and the text model answering module are scheduled, and the large language model is responsible for text understanding and answer generation. When the intent recognition result is an image-based task, the following modules are scheduled: multimodal structuring processing module, multimodal content vectorization module, cross-modal knowledge base retrieval module, multimodal large model answering module, and text model answering module. The text model answering module is responsible for image-structured text answering, and the multimodal large model is responsible for image-text question answering. When the intent recognition result is a cross-modal task, the text model response module and the multimodal large model response module are called in tandem to achieve parallel processing and result fusion of the multimodal large model and the large language model.

[0037] This embodiment uses dynamic scheduling to achieve optimal collaboration among multiple models.

[0038] Task-driven scheduling: Text-based tasks: Focus on calling the text processing module to leverage the text understanding advantages of the large language model.

[0039] Image-related tasks: Enable multimodal large model and text processing modules to specifically handle image structured parsing and image-text question answering.

[0040] Cross-modal tasks: Parallel invocation of two types of modules to fuse and process mixed text and image information, thereby improving the ability to solve complex problems.

[0041] In a specific example, the file parsing and preprocessing module parses energy-related documents in PDF, Word, PPT, and Excel formats, extracting text, tables, images, and formulas. The image processing includes: classifying images into six categories using an image classification model: general images, charts, flowcharts, operation diagrams, circuit diagrams, and schematic diagrams; performing OCR text extraction on each type of image; and then calling a multimodal large model to output structured information (flowcharts output node-edge format JSON, circuit diagrams output Netlist format structures, etc.).

[0042] In the specific implementation, taking PDF parsing as an example (PDF parsing is the most complex among all file types), the specific parsing process is as follows: Figure 2 The process includes the following steps: (1) PDF page reading: Read the PDF and obtain all pages through the Pymupdf parsing engine API (Python); (2) PDF page image rotation detection: The image rotation detection model determines the page rotation and performs image correction if rotation is detected. (3) Remove watermarks / official seals from PDF page images: Official seal detection finds and removes all official seals on the page, official seal text recognition extracts the official seal text, and watermark detection finds and removes all watermarks on the page. (4) Page text extraction: Page type determination. If it is a scanned page, the text in the image is extracted by OCR; if it is a non-scanned page, the PDF page text is extracted directly by pymupdf. (5) Page layout analysis: Find all the category information on the page and its corresponding position through page layout analysis: title, paragraph, header and footer, table, image, formula, etc., and remove duplicates according to certain rules; (6) Table recognition: The rotation angle of the table image returned by the layout analysis in (5) is identified and extracted by the table rotation detection model, and the rotated table image is corrected by the CV method; the table structure is identified by the table structure recognition model; and the text obtained in (4) is matched (the rules are written according to the position) to generate the complete HTML data of the table. (7) Formula detection + formula recognition: Formula detection finds the formulas and their corresponding positions on the page, performs formula recognition on the returned formula images, and returns the formulas in LaTeX format; (8) Multimodal structured parsing: This step is the focus of this invention in order to obtain the key information contained in the images within documents in the energy field. This invention trains an image classification model with six categories: general images, charts, flowcharts, operation diagrams, circuit diagrams, and schematic diagrams. When parsing images in a document, the image classification model first obtains the specific category of the image. If the type is a general image, the text in the image is obtained through OCR, and the detailed descriptive text of the image is obtained through a multimodal large model. If the type is a chart, the text in the chart is obtained through OCR, and the corresponding table HTML description of the chart is obtained through a chart large model. If the type is a flowchart, the text in the image is obtained through OCR, and the flowchart is parsed into a graph-structured JSON format using a multimodal large model. If the type is an operation diagram, the text in the image is obtained through OCR, and the hierarchical graph JSON of the image is obtained through a multimodal large model. If the type is a schematic diagram, the text in the image is obtained through OCR, and the semantic graph structured JSON parsing output is obtained through a multimodal large model. If the type is a circuit diagram, the text in the image is obtained through OCR, and the structural information in JSON format with component-level connection information is obtained through a multimodal large model. (9) Page layout post-processing: Based on the classification results and position coordinates of the page layout analysis in (5) and the text and text position coordinates returned in (4), the text is positioned in the corresponding page layout category according to certain rules, and the correct order of the text in the paragraph is ensured. Then the paragraph is sorted. (10) Page merging: The sorting of all paragraphs on a page is achieved through certain rules, and the content of all pages in the PDF is integrated.

[0043] This embodiment is compatible with multiple formats and features sophisticated image classification and processing.

[0044] Document parsing capabilities: Supports common document formats in the energy field such as PDF, Word, PPT, and Excel, and comprehensively extracts text, tables, images, and formulas.

[0045] Fine-grained image classification: Images are divided into 6 categories (general images, charts, flowcharts, operation diagrams, circuit diagrams, and schematic diagrams), and customized OCR and structured parsing are performed for different types to improve the utilization rate of image information.

[0046] In a specific example, the multimodal content vectorization module fuses and encodes the image ontology, image OCR text, image structured JSON, and image category tags to generate an embedding vector in a unified semantic space; the text content vectorization module encodes text paragraphs and user questions into semantic vectors; the two types of vectors together support the cross-modal knowledge base retrieval module's cross-modal indexing and semantic matching.

[0047] It should be noted that the multimodal content vectorization module in this embodiment fuses and encodes the image ontology, structured JSON, image OCR text, and image category tags to generate an embedding vector with a unified semantic space. This vector serves as one of the primary keys for image question answering and also participates in semantic alignment with the text question vector. Specific format examples for each category are as follows: Flowchart: A flowchart modeling approach using a node-edge structure. Output includes node type (e.g., "Start", "Decision", "Operation"), node text, node coordinates, start and end nodes of edges, connection text, and directionality. Example: { "nodes": [{"id": "n1", "type": "Start", "text": "System Startup", "position": {"x": 100, "y": 50}}], "edges": [{"from": "n1", "to": "n2", "label": "Next step"}] } Operation diagram: Similar in structure to a flowchart, but with greater emphasis on the sequence of human-device interactions or panel operations. The structure includes interaction type (click, switch, input), operation target (button, control), control identifier, etc. Example: { "steps": [{"step": 1, "action": "Click", "target": "Start button", "position": {"x": 220,"y": 310}}] } Charts: The structure includes chart type, axis information, legend, data point coordinates, units, etc. Example: { "type": "Line chart", "xAxis": "Time", "yAxis": "Power (kW)", "series": [{"label": "Phase A", "values": [10, 15, 20]}]} General images (such as on-site photos, equipment scene images, etc.): Output is OCR text content + image description semantic tags + location annotations. Example: { "ocr_text": "Transformer Model: S11-500kVA", "tags": ["device", "tag identifier"], "regions": [{"x": 150, "y": 200, "w": 80, "h": 30}] } Schematic diagram: A semantic structure consisting of components, directions, annotations, and spatial layout, suitable for logical illustrations of control systems, energy structures, etc. Example: { "components": [{"id": "c1", "label": "Wind Turbine", "position": {"x":100, "y": 150}}, {"id": "c2", "label": "Substation", "position": {"x": 300, "y":150}}], "connections": [{"from": "c1", "to": "c2", "label": "power transmission"}] } Circuit diagram: Parsed into component nodes + connection edges + electrical attributes, the output includes structural information such as electrical component type, port number, connection relationship, voltage and current parameters, etc. Example: { "components": [ {"id": "R1", "type": "resistance", "value": "100Ω", "position": {"x": 100,"y": 120}}, {"id": "C1", "type": "capacitor", "value": "10μF", "position": {"x": 200,"y": 120}} ], "wires": [{"from": "R1", "to": "C1", "path": [[100,120], [200,120]]}] }

[0048] This embodiment features multimodal vector fusion, supporting cross-modal retrieval.

[0049] Unified encoding: The image ontology, OCR text, structured JSON, and category labels are fused and encoded to generate an embedding vector with a unified semantic space, breaking down the semantic gap between modalities.

[0050] Cross-modal matching: Similarity is calculated between text vectors and image vectors in the same space, supporting cross-modal application scenarios such as "text retrieval of images" or "image retrieval of text." In a specific example, the cross-modal knowledge base retrieval module performs cross-modal semantic retrieval based on text vectors and image embedding vectors, supporting the following strategies: When a question is asked in text, relevant images and text information are retrieved based on text vectors. When querying images, images with similar structures and associated text are retrieved based on the image embedding vector; When asking questions involving both text and images, the system retrieves and matches mixed text and image information based on a hybrid vector; the retrieved results support rank sorting, multiple rounds of re-detection, and conditional filtering.

[0051] This embodiment employs a flexible retrieval strategy to improve the relevance of answers.

[0052] Multimodal retrieval support: Text-based questions: Recall relevant images and text to aid in explanation.

[0053] Image-based questioning: Retrieves images with similar structures and associated text, supporting image understanding and analysis.

[0054] Image and text combined questioning: Hybrid vector retrieval of matched image and text information to handle complex scenarios.

[0055] Intelligent sorting and filtering: Supports rank sorting, multi-round re-examination and conditional filtering to improve search accuracy.

[0056] In a specific example, the multi-domain energy knowledge base ingestion module constructs a multimodal vector knowledge base, storing text vectors, image embedding vectors, and metadata (including image addresses, page numbers, and document identifiers). It supports indexing at the page, paragraph, and image block granularity, providing data support for cross-modal indexing.

[0057] It should be noted that the multi-domain energy knowledge base input module in this embodiment is responsible for building a multimodal vector knowledge base, supporting index building at the granularity of pages, paragraphs, and image blocks, and storing meta-information such as structured information, image addresses, related documents, and chart types, forming a composite knowledge base that supports semantic retrieval and structural referencing.

[0058] This embodiment features a fine-grained knowledge base, supporting precise retrieval.

[0059] Multimodal vector storage: Construct a knowledge base containing text vectors, image embedding vectors, and metadata.

[0060] Fine-grained indexing: Indexes are built by page, paragraph, and image block, supporting high-precision positioning, avoiding information redundancy, and improving retrieval efficiency.

[0061] In a specific example, the multimodal large model answering module calls the GPT-4V and Qwen-VL multimodal models to parse the images and structured information returned by the cross-modal knowledge base retrieval module and generate image-related question-and-answer results; the text model answering module calls the large language model to integrate the retrieved text and structured JSON and generate text-based question-and-answer results.

[0062] This embodiment features model adaptation and optimization to enhance professional question-and-answer capabilities.

[0063] Professional model call: Multimodal large models (GPT-4V, Qwen-VL): handle image-related problems, such as chart interpretation and device operation instructions.

[0064] Large language models: handle text-based problems, such as concept explanation and principle analysis.

[0065] Structured information utilization: The model directly processes the retrieved structured JSON, enhancing the understanding and expression of professional knowledge in the energy field.

[0066] In a specific example, the answer integration and structure reference module receives the outputs of the multimodal large model answer module and the text model answer module, and generates a final answer by combining the conversation context stored in the memory module. When the answer involves an image, it includes the corresponding page number and image number; when it involves text, it includes the paragraph number and document identifier to ensure that the answer is traceable.

[0067] The answer in this example is traceable, which enhances its credibility.

[0068] Multi-source integration: Integrates the outputs of multimodal large models and text models to generate more comprehensive answers.

[0069] Structured citations: The answers include page numbers for images, paragraph numbers for text, and document identifiers, supporting the tracing of the answer's source and enhancing its authority and credibility.

[0070] In a specific example, the memory module stores the user's historical questions, intermediate search results, and generated answers, providing contextual information only to the large language model to achieve semantic continuity and historical reference in multi-turn dialogues.

[0071] This embodiment is context-aware and supports coherent dialogue.

[0072] Conversation memory: Stores historical questions, search results, and generated answers, providing contextual information for large language models.

[0073] Multi-round dialogue: In consulting on complex issues in the energy sector, it supports semantic continuity and historical reference, enhancing user experience and the depth of problem-solving.

[0074] In summary, the system in this embodiment significantly improves the professionalism, accuracy, and interpretability of knowledge base question answering in the energy field through innovative designs such as multi-model collaboration, cross-modal processing, fine-grained indexing, and structured referencing. It is particularly suitable for processing energy technology documents, standards and specifications, and operation guides containing a large amount of mixed text and graphics information, providing energy field experts and practitioners with an efficient intelligent auxiliary tool.

[0075] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A question-answering system for an intelligent knowledge base in the energy field based on multi-model collaboration, characterized in that, It comprises 13 core modules: a file parsing and preprocessing module, a text slicing and multimodal vectorization module, a multi-domain energy knowledge base input module, an intent recognition and task classification module, a collaborative scheduling module, a multimodal structured processing module, a multimodal content vectorization module, a text content vectorization module, a cross-modal knowledge base retrieval module, a memory module, a multimodal large model answering module, a text model answering module, and an answer integration and structure referencing module. The collaborative scheduling module receives the output results of the intent recognition and task classification module through a preset data interface, realizing the collaborative scheduling of the multimodal large model and the large language model. It dynamically calls the corresponding module to process multiple types of data based on the data types involved in the question, and supports cross-modal indexing through the unified semantic encoding of the multimodal content vectorization module and the text content vectorization module. All modules are interconnected through preset data interfaces and control signaling, forming a complete closed loop of "document parsing - knowledge base input - intent recognition - collaborative scheduling - cross-modal retrieval - intelligent answering", which is suitable for distributed system deployment scenarios.

2. The system according to claim 1, characterized in that, The intent recognition and task classification module performs semantic analysis on user questions using BERT or LLM classifiers, identifies the data types and task requirements involved in the questions, and uses the recognition results as the scheduling basis for the collaborative scheduling module.

3. The system according to claim 1, characterized in that, The collaborative scheduling logic of the collaborative scheduling module is as follows: When the intent recognition result is a text-based task, the text content vectorization module, the cross-modal knowledge base retrieval module, and the text model answering module are scheduled, and the large language model is responsible for text understanding and answer generation. When the intent recognition result is an image-based task, the following modules are scheduled: multimodal structuring processing module, multimodal content vectorization module, cross-modal knowledge base retrieval module, multimodal large model answering module, and text model answering module. The text model answering module is responsible for image-structured text answering, and the multimodal large model is responsible for image-text question answering. When the intent recognition result is a cross-modal task, the text model response module and the multimodal large model response module are called in tandem to achieve parallel processing and result fusion of the multimodal large model and the large language model.

4. The system according to claim 1, characterized in that, The document parsing and preprocessing module parses energy-related documents in PDF, Word, PPT, and Excel formats, extracting text, tables, images, and formulas. The image processing includes classifying images into six categories using an image classification model: general images, charts, flowcharts, operation diagrams, circuit diagrams, and schematic diagrams. After performing OCR text extraction on each type of image, the module calls a multimodal large model to output structured information.

5. The system according to claim 1, characterized in that, The multimodal content vectorization module integrates and encodes the image ontology, image OCR text, image structured JSON, and image category tags to generate an embedding vector in a unified semantic space; the text content vectorization module encodes text paragraphs and user questions into semantic vectors; both types of vectors jointly support the cross-modal knowledge base retrieval module's cross-modal indexing and semantic matching.

6. The system according to claim 1, characterized in that, The cross-modal knowledge base retrieval module performs cross-modal semantic retrieval based on text vectors and image embedding vectors, and supports the following strategies: When a question is asked in text, relevant images and text information are retrieved based on text vectors. When querying images, images with similar structures and associated text are retrieved based on the image embedding vector; When asking questions involving both text and images, the system retrieves and matches mixed text and image information based on a hybrid vector; the retrieved results support rank sorting, multiple rounds of re-detection, and conditional filtering.

7. The system according to claim 1, characterized in that, The multi-domain energy knowledge base ingestion module constructs a multimodal vector knowledge base, storing text vectors, image embedding vectors, and metadata. It supports indexing at the page, paragraph, and image block granularity, providing data support for cross-modal indexing.

8. The system according to claim 1, characterized in that, The multimodal large model answering module calls the GPT-4V and Qwen-VL multimodal models to parse the images and structured information returned by the cross-modal knowledge base retrieval module and generate image-related question-and-answer results. The text model answering module calls the large language model to integrate the retrieved text and structured JSON to generate text-based question-and-answer results.

9. The system according to claim 1, characterized in that, The answer integration and structure reference module receives the outputs from the multimodal large model answer module and the text model answer module, and generates the final answer by combining the conversation context stored in the memory module. When the answer involves an image, it includes the corresponding page number and image number; when it involves text, it includes the paragraph number and document identifier to ensure that the answer is traceable.

10. The system according to claim 1, characterized in that, The memory module stores the user's historical questions, intermediate search results, and generated answers in the conversation, and only provides contextual information to the large language model to achieve semantic continuity and historical reference in multi-turn dialogues.

Images