Multi-dimensional knowledge base construction and query method based on multi-modal large model

Abstract

The invention relates to a multi-dimensional knowledge base construction and query method based on a multi-modal large model, and belongs to the technical field of AI large model application. The method comprises the following steps of: analyzing a text file containing text, table and picture contents; selecting a pre-trained large language model and a multi-modal large model, generating text abstracts and picture abstracts by utilizing the models, and enabling storage paths of the picture abstracts to correspond to storage paths of pictures one by one; selecting a text embedding model to construct a vector database and a retriever; and constructing a retrieval chain based on the multi-modal large model. According to the method, construction from a single text knowledge base to a multi-dimensional knowledge base is achieved, a multi-dimensional information retrieval method is provided, the problem of construction of the multi-dimensional knowledge base based on the vectorization technology is solved to a certain extent, and the large model output quality based on the retrieval enhancement generation technology is remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of AI large model application technology, specifically involving a method for constructing and querying a multi-dimensional knowledge base based on a multimodal large model. Background Technology

[0002] With the deep integration of the digital economy and artificial intelligence technology, multimodal data such as text, tables, formulas, and images are being used more and more widely in various professional fields such as electrical engineering, aerospace, financial analysis, education and scientific research. Building an intelligent knowledge base that supports multimodal data has become a core requirement for improving the efficiency of industry information processing and assisting in accurate decision-making. Although the relevant technologies have formed a certain foundation for development, there are still significant shortcomings in terms of adaptability to professional scenarios and depth of complex data processing.

[0003] Existing multimodal technologies exhibit a diversified development pattern, with different technical routes each having their own focus but also limitations: general-purpose multimodal data processing technologies, such as CLIP and ALBEF, have the core advantage of achieving cross-modal semantic alignment of text and images, meeting basic text and image retrieval and matching needs in general scenarios. However, they are not optimized for the structured features and domain knowledge characteristics of professional documents. For structured data such as electrical engineering parameter tables and financial analysis tables, they can only extract surface text and cannot parse row and column logic and numerical relationships; they also have difficulty recognizing the semantic connotations of professional symbols in images such as engineering drawings and professional formulas, resulting in incomplete knowledge extraction and a tendency for semantic bias; Adobe... Document-oriented multimodal parsing technologies, centered around Acrobat and LayoutLM models, excel at document layout analysis and precise segmentation of text blocks, table blocks, and image blocks, enabling basic structured transformation of multimodal content. However, they lack a search-oriented summary generation mechanism, failing to extract core information, leading to inefficient subsequent searches requiring traversal of complete data. Furthermore, they rely solely on layout position to establish simple associations, lacking semantic-level cross-modal deep mapping, making them prone to information fragmentation when facing cross-modal query needs. They also lack sufficient logical parsing capabilities for professional formulas and complex charts. LangChain multimodal... Retrieval Enhancement Generation (RAG) driven multimodal query technology, represented by the state framework and the Milvus+LLaVA joint solution, has opened up the "storage-retrieval-generation" link. It can match multimodal data based on user queries and generate natural language answers. However, the modality embedding strategy is singular, mostly adopting the indirect method of "image to text + text embedding", which does not give full play to the cross-modal embedding capability of the multimodal large model. The coupling design between the vector database and the retrieval chain is unreasonable, and there is a lack of dedicated index structure for structured data such as tables. It cannot meet the needs of refined query, and the generated results are not adapted to the structured output requirements of professional scenarios. Furthermore, while some customized multimodal technologies for fields such as medicine and law incorporate domain-specific knowledge and can identify special modal elements in professional scenarios, their applicability is narrow, their multimodal integration capabilities are limited, and their retrieval strategies are fixed, making them difficult to transfer to other professional fields. At the same time, existing technologies also suffer from common shortcomings: data preprocessing uses general algorithms without optimization for professional data characteristics, easily leading to the loss of key details; knowledge integration lacks consistency verification mechanisms, making it impossible to identify multimodal information conflicts; retrieval interaction supports single-modal input, making it difficult to adapt to mixed and fuzzy query needs; and the system has poor scalability, with high maintenance costs when adding modalities or switching domains. These problems result in significant deficiencies in the accuracy of professional document parsing, the depth of multimodal knowledge association, and the collaborative efficiency of retrieval and generation, failing to meet the needs of relevant fields for efficient integration and accurate querying of multi-dimensional information. An integrated approach is urgently needed to overcome these difficulties. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies and provide a method for constructing and querying a multi-dimensional knowledge base based on a multimodal large model.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A method for constructing and querying a multi-dimensional knowledge base based on a multimodal large model includes the following steps: Step S1: Parse the document file, extract text blocks, table structure data and images from the document file, save the images to the specified directory and obtain an ordered list of image paths; Step S2: Select a pre-trained large text model, a large multimodal model that supports cross-modal understanding of text and images, and a text embedding model, and configure the API keys required for model calls; Using a large text model, concise summaries adapted to retrieval scenarios are generated in batches from extracted text blocks and table structure data. A multimodal large model is used to generate a retrieval-style image summary for each image in the image path list, and a one-to-one correspondence between the image summary and the corresponding image storage path is established, forming a set of text table summaries and a set of image summaries. Step S3: Based on the selected text embedding model, text table summary set, image summary set, document file parsing results, and image path list in S1, construct a vector database, a memory storage module, and a multi-vector searcher; wherein, the vector database adopts the Chroma database and specifies a unique set name; the memory storage module is used to store the original document elements; the multi-vector searcher associates the vector database and the memory storage module through a unique identifier and configures the parameter of the number of search results returned; Step S4: Construct a retrieval chain based on a multimodal large model. This retrieval chain includes a question delivery module, a context processing module, a prompt word generation module, a multimodal large model reasoning module, and an output parsing module. Based on the relevant results returned by the multi-vector retrieval machine and the user query, generate a structured answer containing image reference tags.

[0006] Furthermore, preferably, in step S1, the document file contains at least one of text, tables, and images; the document file includes text files and PDF files.

[0007] Furthermore, preferably, the specific method of step S1 is as follows: Step S11: Obtain the storage path of the document file and determine the specified target directory for saving the parsed and extracted images; Step S12: Call a document parsing tool that supports image and text extraction and table structure recognition to perform parsing operations on the document file, enable the image extraction function to extract image elements in the file, enable the table structure recognition function to obtain structured table data; at the same time, set the text segmentation parameters, and use the method of segmenting by document title level to segment the text content into blocks to obtain text blocks; Step S13: Save the image elements extracted during the parsing process to the specified directory determined in step S11, and integrate the segmented text blocks, structured table data and image elements with storage paths to form a document parsing result set containing multiple types of elements. Step S14: Traverse the specified target directory where the images are saved, filter out valid image files with the extensions `.jpg`, `.jpeg`, and `.png`, and organize them into an ordered list of image paths according to the order in which the images appear in the original document.

[0008] Furthermore, preferably, in step S12, the text segmentation parameters are set as follows: a maximum number of characters in a single text block of 4000, a character threshold of 3800 for triggering the segmentation of a new text block, and a character threshold of 2000 for merging small text blocks.

[0009] Furthermore, preferably, the specific method of step S2 is as follows: Step S21: Initialize the selected large text model, multimodal large model, and text embedding model, and configure the DashScope API key to enable normal model calls; Step S22: Construct a text table summary generation chain, which includes a document parameter mapping module, a prompt word template module, a text large model inference module, and an output parsing module connected in sequence; The document parameter mapping module is used to ensure that images have a correct correspondence with their summaries; The prompt word template module is used to constrain the content generated by large AI models; The text-based large model inference module is used to enhance the content generation of the large model by retrieving content based on user-input keywords; The output parsing module is used to convert the content generated by the large model and its corresponding images into HTML format for output. The prompt word template module explicitly instructs users to generate concise summaries suitable for retrieval, which must include the core entities, logical relationships, and key data of the text and tables to support accurate retrieval of the original text and table elements. Step S23: Input the document parsing result set generated in Step S1 into the text table summary generation chain, generate text table summaries in batch processing mode with maximum concurrency, and add each text table summary to the summary list to form a text table summary set; for each image in the image path list, first pass the path validity check, and generate a local image URI for valid paths that pass the check; Step S24: Input the local image URI and text command together into the multimodal large model to obtain the image summary output by the model; the text command is to generate a concise image summary optimized for retrieval, which must include the core content of the image, key features and contextual association information with the original document; Step S25: Perform format validation on the output of the multimodal large model: if the output is a list containing the `text` key, extract the value corresponding to the `text` key; if the output is a string, use it directly as an image summary; if the output format is unknown or invalid, mark it as unable to generate an image summary; finally, form a set of image summaries that correspond one-to-one with the list of image paths, ensuring that each image path is associated with a unique image summary.

[0010] Furthermore, preferably, in step S21, the large text model adopts the `QWEN_MAX_MODEL` model; the multimodal large model adopts the `QWEN_VL_MAX_MODEL` model; and the text embedding model adopts the `TEXT_EMBEDDING_MODEL` model.

[0011] Furthermore, preferably, in step S23, the path validity verification includes verifying the existence of the path, the file type, and accessibility, that is, whether the image exists under each path; the URI format is `file: / / +absolute path of the image`.

[0012] Furthermore, preferably, the specific method of step S3 is as follows: Step S31: Input the text embedding model, the text table summary set, the document parsing result set, and the specified Chroma database set name, and perform the following operations: (1) Create a Chroma vector database instance corresponding to the name of the collection to persistently store the vector representation of the text table summary after being transformed by the text embedding model; (2) Create a memory storage module to store the original document elements in the document parsing result set; the original document elements include text blocks, table structure data, and image storage paths; (3) Generate a UUID for each original document element as a unique identifier, construct a text table summary-UUID data pair, convert the text table summary into a high-dimensional vector through a text embedding model, and store the high-dimensional vector content in the Chroma vector database. (4) Establish a mapping relationship between UUID and original document element, and store the mapping relationship in batches to the memory storage module; Step S32: Call the document addition method again, input the text embedding model, image summary set, image list and the name of the same Chroma database set, repeat the operations of (1)-(4) in step S31, complete the storage and association of image summary vector and image list, and further strengthen the one-to-one correspondence between image summary and image storage path; Step S33: Based on the created Chroma vector database, memory storage module, and UUID, construct a multi-vector retrieval tool; this multi-vector retrieval tool supports matching user queries with summary vectors through vector similarity algorithms, and then obtaining the corresponding original document elements or image lists through UUID association.

[0013] Furthermore, preferably, the specific method of step S4 is as follows: Step S41: Construct a context processing module. The function of this module is to call the multi-vector retrieval tool constructed in step S3 to obtain the original results related to the user query, and to construct an image text type splitting function to separate the text content and image path in the original results. After removing invalid or duplicate information, it is constructed into a list of text strings containing tags to form structured context information. Step S42: Construct a prompt word generation module. This module receives user queries and structured context information, and integrates them into structured prompt words according to a preset template. Step S43: Construct a retrieval chain based on a multimodal large model. The execution order of the chain structure is: question propagation, context processing, prompt word generation, multimodal large model reasoning, and output parsing. Step S44: Input the user query into the search chain, execute the search-inference process, and finally generate the query results in Markdown format, where images are represented by `` Use image tags to reference images, ensuring that the image reference location logically matches the answer content.

[0014] Furthermore, preferably, in step S42, the structured prompt explicitly instructs that a comprehensive and professional answer be generated by combining the provided text content and image information, the answer must be logically coherent, and the image references must be marked with corresponding image tags to ensure matching of text and image information; The retrieval-reasoning process is as follows: a multi-vector retrieval module obtains relevant results; a context processing module organizes information; a prompt word generation module constructs prompt words; a multimodal large model generates answers; and an output parsing module processes the format.

[0015] In step S33 of this invention, the number of search results returned is set as a parameter (k=7). This search engine supports matching user queries with summary vectors through vector similarity algorithms (such as cosine similarity), and then obtains the corresponding original document elements or image lists through UUID association.

[0016] In step S43 of this invention, a retrieval chain based on a multimodal large model is constructed. The execution order of the chain structure is as follows: Question transmission module (receives user queries and transmits them to the context processing module) → Context processing module (outputs structured context information) → Prompt word generation module (outputs structured prompt words) → Multimodal large model reasoning module (generates answers based on prompt words) → Output parsing module (converts the model output to Str format to ensure format compliance). In step S44 of this invention, the user query is input into the retrieval chain, and the "retrieval-reasoning" process is executed: the multi-vector retrieval tool obtains relevant results → the context processing module organizes information → the prompt word generation module constructs prompt words → the multimodal large model generates answers → the output parsing module processes the format; finally, a Markdown format query result is generated, where images are in `` format. Use image tags to reference images, ensuring that the image reference location logically matches the answer content.

[0017] To address the issues of incomplete data integration, poor query efficiency, and low structuring in current multimodal knowledge base construction, as well as the demand for precise multi-dimensional information retrieval in professional fields, this invention proposes a multi-dimensional knowledge base construction and query method based on a multimodal large model. This method integrates efficient multimodal data parsing, retrieval-based summary generation, vectorized storage, and intelligent retrieval. It simultaneously processes text, tables, and image data from documents, optimizes retrieval-based summary generation, establishes a precise correlation between the summary and the original data, strengthens the coupling between the vector database and the multimodal retrieval chain, and overcomes the pain points of incomplete multimodal data coverage, information matching deviations, and inefficient query responses in existing technologies. This provides a comprehensive, accurate, and standardized knowledge base construction and query solution for professional fields such as electrical engineering and document analysis.

[0018] Compared with the prior art, the beneficial effects of this invention are as follows: 1. More comprehensive multimodal data integration: Compared with traditional solutions that only process single text data, this invention can simultaneously parse text, tables and images in a document. By extracting table data in a structured manner and associating image storage paths, it can achieve integrated integration of multiple types of data, solving the problem of incomplete multimodal data coverage in existing technologies.

[0019] 2. Significantly improved search accuracy: Unlike existing multimodal solutions that lack optimized summary generation, this invention designs search-oriented summaries for text tables and images for search scenarios. It establishes a precise association between the summary and the original data by combining unique identifiers (UUIDs), and uses similarity retrieval from the Chroma vector database to greatly reduce information matching bias and improve the accuracy of search results.

[0020] 3. Higher query efficiency and structured results: Existing vector databases have weak coupling with the retrieval chain. This invention achieves efficient "retrieval-reasoning-output" flow by deeply coupling a multi-vector retrieval device with a multi-modal retrieval chain, integrating context processing, prompt word optimization, and format parsing modules. It also generates structured results in Markdown format with image references, meeting the needs of professional fields for query efficiency and result standardization. Attached Figure Description

[0021] Figure 1 This is an overview diagram of the overall process of constructing and querying a multi-dimensional knowledge base based on a multimodal large model, used to illustrate the full-link execution logic of this method from startup to output results; Figure 2 This is a flowchart for parsing PDF files, used to illustrate the steps and organization logic for extracting text, tables, and image elements during the document parsing process. Figure 3 A flowchart for multimodal summarization is provided to illustrate the parallel generation process of text / table summaries and image summaries, as well as the accuracy assurance mechanism. Figure 4 A flowchart for constructing a vector database and a search engine is provided to illustrate how the summary vectors are associated with the original document elements and the construction logic of the search engine. Figure 5 Build and query flowcharts for RAGs to depict the modular composition, interaction relationships, and execution flow of user queries in the multimodal RAG retrieval chain; Figure 6 This is a key data flow diagram, used to illustrate the entire data transmission path and relationships of multimodal data from parsing to query result output; Figure 7 Screenshot of the answer provided on an existing general platform; Figure 8 The image shows the effect of using a multi-dimensional knowledge base construction and query method based on a multimodal large model to answer questions. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the embodiments.

[0023] Those skilled in the art will understand that the following embodiments are for illustrative purposes only and should not be construed as limiting the scope of the invention. Where specific techniques or conditions are not specified in the embodiments, they are performed in accordance with the techniques or conditions described in the literature in the field or according to the product instructions. Materials or equipment whose manufacturers are not specified are all conventional products that can be obtained by purchase.

[0024] Example 1 A method for constructing and querying a multi-dimensional knowledge base based on a multimodal large model includes the following steps: Step S1: Parse the document file, extract text blocks, table structure data and images from the document file, save the images to the specified directory and obtain an ordered list of image paths; Step S2: Select a pre-trained large text model, a large multimodal model that supports cross-modal understanding of text and images, and a text embedding model, and configure the API keys required for model calls; Using a large text model, concise summaries adapted to retrieval scenarios are generated in batches from extracted text blocks and table structure data. A multimodal large model is used to generate a retrieval-style image summary for each image in the image path list, and a one-to-one correspondence between the image summary and the corresponding image storage path is established, forming a set of text table summaries and a set of image summaries. Step S3: Based on the selected text embedding model, text table summary set, image summary set, document file parsing results, and image path list in S1, construct a vector database, a memory storage module, and a multi-vector searcher; wherein, the vector database adopts the Chroma database and specifies a unique set name; the memory storage module is used to store the original document elements; the multi-vector searcher associates the vector database and the memory storage module through a unique identifier and configures the parameter of the number of search results returned; Step S4: Construct a retrieval chain based on a multimodal large model. This retrieval chain includes a question delivery module, a context processing module, a prompt word generation module, a multimodal large model reasoning module, and an output parsing module. Based on the relevant results returned by the multi-vector retrieval machine and the user query, generate a structured answer containing image reference tags.

[0025] Example 2 like Figures 1-6 As shown, the method for constructing and querying a multi-dimensional knowledge base based on a multimodal large model includes the following steps: Step S1: Parse the document file, extract text blocks, table structure data and images from the document file, save the images to the specified directory and obtain an ordered list of image paths; Step S2: Select a pre-trained large text model, a large multimodal model that supports cross-modal understanding of text and images, and a text embedding model, and configure the API keys required for model calls; Using a large text model, concise summaries adapted to retrieval scenarios are generated in batches from extracted text blocks and table structure data. A multimodal large model is used to generate a retrieval-style image summary for each image in the image path list, and a one-to-one correspondence between the image summary and the corresponding image storage path is established, forming a set of text table summaries and a set of image summaries. Step S3: Based on the selected text embedding model, text table summary set, image summary set, document file parsing results, and image path list in S1, construct a vector database, a memory storage module, and a multi-vector searcher; wherein, the vector database adopts the Chroma database and specifies a unique set name; the memory storage module is used to store the original document elements; the multi-vector searcher associates the vector database and the memory storage module through a unique identifier and configures the parameter of the number of search results returned; Step S4: Construct a retrieval chain based on a multimodal large model. This retrieval chain includes a question delivery module, a context processing module, a prompt word generation module, a multimodal large model reasoning module, and an output parsing module. Based on the relevant results returned by the multi-vector retrieval machine and the user query, generate a structured answer containing image reference tags.

[0026] In step S1, the document file contains at least one of text, tables, and images; the document file includes text files and PDF files.

[0027] The specific method for step S1 is as follows: Step S11: Obtain the storage path of the document file and determine the specified target directory for saving the parsed and extracted images; Step S12: Call a document parsing tool that supports image and text extraction and table structure recognition to perform parsing operations on the document file, enable the image extraction function to extract image elements in the file, enable the table structure recognition function to obtain structured table data; at the same time, set the text segmentation parameters, and use the method of segmenting by document title level to segment the text content into blocks to obtain text blocks; Step S13: Save the image elements extracted during the parsing process to the specified directory determined in step S11, and integrate the segmented text blocks, structured table data and image elements with storage paths to form a document parsing result set containing multiple types of elements. Step S14: Traverse the specified target directory where the images are saved, filter out valid image files with the extensions `.jpg`, `.jpeg`, and `.png`, and organize them into an ordered list of image paths according to the order in which the images appear in the original document.

[0028] In step S12, the text segmentation parameters are set as follows: maximum number of characters in a single text block is 4000, the threshold for triggering the segmentation of a new text block is 3800, and the threshold for merging small text blocks is 2000.

[0029] The specific method for step S2 is as follows: Step S21: Initialize the selected large text model, multimodal large model, and text embedding model, and configure the DashScope API key to enable normal model calls; Step S22: Construct a text table summary generation chain, which includes a document parameter mapping module, a prompt word template module, a text large model inference module, and an output parsing module connected in sequence; The document parameter mapping module is used to ensure that images have a correct correspondence with their summaries; The prompt word template module is used to constrain the content generated by large AI models; The text-based large model inference module is used to enhance the content generation of the large model by retrieving content based on user-input keywords; The output parsing module is used to convert the content generated by the large model and its corresponding images into HTML format for output. The prompt word template module explicitly instructs users to generate concise summaries suitable for retrieval, which must include the core entities, logical relationships, and key data of the text and tables to support accurate retrieval of the original text and table elements. Step S23: Input the document parsing result set generated in Step S1 into the text table summary generation chain, generate text table summaries in batch processing mode with maximum concurrency, and add each text table summary to the summary list to form a text table summary set; for each image in the image path list, first pass the path validity check, and generate a local image URI for valid paths that pass the check; Step S24: Input the local image URI and text command together into the multimodal large model to obtain the image summary output by the model; the text command is to generate a concise image summary optimized for retrieval, which must include the core content of the image, key features and contextual association information with the original document; Step S25: Perform format validation on the output of the multimodal large model: if the output is a list containing the `text` key, extract the value corresponding to the `text` key; if the output is a string, use it directly as an image summary; if the output format is unknown or invalid, mark it as unable to generate an image summary; finally, form a set of image summaries that correspond one-to-one with the list of image paths, ensuring that each image path is associated with a unique image summary.

[0030] In step S21, the large text model uses the `QWEN_MAX_MODEL` model; the multimodal large model uses the `QWEN_VL_MAX_MODEL` model; and the text embedding model uses the `TEXT_EMBEDDING_MODEL` model.

[0031] In step S23, the path validity verification includes verifying the existence of the path, the file type, and accessibility, that is, whether the image under each path exists; the URI format is `file: / / +absolute path of the image`.

[0032] The specific method for step S3 is as follows: Step S31: Input the text embedding model, the text table summary set, the document parsing result set, and the specified Chroma database set name, and perform the following operations: (1) Create a Chroma vector database instance corresponding to the name of the collection to persistently store the vector representation of the text table summary after being transformed by the text embedding model; (2) Create a memory storage module to store the original document elements in the document parsing result set; the original document elements include text blocks, table structure data, and image storage paths; (3) Generate a UUID for each original document element as a unique identifier, construct a text table summary-UUID data pair, convert the text table summary into a high-dimensional vector through a text embedding model, and store the high-dimensional vector content in the Chroma vector database. (4) Establish a mapping relationship between UUID and original document element, and store the mapping relationship in batches to the memory storage module; Step S32: Call the document addition method again, input the text embedding model, image summary set, image list and the name of the same Chroma database set, repeat the operations of (1)-(4) in step S31, complete the storage and association of image summary vector and image list, and further strengthen the one-to-one correspondence between image summary and image storage path; Step S33: Based on the created Chroma vector database, memory storage module, and UUID, construct a multi-vector retrieval tool; this multi-vector retrieval tool supports matching user queries with summary vectors through vector similarity algorithms, and then obtaining the corresponding original document elements or image lists through UUID association.

[0033] The specific method for step S4 is as follows: Step S41: Construct a context processing module. The function of this module is to call the multi-vector retrieval tool constructed in step S3 to obtain the original results related to the user query, and to construct an image text type splitting function to separate the text content and image path in the original results. After removing invalid or duplicate information, it is constructed into a list of text strings containing tags to form structured context information. Step S42: Construct a prompt word generation module. This module receives user queries and structured context information, and integrates them into structured prompt words according to a preset template. Step S43: Construct a retrieval chain based on a multimodal large model. The execution order of the chain structure is: question propagation, context processing, prompt word generation, multimodal large model reasoning, and output parsing. Step S44: Input the user query into the search chain, execute the search-inference process, and finally generate the query results in Markdown format, where images are represented by `` Use image tags to reference images, ensuring that the image reference location logically matches the answer content.

[0034] In step S42, the structured prompts clearly indicate the following instructions: combine the provided text content and image information to generate a comprehensive and professional answer. The answer must be logically coherent, and the image references must be marked with corresponding image tags to ensure that the text and image information match. The retrieval-reasoning process is as follows: a multi-vector retrieval module obtains relevant results; a context processing module organizes information; a prompt word generation module constructs prompt words; a multimodal large model generates answers; and an output parsing module processes the format.

[0035] Application Examples like Figures 1-6 As shown, the method for constructing and querying a multi-dimensional knowledge base based on a multimodal large model includes the following steps: Step S1: Use the Poppler tool to parse document files (including but not limited to text files and PDF files) containing text, tables, and images, extract text blocks, table structure data, and images, save the images to the specified directory, and obtain an ordered list of image paths; Step S2: Select a pre-trained text big data model, a multimodal big data model that supports cross-modal understanding of text and images, and a text embedding model. Configure the API key required for model calls. Use the text big data model to generate concise summaries adapted to the retrieval scenario in batches from the extracted text blocks and table structure data. Use the multimodal big data model to generate retrieval-style image summaries for each image in the image path list. Establish a one-to-one correspondence between image summaries and corresponding image storage paths to form a set of text table summaries and a set of image summaries. Step S3: Based on the selected text embedding model, text table summary set, image summary set, document file parsing results and image list, construct a vector database, a memory storage module and a multi-vector searcher. The vector database adopts the Chroma database and specifies a unique set name. The memory storage module is used to store the original document elements. The multi-vector searcher associates the vector database and the memory storage module with a unique identifier and configures the number of search results returned. Step S4: Construct a retrieval chain based on a multimodal large model. This retrieval chain includes a question delivery module, a context processing module, a prompt word generation module, a multimodal large model reasoning module, and an output parsing module. Based on the relevant results returned by the multi-vector retrieval machine and the user query, generate a structured answer containing image reference tags.

[0036] The specific implementation method of step S1: Step S11: In this embodiment, the system path is first initialized and the resource directory is configured to ensure that the input and output paths during the document parsing process are controllable and reproducible. The storage location of the document to be processed is determined by reading preset configuration parameters. This path is dynamically generated based on the project root directory, ensuring that the system can accurately locate the target file in different deployment environments. Simultaneously, an output storage directory for image elements is specified to centrally store the image content extracted from the document. The system automatically detects whether this directory exists; if it does not exist, the corresponding folder is created to ensure that the image data has a legitimate and stable persistent storage path.

[0037] Step S12: Document Parsing and Text Structured Chunking. A document parsing engine supporting multimodal content recognition is invoked to perform deep parsing of the PDF file, accurately extracting text, tables, and image elements, and performing semantically coherent chunking processing on the text content. Image extraction is enabled during parsing to ensure that embedded images are completely captured; simultaneously, table structure recognition is enabled to restore the table's hierarchical structure, header information, row and column relationships, and cell content mapping, generating structured table data.

[0038] For text segmentation, a semantic structure-based segmentation strategy is adopted, dividing the text according to the document heading hierarchy (such as first-level headings, second-level headings) to ensure that the generated text blocks are consistent with the chapter logic of the original document. Specific parameters are set as follows: the maximum number of characters in a single text block is 4000 to prevent excessively large blocks from affecting subsequent processing efficiency; when the cumulative character count reaches 3800, a new block is triggered to preserve semantic integrity; adjacent small text blocks with a length of less than 2000 characters are merged to avoid fragmented information. To ensure the accuracy of image extraction and page layout analysis, external toolsets are used to achieve high-fidelity image rendering and spatial positioning, improving the accuracy of multimodal element parsing.

[0039] Step S13: Integration of Parsing Results and Generation of Multimodal Element Set. After extracting multimodal elements, the system organizes and associates various parsing results to form a structured intermediate data representation. Extracted image elements are named according to preset rules and saved sequentially to a specified directory. The naming convention reflects their order of appearance in the original document, ensuring spatial consistency. The segmented text paragraphs, structured table data, and image elements with storage paths are integrated into a unified data set. The metadata of each image element includes its absolute or relative path reference in the storage directory, effectively binding content to its physical location and providing complete contextual information for subsequent processing.

[0040] Step S14: To support subsequent batch summarization and retrieval association of image content, a path index list that strictly corresponds to the order in which images appear in the document needs to be constructed. This is achieved by the system traversing the image output directory and filtering valid image files using file extensions, retaining only common image formats (including .jpg, .jpeg, and .png) and excluding temporary files or other non-image file types. The filtered image files are then sorted naturally by name to ensure that the order of the generated path list is completely consistent with the layout order of the images in the original document, thus guaranteeing the temporal correspondence between visual information and the document context.

[0041] Specific implementation method of step S2 Step S21: In this embodiment, the initialization and access permission configuration of various models required for multimodal processing are first completed. The system loads the large text language model, the multimodal vision-language model, and the text embedding model, and injects the corresponding API keys through configuration parameters to ensure secure calls to model services. The large text language model uses the QWEN_MAX_MODEL model to process plain text content and generate summaries; the multimodal large model uses the QWEN_VL_MAX_MODEL model, which has image-text joint understanding capabilities and is used for semantic parsing and summarization of image content; the text embedding model uses TEXT_EMBEDDING_MODEL for subsequent vectorization operations. The initialization of the above models is completed when summarization generation starts, ensuring that subsequent processing chains can stably call the corresponding model services.

[0042] Step S22: Construct a text and table summary generation chain. A summary generation processing chain for text and table content is constructed. This chain consists of multiple interconnected functional modules, enabling automated generation from raw content to search-optimized summaries. The processing chain includes the following components: (1) Document parameter mapping module: Extracts the text content of the input document elements as the input source for summary generation; (2) Prompt word template module: It uses predefined prompt word templates to clearly indicate the model role and task objectives. Its core instruction is: "You are an assistant responsible for generating table and text summaries for retrieval. These summaries will be embedded and used to retrieve original text or table elements. Please provide a concise summary of the table or text that has been optimized for retrieval." (3) Large Language Model Inference Module: Input the constructed prompt words into the initialized QWEN_MAX_MODEL model and perform the summary inference task; (4) Output parsing module: The original response returned by the docking model is formatted and the summary results in plain text form are extracted to ensure that the output content is clearly structured and can be directly used for subsequent processing.

[0043] Step S23: Batch generation of text and table summaries. The document parsing result set (containing segmented text paragraphs and structured table data) output from Step S1 is input into the summary generation chain constructed in Step S22 to perform batch summary generation. The system adopts a concurrent processing mechanism, with a maximum concurrency of 5, to improve processing efficiency and control resource consumption. All elements to be summarized are processed in parallel through batch calls to generate a unified set of text and table summaries. If an exception occurs during processing, the system has an error capture and fault tolerance mechanism, recording exception information and returning the corresponding status to ensure the robustness of the overall process.

[0044] Step S24: Image Path Validation and Local URI Generation. Each item in the ordered image path list generated in Step S1 is validated for validity. Validation includes: whether the file path exists, whether the file format is a supported image type (e.g., .jpg, .jpeg, .png), and whether the file can be read correctly. Only valid paths that pass validation are processed. For image paths that meet the criteria, they are converted into standard local resource URIs, in the format of a file: / / prefix followed by the absolute path of the image, ensuring compliance with the multimodal model's input resource format requirements. This process guarantees that all images processed are accessible and format compliant.

[0045] Step S25: Generate Image Summary. The validated local image URI and preset text instructions are input into the QWEN_VL_MAX_MODEL multimodal large model to generate an image summary optimized for retrieval. The input instructions explicitly require the model to generate concise, semantically rich summary content, covering the core objects of the image, key visual features, and their contextual information in the original document. The system performs summary generation on all valid images one by one, ensuring that each image obtains a corresponding semantic description, forming a complete image summary sequence.

[0046] The image summary results returned by the multimodal model undergo format consistency verification and cleaning. If the output is structured data (such as a dictionary or list) containing a "text" field, its text content is extracted as a valid summary; if the output is a plain string, it is directly retained; if the output is empty, has an abnormal format, or fails to parse, it is marked as "Unable to generate image summary" to prevent invalid data from entering subsequent processes. After verification, the system constructs an image summary set that strictly corresponds to the original image path list, maintaining a one-to-one mapping between paths and summaries. The final output consists of a structured data unit composed of the image summary set and the original image path list, for use by the subsequent vectorization and indexing modules, ensuring the semantic integrity and traceability of the multimodal data.

[0047] The specific implementation of step S3. Step S3 is used to construct a unified vector database and a multi-vector search engine, enabling vectorized storage and efficient retrieval of multimodal document elements such as text, tables, and images. Specifically, it includes the following steps: Step S31: Process the text and table summaries and their original document elements. The system calls the general document addition method, with input parameters including: text embedding model, text and table summary set, document parsing result set (containing original elements such as chunked text, structured tables and related paths), and preset vector database set name.

[0048] The specific execution process is as follows: (1) Create a persistent Chroma vector database instance, identified by the specified collection name, to store high-dimensional vectors generated by the text embedding model; (2) Initialize a memory storage module to cache original document elements, including text paragraphs, structured table data and image metadata, to ensure that the original content can be accessed quickly; (3) Generate a globally unique identifier (UUID) for each original element in the document parsing result set to form a unified list of document IDs; (4) Construct each text or table summary into a logical document object, the content of which is the summary text, and the corresponding UUID is recorded in the metadata, forming a “summary-UUID” binding relationship; (5) Convert all summaries into high-dimensional vectors using a text embedding model and write them in batches to the Chroma vector database; (6) Establish a mapping relationship between “UUID-original document element” in the memory storage module to realize the bidirectional association between the vector and the original content.

[0049] At this point, the vectors of the text and table summaries have been stored in the vector database, while the original content can be accurately retrieved in memory using the UUID.

[0050] Step S32: Process the image summary and its corresponding image list. After completing the text and table processing, the system calls the same document addition method again to vectorize and store the image summary. The input parameters are updated to: the same text embedding model, the image summary set, the original image path list, and the same vector database set name as in step S31.

[0051] The execution process is consistent with step S31, and specifically includes: (1) Reuse the existing Chroma vector database collection and append the vector data of the image summary to it; (2) Generate a unique UUID for each image and construct an "image summary - UUID" data pair; (3) The image summary is converted into a high-dimensional vector through a text embedding model and stored in the same vector database set; (4) Establish a mapping relationship between "UUID and image path" in the memory storage module to realize reverse lookup of image resources.

[0052] This step achieves decoupled storage and unified indexing of image summary vectors and original image paths, forming a chained access mechanism of "summary vector → UUID → image path", ensuring that subsequent searches can accurately locate the original image resources.

[0053] Step S33: Construct a multi-vector search engine. Based on the shared components built in the first two steps—a unified Chroma vector database instance, an in-memory storage module, and a unified UUID identifier key name—the system constructs a multi-vector search engine to support joint retrieval of cross-modal content.

[0054] The key configurations for this search engine are as follows: (1) The vector matching algorithm uses cosine similarity, which is supported by Chroma and is used to measure the semantic similarity between the query vector and the summary vector; (2) Set the number of search results returned parameter k=7, that is, each query returns the top 7 most relevant results; (3) The retrieval process is as follows: the user query is converted into a vector through the same text embedding model, an approximate nearest neighbor search is performed in the vector database to obtain the top-k most similar summary vectors and their metadata UUIDs, and then the corresponding original document elements or image paths are extracted from the memory storage module through the UUIDs.

[0055] This multi-vector search engine enables unified indexing and efficient retrieval of multimodal content such as text, tables, and images, and supports accurate retrieval based on semantic similarity.

[0056] Step S4 includes the following specific sub-steps: Step S41: Construct a context processing module to perform structured parsing of the original search results. This module is responsible for structured parsing and cleaning of the search results. The system calls the multi-vector searcher constructed in step S3 to obtain relevant results based on the user query. The results contain text fragments and image path information. By calling the image and text type separation function, the text content and image paths in the original search results are classified and processed to form independent data streams. Subsequently, deduplication and validity checks are performed to remove duplicate entries or invalid paths (such as image paths where the file does not exist), and finally, structured context information is output. This information is organized in dictionary form and contains two fields: "texts" (a list of text content) and "images" (a list of valid image paths), which serve as the basic input for subsequent prompt word generation.

[0057] Step S42: Construct a prompt word generation module to generate structured prompts for multimodal large models. This module receives user queries and structured context information, and generates prompt words that meet the input requirements of multimodal large models.

[0058] The specific process is as follows: (1) Traverse each valid image path in the context and construct a multimodal message containing image tag descriptions and local resource references to ensure that the model can recognize and associate the image content; (2) Combine all relevant text content into a single reference knowledge text by separating it with line breaks; (3) Construct the main question message, which contains explicit instructions: `You are a professional electrical engineer and circuit theory expert. Your task is to answer questions related to basic circuit concepts and laws in Chinese. Please provide a comprehensive answer based on the provided images and text, combined with your extensive knowledge and analytical skills. The answer must be logically coherent, the images and text must match, and the images must be properly labeled.` The images are used as markers and are naturally integrated into the corresponding positions of the answers. Ultimately, user questions, reference text, and image information are integrated into a structured sequence of multimodal prompt messages for use in large model inference.

[0059] Step S43: Construct the Retrieval Chain (RAG Chain) execution structure based on a multimodal large model. Build an end-to-end retrieval-generation chain, consisting of the following modules connected in sequence: (1) Question transmission: Receive the user's original query and pass it directly to the subsequent modules; (2) Context processing: Integrates a multi-vector searcher and a type separation function, responsible for retrieving and outputting the structured context; (3) Prompt generation: Call the predefined prompt function to integrate the user's question with the context into a multimodal prompt; (4) Multimodal large model inference: The Qwen-VL-Max multimodal large model is used to receive prompt messages and generate original responses containing text and image references; (5) Output parsing: Convert the structured response output by the model into a standard string format to ensure that the output content can be displayed and the format is compliant.

[0060] This retrieval chain automates the entire process of "query → retrieval → generation → output" through modular serialization.

[0061] Step S44: Execute the "retrieval-reasoning" process, generate a Markdown-formatted image and text answer, input the user query into the RAG chain constructed in step S43, triggering the complete "retrieval-reasoning" execution process: The multi-vector retrieval module matches the most relevant text and image data based on the semantics of the user query; the context processing module performs type separation and cleaning of the retrieval results, outputting structured information; the prompt word generation module constructs multimodal prompts containing image and text context and explicit instructions; the multimodal large model (such as Qwen-VL-Max) generates a coherent and professional natural language answer based on the prompts, and the model automatically identifies and references image tags during the generation process; the output parsing module converts the model output into a plain text string and retains... Image citation tags in image tag format. The final generated answer is Markdown formatted text, where image citations are in the format of... The transmitted images are embedded in the answer content in the form of actual tags, ensuring logical matching of image and text positions and clear layout. The system outputs this result to the front-end interface or log system, completing a full query response.

[0062] Figure 7 Screenshot of the answer provided on an existing general platform; Figure 8 The image shows the result of using a multi-dimensional knowledge base construction and query method based on a multimodal large model. In summary, this invention proposes a multi-dimensional knowledge base construction and query method based on a multimodal large model, aiming to solve the problems of incomplete integration, low query efficiency, and poor structure in existing technologies when processing multimodal data. By introducing an integrated design of efficient multimodal data parsing, retrieval-based summary generation, vectorized storage, and intelligent retrieval, this invention can not only simultaneously process text, tables, and image data in documents, but also optimize the generation of retrieval-based summaries and establish a precise association mechanism between summaries and original data. In particular, this invention strengthens the coupling between the vector database and the multimodal retrieval chain, and significantly improves the accuracy of information matching and the quality of search results by utilizing unique identifiers (UUIDs) and the similarity retrieval function of the Chroma vector database. Furthermore, through deep integration of context processing, prompt word optimization, and format parsing modules, this invention achieves efficient workflow operation from "retrieval" to "reasoning" to "output," ultimately generating structured query results with image references in Markdown format that meet the needs of professional fields. Therefore, compared with existing technologies, this invention not only provides a more comprehensive data integration solution, but also significantly improves the accuracy of retrieval and query efficiency, providing a knowledge base construction and query solution with completeness, accuracy and standardization for professional fields such as electrical engineering and document analysis.

[0063] The following is a comparison of the multimodal knowledge base indexing method constructed using the above document approach with the processing of the same document by the Tongyi Qianwen platform.

[0064] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A method for multi-dimensional knowledge base construction and query based on a multi-modal large model, characterized in that, Comprise the following steps: Step S1: parse the document file, extract the text block, table structure data and picture in the document file, save the picture to the specified directory and get the ordered picture path list; Step S2: select a pre-trained text large model, a multi-modal large model supporting cross-modal understanding of text and picture, and a text embedding model, and configure the API key required for model calling; Batch generate concise summaries suitable for retrieval scenarios using the text large model on the extracted text block and table structure data; Generate retrieval image summaries for each picture in the picture path list through the multi-modal large model, establish a one-to-one correspondence between the picture summary and the corresponding picture storage path, and form a text table summary set and a picture summary set; Step S3: based on the selected text embedding model, text table summary set, picture summary set, document file parsing result and picture path list in S1, construct a vector database, an in-memory storage module and a multi-vector retriever; wherein the vector database uses Chroma database and specifies a dedicated collection name; the in-memory storage module is used to save the original document elements; the multi-vector retriever is associated with the vector database and the in-memory storage module through a unique identifier, and the number of retrieval return results is configured; Step S4: construct a retrieval chain based on the multi-modal large model, which includes a question transmission module, a context processing module, a prompt word generation module, a multi-modal large model reasoning module and an output analysis module, generates a structured answer containing a picture reference label based on the relevant results returned by the multi-vector retriever and the user query.

2. The multi-modal large model-based multi-dimensional knowledge base construction and query method according to claim 1, characterized in that, In step S1, the document file contains at least one of text, table and picture; the document file includes text file and PDF file. 3.The multi-modal large model based multi-dimensional knowledge base construction and query method according to claim 1, characterized in that, The specific method of step S1 is: Step S11: get the storage path of the document file, and determine the specified target directory for saving the parsed and extracted pictures; Step S12: call a document parsing tool supporting text and picture extraction and table structure identification, perform parsing operation on the document file, enable picture extraction function to extract picture elements in the file, and enable table structure identification function to obtain structured table data; at the same time, set the text blocking parameter, and adopt the method of blocking according to the document title level to block the text content, to obtain the text block; Step S13: save the picture elements extracted in the parsing process to the specified directory determined in step S11, integrate the blocked text block, structured table data and picture elements with storage path, and form a document parsing result set containing multiple types of elements; Step S14: traverse the specified target directory where the pictures are saved, filter out the valid picture files with `.jpg`, `.jpeg` and `.png` suffixes, and arrange the ordered picture path list according to the order of picture appearance in the original document.

4. The multi-modal large model-based multi-dimensional knowledge base construction and query method according to claim 3, characterized in that, In step S12, the setting of the text blocking parameter is as follows: the maximum number of characters in a single text block is 4000, the character number threshold for triggering new text block segmentation is 3800, and the character number threshold for merging small text blocks is 2000. 5.The multi-modal large model based multi-dimensional knowledge base construction and query method according to claim 1, characterized in that, The specific method of step S2 is: Step S21: initialize the selected text large model, multi-modal large model and text embedding model, and configure the DashScope API key to enable normal calling of the model; Step S22: build a text table summary generation chain, which includes a document parameter mapping module, a prompt word template module, a text large model inference module and an output analysis module connected in turn; The document parameter mapping module is used to establish a correct correspondence between the picture and its summary; The prompt word template module is used to constrain the content generated by the AI large model; The text large model inference module is used to enhance the content generation of the large model by the content retrieved by the user input key; The output analysis module is used to convert the content generated by the large model and its corresponding picture into HTML format for output; The prompt word template in the prompt word template module clearly instructs to generate a concise summary suitable for retrieval, which needs to include the core entities, logical relationships and key data of the text and table to support the subsequent accurate retrieval of the original text and table elements; Step S23: input the document parsing result set generated in step S1 into the text table summary generation chain to generate text table summaries in a maximum concurrent number batch processing manner, add each text table summary to the summary list to form a text table summary set; for each picture in the picture path list, first perform path validity verification, and for the valid paths that pass the verification, generate a local picture URI; Step S24: input the local picture URI and the text instruction into the multi-modal large model to obtain the picture summary output by the model; the text instruction is to generate a concise image summary optimized for retrieval, which needs to include the core content, key features and context association information of the original document of the picture; 6.The multi-modal large model based multi-dimensional knowledge base construction and query method according to claim 5, characterized in that, Step S25: perform format verification on the multi-modal large model output result: if the output is a list containing a `text` key, extract the value corresponding to the `text` key; if the output is a string, directly take it as the picture summary; if the output format is unknown or invalid, mark it as unable to generate a picture summary; finally, form a picture summary set corresponding to the picture path list one by one to ensure that each picture path is associated with a unique picture summary.

7. The multi-modal large model based multi-dimensional knowledge base construction and query method according to claim 5, characterized in that, In step S21, the text large model uses the `QWEN_MAX_MODEL` model; the multi-modal large model uses the `QWEN_VL_MAX_MODEL` model; and the text embedding model uses the `TEXT_EMBEDDING_MODEL`. 8.The multi-modal large model based multi-dimensional knowledge base construction and query method according to claim 1, characterized in that, In step S23, the path validity verification includes verifying the existence, file type and accessibility of the path, i.e., whether the picture under each path exists; the URI format is `file: / / +picture absolute path`. The specific method of step S3 is: Step S31: input the text embedding model, the text table summary set, the document parsing result set and the specified Chroma database set name, and perform the following operations: (1) create a Chroma vector database instance corresponding to the set name, which is used to persistently store the vector representation of the text table summary converted by the text embedding model; (2) Create a memory storage module to save the original document elements in the document parsing result set; the original document elements include text blocks, table structure data, and picture storage paths; (3) Generate a UUID for each original document element as a unique identifier, construct a text table summary-UUID data pair, and store the high-dimensional vector content in the Chroma vector database after converting the text table summary into a high-dimensional vector through the text embedding model; (4) Establish a mapping relationship between UUID and the original document element, and store the mapping relationship in batches to the memory storage module; Step S32: Call the document adding method again, input the text embedding model, picture summary set, picture list, and the same Chroma database set name, repeat the operations of (1)-(4) in step S31, complete the storage and association of the picture summary vector and the picture list, and further strengthen the one-to-one correspondence between the picture summary and the picture storage path; Step S33: Based on the created Chroma vector database, memory storage module, and UUID, construct a multi-vector retriever; the multi-vector retriever supports matching user queries with summary vectors through vector similarity algorithms, and then obtains the corresponding original document elements or picture lists through UUID association. 9.The multi-modal large model based multi-dimensional knowledge base construction and query method according to claim 1, characterized in that, The specific method of step S4 is: Step S41: Construct a context processing module, which functions to: call the multi-vector retriever constructed in step S3 to obtain the original results related to the user query, separate the text content and picture path in the original results by constructing an image text type splitting function, remove invalid or duplicate information, and then construct a text string list containing tags to form structured context information; Step S42: Construct a prompt word generation module, which receives user queries and structured context information and integrates them into structured prompt words according to a preset template; Step S43: Construct a retrieval chain based on a multi-modal large model, and the chain structure execution order is: question transmission, context processing, prompt word generation, multi-modal large model reasoning, and output analysis; Step S44: input the user query into the retrieval chain, execute the retrieval-reasoning process, and finally generate the query result in Markdown format, where the pictures are referenced in the form of picture markers` picture markers to ensure that the picture reference position matches the answer content logically.

10. The multi-modal large model based multi-dimensional knowledge base construction and query method according to claim 9, characterized in that, In step S42, the structured prompt word explicitly instructs to generate a comprehensive and professional answer based on the provided text content and picture information, the answer needs to be logically coherent, and the picture reference needs to be labeled with corresponding picture markers to ensure that the text and picture information match; The retrieval-reasoning process is as follows: the multi-vector retriever obtains relevant results; the context processing module organizes information; the prompt word generation module constructs prompt words; the multi-modal large model generates answers; and the output analysis module processes the format.

Images