Document analysis and retrieval enhancement generation method fused with multi-modal embedding model

Abstract

The invention relates to a document analysis and retrieval enhancement generation method fused with a multi-modal embedding model, and belongs to the technical field of AI large model application. The method comprises the following steps of: analyzing a file and extracting contents of texts, tables and pictures; performing joint semantic embedding on the extracted texts, tables and pictures by using a multi-modal embedding model to generate uniform vector representation, and constructing a vector database supporting multi-modal data; and constructing a multi-modal retrieval chain based on a vector retriever. According to the method, total element analysis and joint modeling of complex documents are achieved, the problem that multi-modal information is difficult to process in a unified mode through a traditional vectorization technology is solved to a certain degree, and the information coverage capacity and the answer accuracy of a retrieval enhancement generation system in a real scene are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of AI large model application technology, specifically involving a document parsing and retrieval enhancement generation method that integrates multimodal embedding models. Background Technology

[0002] Against the backdrop of rapid development in artificial intelligence and large-scale models, Retrieval Augmentation (RAG) technology has become crucial for enhancing the professional question-answering capabilities of large language models. Traditional methods rely on plain text vectorization, segmenting documents into text fragments, processing them through a single-modal embedding model, and storing them in a vector database for retrieval. However, real-world documents such as scientific research papers and technical reports often contain multimodal information including text, tables, and images. Existing technologies often employ OCR recognition or directly ignore the deep semantics of images and tables, leading to the loss of key information and making it difficult to achieve cross-modal semantic alignment.

[0003] Meanwhile, mainstream vectorization methods lack the ability to effectively model images and structured data, and cannot generate unified multimodal vector representations, which limits the coverage and accuracy of retrieval. Although multimodal embedding models have the potential for text and image understanding, there is still a lack of effective solutions on how to systematically integrate them into the document parsing and retrieval process to achieve collaborative embedding and joint retrieval of multimodal content.

[0004] Therefore, an innovative method is needed that can comprehensively analyze and uniformly model text, tables, and images to improve the efficiency of knowledge utilization and the quality of generation in complex document scenarios. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies and provide a document parsing and retrieval enhancement generation method that integrates a multimodal embedding model.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A document parsing and retrieval enhancement generation method integrating a multimodal embedding model includes the following steps: Step S1: Parse the PDF file, extract text, tables, and images, and save the images to the specified directory; Step S2: Use a multimodal embedding model to perform joint semantic embedding on the extracted text, tables and images to generate a unified vector representation, and build a vector database that supports multimodal data, storing and indexing text fragments, table data and image paths as multimodal document units; Step S3: Construct a multimodal retrieval chain based on the vector retrieval tool, realize cross-modal semantic retrieval according to the user query, and generate a natural language response through a large language model.

[0007] Furthermore, preferably, the specific method of step S1 is as follows: Step S11: Take the PDF file to be parsed as the target document, obtain the storage path of the target document, and specify the output directory to save the extracted images; if the output directory does not exist, it will be created automatically. Step S12: Call the document parsing interface based on the Unstructured framework to perform parsing operations on the target document, generating a collection of document elements of multiple types, including text fragments, tables, and image blocks; Step S13: Filter and classify the element types from the multi-type document element set in Step S12. Convert the elements that belong to the table type into string form or HTML format structured data and retain them. Convert the composite elements that belong to text fragments into plain text form and summarize them to form an intermediate dataset containing structured table data and block text content. At the same time, save the image blocks in order to the image output directory specified in Step S11. Step S14: Traverse the image output directory set in step S11, retrieve all image files with the extensions .jpg, .jpeg, or .png, arrange them according to the natural order of file names or the order of timestamps, and generate an ordered list of complete storage paths for image files.

[0008] Furthermore, preferably, in step S12, the parsing parameters are configured as follows: Enable the image element extraction function to identify and separate embedded images in the document; Enable the table structure recognition function to restore the table's hierarchy and layout information; Set relevant thresholds for text content segmentation and segment the text according to the document's semantic structure using a title-based strategy.

[0009] Furthermore, preferably, the relevant threshold is as follows: the maximum number of characters in a single text block is limited to 4000, and when the cumulative number of characters reaches 3800, the segmentation of a new text block is triggered, and adjacent text blocks with a length of less than 2000 characters are merged.

[0010] Furthermore, preferably, the specific method of step S2 is as follows: Step S21: Initialize the multimodal embedding model; The multimodal embedding model is built based on the multimodal vectorization interface of the DashScope platform. The model name is configured as MULTI_MODAL_EMBEDDING_MODEL, and the corresponding API access key is set to enable remote service calls. The multimodal embedding model supports joint vector representation of text content and image files, and has three functional interfaces: text embedding, query embedding, and image embedding. Step S22: Input the segmented text and structured table data extracted in step S1 as a set of document text elements into the multimodal embedding model, and call its embed_documents interface to generate vector representations of the corresponding text elements one by one; wherein each segmented text or structured table data is encapsulated as an input object containing a text field, and after being encoded by the model, its output embedding vector is extracted to form a vector set of text elements. Step S23: For each local image in the ordered image path list generated in step S1, perform path validity verification to confirm that the file exists and is readable, and convert it into a local resource URI format that conforms to the requirements of the DashScope API; Step S24: Input the converted list of local image URIs into the embed_image interface of the multimodal embedding model; the model uploads the local images to the cloud OSS service through its internal mechanism and performs visual feature extraction; returns the high-dimensional embedding vector corresponding to each image; during the vector acquisition process, records the processing status code of each image and the values ​​of the first few dimensions of the vector for log tracking; finally, integrates all successfully encoded image vectors to form a vector set of image elements. Step S25: Input the set of text element vectors, the set of image element vectors, and the corresponding original element data into the vector database construction module. The vector storage manager establishes a vector index structure that supports multimodal retrieval. Generate a unified vector space representation that integrates text and image semantic information and store it in the multimodal vector knowledge base to complete the initial construction of the multimodal vector knowledge base.

[0011] Furthermore, preferably, in step S25, when establishing the vector index structure, the vector storage manager needs to synchronously record the association mapping relationship between text element vectors, image element vectors and original element data. This mapping relationship is stored in the form of key-value pairs, where the key is a unique identifier for the vector, and the value contains the type, content and page number position information of the corresponding original element in the original PDF document.

[0012] Furthermore, preferably, the specific method for constructing the multimodal retrieval chain based on the vector retrieval unit in step S3 is as follows: Step S31: Call the RAG chain construction method based on the LangChain framework, input the retrieval tool and the specified large language model name, and perform the following operations: (1) Initialize the large language model instance, and generate the answer content based on the input large language model name through the semantic understanding and reasoning mechanism inside the model; (2) Construct a multimodal retrieval-processing pipeline, obtain a set of documents related to the user query through the retrieval tool, and pass the set of documents into the document classification and processing module; (3) The document classification and processing module classifies the retrieved multimodal document units and determines whether the document content type is text or image: if it is an image file read through an image path, it is scaled and saved to a temporary path; if it is text, the text content is extracted directly, and finally the dictionary structure data containing the image path list and the text list is output. Step S32: Construct a query-context mapping relationship, associate the user query with the image path list and text list processed in step S31, and form structured data containing three fields: question, text context, and image context path; Step S33: Format the structured data formed in step S32 using the prompt word function to generate prompt words that meet the input requirements of the large language model; call the multimodal input interface of the large language model to process the prompt words. The processing logic is as follows: the model first parses the text context and image path in the prompt words, reads the corresponding image file through the image path and extracts visual features, and then performs joint reasoning by combining text semantics and visual features; finally, the output parser converts the model's returned result into string format to complete the response to the user's query.

[0013] Furthermore, preferably, in step S33, the template format of the prompt word function is to answer the question based on the following context: text context: {text_list}; image path: {image_path_list}; question: {user_query}.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the document parsing and retrieval enhancement generation method with integrated multimodal embedding model as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the document parsing and retrieval enhancement generation method fused with a multimodal embedding model as described above.

[0016] In this invention, the PDF file to be parsed is taken as the target document, the storage path of the target document is obtained, and the output directory for saving the extracted images is specified; if the output directory does not exist, it is automatically created to ensure that subsequent image files can be written normally.

[0017] In step S12 of this invention, the document parsing interface based on the Unstructured framework is a term in the prior art. Unstructured is an open-source document parsing tool library. Its standardized interface supports the extraction of multiple elements from PDF, DOC and other document formats. It can be directly called and parameters can be configured to achieve the separation of text, tables and images. It is a technical means known to those skilled in the art.

[0018] The relevant thresholds for text content segmentation are set, and a title-based strategy is adopted to segment the text according to the semantic structure of the document. Specifically, the title-based strategy refers to using the title text in the PDF document (such as first-level, second-level, and third-level titles, identified by font size, bolding style, and paragraph spacing) as the segmentation boundary, dividing "a title + all its corresponding body text fragments" into an independent segment text. If there is no body text between adjacent titles, the title is treated as a separate segment text. This strategy can ensure the semantic integrity within the segment text and avoid semantic fragmentation caused by the mixing of text fragments across titles.

[0019] In step S13 of this invention, each element type is selected and classified from the multi-type document element set in step S12; elements belonging to the table type are converted into string form or HTML format structured data and retained; composite elements belonging to text fragments are converted into plain text form and summarized to form an intermediate dataset containing structured table data and segmented text content; at the same time, image blocks are saved in sequence to the image output directory specified in step S11. In this context, the basic text units extracted during the document parsing stage are called "text fragments," and the text units formed after block processing are called "block text." The CompositeElement is an element type defined by the Unstructured framework, which refers to an aggregated text structure containing multiple sub-text elements (such as paragraphs, headings, and list items). Since PDF documents often contain related text combinations such as "heading + body paragraph" and "list heading + list item," using this type can preserve the semantic relationship between texts and avoid contextual fragmentation caused by independent extraction.

[0020] In step S14 of this invention, the image output directory set in step S11 is traversed, all image files with extensions of .jpg, .jpeg, or .png are retrieved, and arranged according to the natural order of file names or the order of timestamps to generate an ordered list of complete image file storage paths, which serves as the basic input for subsequent image referencing and analysis.

[0021] Sorting method description: ① Natural file name sorting: refers to sorting by the numerical sequence in the image block file name in ascending order, such as "image_1.jpg", "image_2.jpg"..."image_n.jpg"; ② Timestamp order: refers to sorting by the system timestamp when the image blocks were saved to the directory in ascending order, that is, the image blocks saved earlier are listed first, and the ones saved later are listed last; Both sorting methods must be consistent with the order in which the image blocks appear in the original PDF document to ensure the contextual accuracy when subsequently associating vectors. This invention preferably adopts the natural sorting method ①.

[0022] In step S21 of this invention, the following terms are explained: ① The multimodal vectorization interface of the DashScope platform is a standardized interface provided by the Alibaba Cloud DashScope large model service platform. It is a term in the prior art and supports joint vectorization of text and images. Those skilled in the art can obtain the calling method through the platform documentation; ② MULTI_MODAL_EMBEDDING_MODEL is the standard model name defined by the DashScope platform for multimodal embedding models. It is a model identification term built into the platform and is used to specify the type of model to be called. Both are existing publicly available technical terms.

[0023] In step S22 of this invention, the segmented text extracted in step S1 is the basic text unit, and the structured table data is processed according to the text format and classified into text class elements; the embed_documents interface is the built-in text embedding interface of the multimodal embedding model described in step S21. This interface supports batch input of text elements and returns the corresponding embedding vectors.

[0024] In step S22 of this invention, each piece of segmented text or structured table data is encapsulated as an input object containing a text field. After being encoded by the model, its output embedding vector is extracted to form a vector set of text-type elements. Here, "its" refers to "the input object containing a text field", that is, after the input object is encoded by the model, the embedding vector corresponding to the input object is extracted from the model output result.

[0025] In step S23 of this invention, for each local image in the ordered image path list generated in step S1, path validity verification is performed to confirm that the file exists and is readable, and it is converted into a local resource URI format that conforms to the requirements of DashScope API, namely file: / / +absolute image path, as a unified resource identifier for image input.

[0026] In step S24 of this invention, the embed_image interface is the built-in image embedding interface of the multimodal embedding model described in step S21, and belongs to the core functional interface of the model, just like the embed_documents interface. In step S24 of this invention: ① Internal mechanism: This refers to the model uploading the file stream corresponding to the local image URI to Alibaba Cloud OSS (Object Storage Service) via the HTTPS protocol. The upload process is automatically completed by the model interface without any additional user operation. This is a built-in data transmission mechanism of the DashScope platform and is existing technology. ② Visual feature extraction: A pre-trained convolutional neural network (such as ResNet or ViT architecture) is used to encode the image features, extracting visual information such as color, texture, and spatial structure. Then, a mapping layer is used to convert the visual features into vectors with the same dimension as the text embedding vector. This is a conventional feature extraction method for multimodal embedding models and is existing technology. ③ High-dimensional definition: This refers to the embedding vector having a dimension of not less than 512 dimensions. In this invention, it is specifically 768 dimensions, which conforms to the conventional high-dimensional vector design standard for multimodal embedding models.

[0027] During vector acquisition, the processing status code and the first few dimensions of the vector for each image are recorded for log tracking. The concepts and examples are explained below: ① Processing status code: This refers to the numerical code returned by the model to identify the image processing result. For example, "200" indicates successful processing, "400" indicates an image format error, and "500" indicates a model service error. Example: If an image is in .bmp format (not a supported format), the status code is recorded as "400". ② First few dimensions of the vector: This refers to the first 10 dimensions of the embedded vector (in this invention, the first 10 dimensions are fixed) used to quickly verify the validity of the vector generation. Example: The first 10 dimensions of the embedded vector for an image are recorded as "[0.123, -0.456, 0.789, 0.012, -0.345, 0.678, -0.901, 0.234, -0.567, 0.890]".

[0028] In step S25 of this invention, the set of text element vectors, the set of image element vectors, and the corresponding original element data are input into the vector database construction module. The vector storage manager establishes a vector index structure that supports multimodal retrieval, generates a unified vector space representation that integrates text and image semantic information, and completes the initial construction of the multimodal vector knowledge base, which serves as the basic index for subsequent similarity retrieval and question-answering matching.

[0029] In step S25 of this invention, ① Vector storage manager: is a built-in functional component of the vector database construction module, and belongs to the module component designed by this invention to realize the construction of vector index; ② Vector index structure: adopts the IVF_FLAT index structure of the FAISS (Facebook AI Similarity Search) open source index library, which belongs to the prior art and supports efficient multimodal vector similarity retrieval. This invention does not improve the core algorithm of the index structure, but only adapts to multimodal vector data through configuration.

[0030] In step S25 of this invention, when establishing the vector index structure, the vector storage manager needs to synchronously record the association mapping relationship between text element vectors, image element vectors and original element data. This mapping relationship is stored in the form of key-value pairs, where the key is a unique identifier for the vector, and the value contains the type (text, table, image), content (block text, HTML format table data, image path) and page number position information of the corresponding original element in the original PDF document, which facilitates quick location and tracing during subsequent retrieval.

[0031] In step S31 of this invention, the RAG chain construction method based on the LangChain framework is invoked. The RAG chain construction method is a standardized implementation method in the field of retrieval enhancement generation in the prior art. Its core is to integrate the retrieval machine, document processor and large language model through link-based design to form a closed-loop process of "retrieval-processing-generation". This invention optimizes the data adaptation and flow logic for multimodal scenarios based on the existing framework.

[0032] Perform the following operations: (1) Initialize the large language model instance. Based on the input large language model name (such as Qwen-VL-Max, GPT-4V and other models that support multimodal input) and API key, create a dialogue interface. This interface is a standardized interaction entry provided by the official large language model. It is used to receive formatted multimodal context data and user queries, and generate answer content through the semantic understanding and reasoning mechanism inside the model. (2) Construct a multimodal retrieval-processing pipeline (definition: refers to an automated processing link that connects operations such as retrieval result filtering, format conversion, and type classification in a fixed process). The retrieval tool (Note: the retrieval tool adopts the Retriever module of the existing open-source vector retrieval component Chroma. This module supports the similarity calculation and Top-K recall of multimodal vectors. This invention does not improve its core retrieval algorithm, but only adapts the multimodal vector data format by configuration) obtains a set of documents related to the user query (including text blocks, structured table data and image files corresponding to image paths, collectively referred to as "multimodal document units"). The set of documents is then passed to the document classification processing module (this module is a built-in functional module of step S3. Its core function is to distinguish the mixed type document units retrieved by retrieval, providing a basis for subsequent targeted processing. It is a functional supplement to the RAG chain in this invention). (3) The document classification processing module classifies the retrieved multimodal document units and determines whether the document content type is text (including structured table data) or image: If it is an image file read through an image path, it is scaled (scaled to a maximum of 640×480 pixels while maintaining the aspect ratio) and saved to a temporary path (reasons: ① to avoid large images occupying too much memory and causing the model interface call to fail; ② to adapt to the size limit of the input image for the large language model; ③ temporary path storage can reduce disk redundancy and is automatically cleaned up after processing); if it is text (including structured table data), the text content is directly extracted (the table data retains the structured information in HTML format), and finally outputs a dictionary structure data containing a list of image paths and a list of text. Step S32: Construct query-context mapping relationship (definition: refers to the mapping rule that establishes semantic association between user query and retrieved multimodal context data, specifically by binding "user query text" and "corresponding text list + image path list" in the form of key-value pairs to ensure that the large language model can obtain the question and related multimodal context at the same time), associate the user query with the image path list and text list processed in step S31 to form structured data containing three fields: "question", "text context" and "image context path"; Step S33: The structured data generated in Step S32 is formatted using a prompt word function (using a custom template function, with the template format being "Answer the question based on the following context: text context: {text_list}; image path: {image_path_list}; question: {user_query}") to generate prompt words that meet the input requirements of the large language model. The multimodal input interface of the large language model (such as the chat.completions interface of the Qwen-VL-Max model on the DashScope platform) is called to process the prompt words. The processing logic is as follows: the model first parses the text context and image path in the prompt words, reads the corresponding image file through the image path and extracts visual features, and then performs joint inference by combining text semantics and visual features. Finally, the output parser (using the StrOutputParser component of the LangChain framework, which is an existing standardized parsing tool, whose function is to convert the JSON format response returned by the large language model into a natural language string) converts the model's returned result into a string format to complete the response to the user query.

[0033] This invention aims to address the problems of information fragmentation, semantic loss, and insufficient cross-modal retrieval capabilities in existing document parsing and retrieval technologies when processing complex multimodal documents. It proposes an enhanced document parsing and retrieval generation method that integrates a multimodal embedding model. This method is designed for PDF documents containing text, tables, and images, achieving unified semantic modeling and efficient retrieval of all elements, and improving the accuracy and completeness of large language models in professional scenarios for question answering.

[0034] The technical solution of this invention mainly includes three core steps: First, in step S1, the input PDF file is deeply parsed to extract the text content, structured table data, and image elements, and the images are saved sequentially to a specified storage directory while retaining their original position and context information; Second, in step S2, an advanced multimodal embedding model is used to perform joint semantic encoding on the extracted text fragments, table data, and their corresponding images to generate a unified vector representation that integrates multimodal information, and a vector database supporting multimodal indexing is constructed to store and index each document unit (containing text, table, or image paths) and its vector embeddings in association; Finally, in step S3, a multimodal retrieval chain is constructed based on a vector retrieval device to receive user natural language queries, perform cross-modal semantic similarity matching, recall the most relevant multimodal document units, and input them as context into a large language model to generate accurate and natural text-image responses.

[0035] This invention introduces a multimodal embedding model and a unified vectorization mechanism to achieve full-modal parsing and joint modeling of complex document content, effectively overcoming the information limitations of traditional single-modal methods and significantly enhancing the information coverage and semantic understanding depth of the retrieval enhancement generation system in real-world application scenarios.

[0036] Compared with the prior art, the beneficial effects of this invention are as follows: (1) Achieved unified semantic modeling of multimodal information: This invention uses a multimodal embedding model to jointly embed text, tables and images to generate a unified vector representation, which breaks through the limitation of traditional methods that only rely on text vectorization, effectively preserves the key visual and structured semantic information in images and tables, and avoids information loss.

[0037] (2) Improved cross-modal semantic retrieval capabilities: By constructing a vector database that supports multimodal indexes, the system can achieve cross-modal semantic matching based on user queries, such as searching for images by text, searching for tables by images, and mixed image and text retrieval, which significantly enhances the semantic understanding depth and retrieval flexibility of the knowledge base.

[0038] (3) Improved the completeness of parsing complex documents: In the PDF parsing stage, not only is the plain text content extracted, but the table data structure and image context position information are also fully preserved, ensuring that the document elements maintain logical connection in the vectorization process, and improving the accuracy and interpretability of subsequent Q&A.

[0039] (4) Enhanced the response quality of the Retrieval Augmentation Generation (RAG) system: By introducing multimodal contextual information as input to the large language model, the generated response is not only based on textual reasoning, but also integrates chart data and visual semantics, making the output results more comprehensive and accurate, especially suitable for professional fields that rely heavily on textual and graphical information, such as scientific research, medical care, and engineering.

[0040] (5) It has good scalability and practicality: the method supports multimodal document input in multiple formats, the vector database design is compatible with mainstream multimodal embedding models, which facilitates system integration and deployment. Compared with traditional content generation, the method can accurately generate content combining text and graphics, providing a feasible technical path for building a high-precision and intelligent knowledge management system. Attached Figure Description

[0041] Figure 1 This is a schematic diagram illustrating the generation of combined text and image content using the document parsing and retrieval enhancement generation method that integrates multimodal embedding models as described in this invention. Figure 2 This is a schematic diagram of the content generated by traditional search enhancement techniques for the same question; Figure 3 This is an overall flowchart of the document parsing and retrieval enhancement generation method that integrates a multimodal embedding model according to the present invention; Figure 4 This is a schematic diagram of the multimodal vector library construction process in an embodiment of the present invention, illustrating the entire process of constructing a multimodal vector library from summary generation, vector conversion to storage association; Figure 5 This is a schematic diagram of the document parsing process in an embodiment of the present invention, showing the complete document parsing process including document path configuration, multi-element extraction, and structured processing; Figure 6 This is a schematic diagram of the overall question-and-answer system workflow in an embodiment of the present invention, depicting the entire process of the question-and-answer system from query input to generation of structured answers; Figure 7 This is a schematic diagram of the electronic device structure of the present invention. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the embodiments.

[0043] Those skilled in the art will understand that the following embodiments are for illustrative purposes only and should not be construed as limiting the scope of the invention. Where specific techniques or conditions are not specified in the embodiments, they are performed in accordance with the techniques or conditions described in the literature in the field or according to the product instructions. Materials or equipment whose manufacturers are not specified are all conventional products that can be obtained by purchase.

[0044] Example 1 A document parsing and retrieval enhancement generation method integrating a multimodal embedding model includes the following steps: Step S1: Parse the PDF file, extract text, tables, and images, and save the images to the specified directory; Step S2: Use a multimodal embedding model to perform joint semantic embedding on the extracted text, tables and images to generate a unified vector representation, and build a vector database that supports multimodal data, storing and indexing text fragments, table data and image paths as multimodal document units; Step S3: Construct a multimodal retrieval chain based on the vector retrieval tool, realize cross-modal semantic retrieval according to the user query, and generate a natural language response through a large language model.

[0045] Example 2 A document parsing and retrieval enhancement generation method integrating a multimodal embedding model includes the following steps: Step S1: Parse the PDF file, extract text, tables, and images, and save the images to the specified directory; Step S2: Use a multimodal embedding model to perform joint semantic embedding on the extracted text, tables and images to generate a unified vector representation, and build a vector database that supports multimodal data, storing and indexing text fragments, table data and image paths as multimodal document units; Step S3: Construct a multimodal retrieval chain based on the vector retrieval tool, realize cross-modal semantic retrieval according to the user query, and generate a natural language response through a large language model.

[0046] A document parsing and retrieval enhancement generation method integrating a multimodal embedding model, characterized by comprising the following steps: Step S1: Parse the PDF file, extract text, tables, and images, and save the images to the specified directory; Step S2: Use a multimodal embedding model to perform joint semantic embedding on the extracted text, tables and images to generate a unified vector representation, and build a vector database that supports multimodal data, storing and indexing text fragments, table data and image paths as multimodal document units; Step S3: Construct a multimodal retrieval chain based on the vector retrieval tool, realize cross-modal semantic retrieval according to the user query, and generate a natural language response through a large language model.

[0047] The specific method for step S1 is as follows: Step S11: Take the PDF file to be parsed as the target document, obtain the storage path of the target document, and specify the output directory to save the extracted images; if the output directory does not exist, it will be created automatically. Step S12: Call the document parsing interface based on the Unstructured framework to perform parsing operations on the target document, generating a collection of document elements of multiple types, including text fragments, tables, and image blocks; Step S13: Filter and classify the various element types from the multi-type document element set in Step S12. Convert the elements that belong to the table type into string form or HTML format structured data and retain them. Convert the composite elements that belong to text fragments into plain text form and summarize them to form an intermediate dataset containing structured table data and block text content. At the same time, save the image blocks in order to the image output directory specified in Step S11. Step S14: Traverse the image output directory set in step S11, retrieve all image files with the extensions .jpg, .jpeg, or .png, arrange them according to the natural order of file names or the order of timestamps, and generate an ordered list of complete storage paths for image files.

[0048] In step S12, during parsing, the parsing parameters are configured as follows: Enable the image element extraction function to identify and separate embedded images in the document; Enable the table structure recognition function to restore the table's hierarchy and layout information; Set relevant thresholds for text content segmentation and segment the text according to the document's semantic structure using a title-based strategy.

[0049] The relevant thresholds are as follows: the maximum number of characters in a single text block is limited to 4000. When the cumulative number of characters reaches 3800, a new text block is triggered, and adjacent text blocks with a length of less than 2000 characters are merged.

[0050] The specific method for step S2 is as follows: Step S21: Initialize the multimodal embedding model; The multimodal embedding model is built based on the multimodal vectorization interface of the DashScope platform. The model name is configured as MULTI_MODAL_EMBEDDING_MODEL, and the corresponding API access key is set to enable remote service calls. The multimodal embedding model supports joint vector representation of text content and image files, and has three functional interfaces: text embedding, query embedding, and image embedding. Step S22: Input the chunked text and structured table data extracted in step S1 as a set of document text elements into the multimodal embedding model, and call its embed_documents interface to generate vector representations of the corresponding text elements one by one; wherein each chunked text or structured table data is encapsulated as an input object containing a text field, and after model encoding, its output embedding vector is extracted to form a vector set of text elements. Step S23: For each local image in the ordered image path list generated in step S1, perform path validity verification to confirm that the file exists and is readable, and convert it into a local resource URI format that conforms to the requirements of the DashScope API; Step S24: Input the converted list of local image URIs into the embed_image interface of the multimodal embedding model; the model uploads the local images to the cloud OSS service through its internal mechanism and performs visual feature extraction; returns the high-dimensional embedding vector corresponding to each image; during the vector acquisition process, records the processing status code of each image and the values ​​of the first few dimensions of the vector for log tracking; finally, integrates all successfully encoded image vectors to form a vector set of image elements. Step S25: Input the set of text element vectors, the set of image element vectors, and the corresponding original element data into the vector database construction module. The vector storage manager establishes a vector index structure that supports multimodal retrieval. Generate a unified vector space representation that integrates text and image semantic information and store it in the multimodal vector knowledge base to complete the initial construction of the multimodal vector knowledge base.

[0051] In step S25, when establishing the vector index structure, the vector storage manager needs to synchronously record the association mapping relationship between text element vectors, image element vectors and original element data. This mapping relationship is stored in the form of key-value pairs, where the key is a unique identifier for the vector and the value contains the type, content and page number position information of the corresponding original element in the original PDF document.

[0052] The specific method for constructing step S3 of the multimodal retrieval chain based on the vector retrieval unit is as follows: Step S31: Call the RAG chain construction method based on the LangChain framework, input the retrieval tool and the specified large language model name, and perform the following operations: (1) Initialize the large language model instance, and generate the answer content based on the input large language model name through the semantic understanding and reasoning mechanism inside the model; (2) Construct a multimodal retrieval-processing pipeline, obtain a set of documents related to the user query through the retrieval tool, and pass the set of documents into the document classification and processing module; (3) The document classification and processing module classifies the retrieved multimodal document units and determines whether the document content type is text or image: if it is an image file read through an image path, it is scaled and saved to a temporary path; if it is text, the text content is extracted directly, and finally the dictionary structure data containing the image path list and the text list is output. Step S32: Construct a query-context mapping relationship, associate the user query with the image path list and text list processed in step S31, and form structured data containing three fields: question, text context, and image context path; Step S33: Format the structured data formed in step S32 using the prompt word function to generate prompt words that meet the input requirements of the large language model; call the multimodal input interface of the large language model to process the prompt words. The processing logic is as follows: the model first parses the text context and image path in the prompt words, reads the corresponding image file through the image path and extracts visual features, and then performs joint reasoning by combining text semantics and visual features; finally, the output parser converts the model's returned result into string format to complete the response to the user's query.

[0053] In step S33, the template format of the prompt word function is to answer the question based on the following context: text context: {text_list}; image path: {image_path_list}; question: {user_query}.

[0054] The system provided in this embodiment of the invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0055] Figure 7 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention, with reference to... Figure 7 The electronic device may include a processor 201, a communications interface 202, a memory 203, and a communication bus 204. The processor 201, communications interface 202, and memory 203 communicate with each other via the communication bus 204. The processor 201 can call logical instructions in the memory 203 to execute the following method step S1: parse a PDF file, extract text, tables, and image content, and save the images to a specified directory. Step S2: Use a multimodal embedding model to perform joint semantic embedding on the extracted text, tables and images to generate a unified vector representation, and build a vector database that supports multimodal data, storing and indexing text fragments, table data and image paths as multimodal document units; Step S3: Construct a multimodal retrieval chain based on the vector retrieval tool, realize cross-modal semantic retrieval according to the user query, and generate a natural language response through a large language model.

[0056] Furthermore, the logical instructions in the aforementioned memory 203 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0057] On the other hand, embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the document parsing and retrieval enhancement generation method fused with the multimodal embedding model provided in the above embodiments, for example including: Step S1: Parse the PDF file, extract text, tables, and images, and save the images to the specified directory; Step S2: Use a multimodal embedding model to perform joint semantic embedding on the extracted text, tables and images to generate a unified vector representation, and build a vector database that supports multimodal data, storing and indexing text fragments, table data and image paths as multimodal document units; Step S3: Construct a multimodal retrieval chain based on the vector retrieval tool, realize cross-modal semantic retrieval according to the user query, and generate a natural language response through a large language model.

[0058] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0059] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0060] Application Implementation like Figures 3 to 6 As shown, this invention provides a document parsing and retrieval enhancement generation method that integrates a multimodal embedding model. This method is applicable to knowledge extraction and intelligent question answering scenarios involving complex documents containing text, tables, and image elements. By fusing textual and visual semantic information, a unified multimodal vector space is constructed, and combined with a retrieval enhancement generation (RAG) mechanism, high-precision, context-aware answer generation is achieved. Step S1: Parse the PDF file, extract text, tables, and image block content, and save the image blocks to the specified directory; Step S2: Use a multimodal embedding model to perform joint semantic embedding on the extracted chunked text, table data and image blocks to generate a unified vector representation, and build a vector database that supports multimodal data, storing and indexing chunked text, HTML format table data and image paths as document units; Step S3: Construct a multimodal retrieval chain based on the vector retrieval tool, realize cross-modal semantic retrieval according to the user query, and generate a natural language response through a large language model.

[0061] For the specific implementation of step S1, please refer to [link / reference]. Figure 3 This illustrates the workflow of document parsing in this invention: In step S11, the system obtains the storage path of the target document and specifies the output directory for saving the extracted image blocks. If the output directory does not exist, the system automatically calls the directory creation interface to create the corresponding folder, ensuring that subsequent image blocks can be written correctly.

[0062] In step S12, the system calls the PDF parsing interface based on the unstructured framework to perform structured parsing operations on the target PDF document. Specifically, the parsing parameters are configured as follows: enable extract_images_in_pdf=True to identify and separate embedded image blocks in the document; enable infer_table_structure=True to restore the hierarchical structure and layout information of the table; set the relevant thresholds for text content segmentation, where the maximum number of characters in a single text block is limited to 4000, and when the cumulative number of characters reaches 3800, the segmentation of a new text block is triggered. Adjacent text blocks with a length of less than 2000 characters are merged, and the chunking_strategy="by_title" strategy is used to segment the text according to the semantic structure of the document (the "by title" strategy works in conjunction with the character count threshold to ensure that the semantics within the segmented text are complete and the length is compliant). After the above processing, the original PDF is parsed into a collection of multiple types of document elements, including text fragments, table objects, and image blocks.

[0063] In step S13, the system sequentially saves the image blocks extracted during the parsing process to the image block output directory specified in step S11. Simultaneously, it filters and categorizes the element types from the parsing results: for elements of table type, they are converted into structured data in HTML format and retained (prioritizing the retention of structured information for easier subsequent vector association and content display); for composite elements (CompositeElement) that are text fragments, they are converted into plain text and summarized to form an intermediate dataset containing structured table data and segmented text content.

[0064] In step S14, the image block output directory set in step S11 is traversed, and all image files with extensions of .jpg, .jpeg, or .png are retrieved and arranged according to the natural order of file names to generate an ordered list of complete storage paths for image blocks. (Sorting method explanation: ① Natural order of file names: refers to ascending order of the numerical sequence in the image block file name, such as "image_1.jpg", "image_2.jpg", ... "image_n.jpg"; ② Timestamp order: refers to ascending order of the system timestamp when the image blocks are saved to the directory, that is, the image blocks saved earlier are listed first, and the ones saved later are listed last; Both sorting methods need to be verified by parsing the page index of the PDF document (each image block is associated with the PDF page number when it was extracted) to ensure that it is consistent with the order of appearance of the image blocks in the original PDF document, to avoid order misalignment caused by storage delay or naming abnormalities, and to ensure the context accuracy when the vectors are associated later. This paper adopts the natural order of ①; Note: Only .jpg, .jpeg, or .png format image blocks are supported. Other format files are directly excluded in the filtering stage of this step and will not enter the subsequent processing flow).

[0065] For the specific implementation method of step S2, please refer to Figure 2 It illustrates the construction process of the multimodal vector library in this invention: In step S21, the system initializes a multimodal embedding model, which is built based on the DashScope platform's multimodal vectorization interface (MultiModalEmbedding). Specifically, the model name is configured as MULTI_MODAL_EMBEDDING_MODEL, and the corresponding API access key is loaded through environment variables or configuration files to enable secure calls to remote services. The multimodal embedding model supports joint vector representation of text content and image blocks, and has three functional interfaces: text embedding (embed_documents), query embedding (embed_query), and image embedding (embed_image).

[0066] In step S22, the system inputs the segmented text content and HTML-formatted structured table data extracted in step S1 as a set of document text elements into the multimodal embedding model. By calling its embed_documents interface, a vector representation of each text or table content is generated one by one. Each input is encapsulated as a JSON object containing a "text" field. After model encoding, its output high-dimensional embedding vector (768 dimensions, conforming to the conventional design standards of multimodal embedding models) is extracted and finally integrated to form a vector set of text elements.

[0067] In step S23, the system performs path validity verification on each image block in the ordered image block path list generated in step S1 to confirm that the file exists and is readable. Then, it converts the path to a local resource URI format conforming to the DashScope API requirements, i.e., file: / / +absolute path of the image block, as the Uniform Resource Identifier for the image input.

[0068] In step S24, the system inputs the converted list of local image patch URIs into the embed_image interface of the multimodal embedding model. The model's internal mechanism uploads the local image patches to the cloud OSS service and uses the pre-trained ViT architecture to extract their visual features (extracting key information such as color, texture, and spatial structure), returning a 768-dimensional high-dimensional embedding vector for each image patch. During vector acquisition, the system records the processing status code for each image patch (e.g., "200" indicates successful processing, "400" indicates parameter error, and "500" indicates service error) and the first 10 dimensions of the vector for log tracking and anomaly investigation (image patches in unsupported formats have already been filtered and excluded in step S14; this step does not require handling format error scenarios). Finally, all successfully encoded image patch vectors are integrated to form a vector set of image elements.

[0069] In step S25, the set of text element vectors, the set of image element vectors, and the corresponding original element data are input into the vector database construction module. This module establishes a vector index structure that supports multimodal retrieval through the core submodule—the vector storage manager (the vector storage manager is the core submodule of the vector database construction module, responsible for the creation and maintenance of vector indexes). (Source and prior art description: ① Vector storage manager: By calling the API interface of the FAISS open-source index library, multimodal vector data is organized and stored according to the IVF_FLAT index structure; ② Vector index structure: The IVF_FLAT index structure of the FAISS (Facebook AI Similarity Search) open-source index library is adopted, which is a prior art and supports efficient multimodal vector similarity retrieval. This invention does not improve the core algorithm of the index structure, but only adapts the multimodal vector data through the parameter configuration of the vector storage manager). A unified vector space representation that integrates text and image semantic information is generated and stored in the multimodal vector knowledge base, completing the initial construction of the multimodal vector knowledge base. Meanwhile, the vector storage manager synchronously records the association mapping relationship between text element vectors, image element vectors and original element data. This mapping relationship is stored in the form of key-value pairs, where the key is a unique identifier for the vector, and the value contains the type (text, table, image), content (block text, HTML format table data, image block path) of the corresponding original element, as well as the page number position information in the original PDF document, which facilitates quick location and tracing during subsequent retrieval.

[0070] For the specific implementation method of step S3, please refer to Figure 3 and Figure 6 These respectively illustrate the user query processing flow and the overall system operation logic: In step S31, the system calls the RAG chain construction method, inputs the retrieval tool and the specified large language model name (such as Qwen-VL-Max), and performs the following operations: (1) Initialize the large language model instance, create a dialogue interface based on the input model name and API key to receive user queries and generate answers; at the same time, call the embed_query interface of the multimodal embedding model to convert the user query text into a 768-dimensional query vector for subsequent cross-modal similarity retrieval; (2) Build a retrieval-processing pipeline, using the retrieval tool (using the Retriever module of the existing open-source vector retrieval component Chroma). (2) Obtain the document set related to the user query from the constructed multimodal vector library (including block text, HTML format table data and image block paths, collectively referred to as "multimodal document units"); (3) Pass the retrieved documents into the document classification processing module, which determines the document content type: if it is an image block path, read the image file through the path, scale it (scale to a maximum of 640×480 pixels, maintaining the aspect ratio) and save it to a temporary path; if it is text data (block text, HTML table data), extract the content directly, and finally output a dictionary structure containing a list of temporary image block paths and a text list.

[0071] In step S32, the system constructs a query-context mapping relationship (referring to the mapping rules that establish semantic associations between user queries and retrieved multimodal context data), associating the user query with the temporary image patch path list and text list processed in step S31, forming structured data containing three fields: "question," "text context," and "image context path." This design enables the large language model to comprehensively analyze relevant image and text context information while understanding the question.

[0072] In step S33, the system formats the structured data using a prompt function to generate a prompt template that meets the input requirements of the large language model (template format: "Answer the question based on the following context: text context: {text_list}; image path: {image_path_list}; question: {user_query}"). Then, the system calls the multimodal input interface of the large language model (such as the chat.completions interface of the Qwen-VL-Max model on the DashScope platform) to perform inference processing on the prompt. The processing logic is as follows: the model first parses the text context and image path in the prompt, reads the corresponding image file through the image path and extracts visual features, then combines the text semantics and visual features for joint inference; and finally, the system converts the raw response returned by the model into a standardized string format using an output parser (such as StrOutputParser), while cleaning up the image files in the temporary directory, ultimately completing the generation of the user query response. In addition, to prevent images from being too large and exceeding the model context window limit, the system's integrated image preprocessing mechanism (such as the resize_image() function) scales the received image files proportionally, compressing their size to a maximum of 640×480 pixels while maintaining the aspect ratio, effectively improving the compatibility and stability of multimodal input.

[0073] Figure 1 This is a schematic diagram illustrating the generation of combined text and image content using the document parsing and retrieval enhancement generation method that integrates multimodal embedding models as described in this invention. Figure 2 This is a schematic diagram of the content generated by traditional search enhancement techniques for the same question.

[0074] In summary, this invention proposes a multi-dimensional knowledge base construction and query scheme to solve the challenge of knowledge extraction from complex PDF documents, and achieves intelligent question answering by combining visual language information. The method includes efficient document parsing technology, decomposing PDFs into structured text, tables, and images to ensure data consistency and integrity. It utilizes a multimodal embedding model for joint semantic embedding, creating a unified vector representation space that allows text and images to be processed within the same framework. The RAG mechanism is introduced to transform user queries into accurate answers, improving accuracy and efficiency and enhancing context awareness. Chroma is used as the vector storage engine to build a multimodal retrieval index structure, improving retrieval performance and result relevance. An integrated image preprocessing mechanism optimizes multimodal input processing and improves system efficiency. Compared to existing technologies, this invention provides a comprehensive, accurate, and efficient solution suitable for complex document structures and multimodal data processing scenarios, demonstrating significant advantages in information integration, retrieval efficiency, and result accuracy, providing strong support for professional fields.

[0075] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of this invention is defined by the appended claims and their equivalents.

Claims

1. A document parsing and retrieval enhancement generation method integrating a multimodal embedding model, characterized in that, It includes the following steps: Step S1: Parse the PDF file, extract text, tables, and images, and save the images to the specified directory; Step S2: Use a multimodal embedding model to perform joint semantic embedding on the extracted text, tables and images to generate a unified vector representation, and build a vector database that supports multimodal data, storing and indexing text fragments, table data and image paths as multimodal document units; Step S3: Construct a multimodal retrieval chain based on the vector retrieval tool, realize cross-modal semantic retrieval according to the user query, and generate a natural language response through a large language model.

2. The document parsing and retrieval enhancement generation method integrating a multimodal embedding model according to claim 1, characterized in that, The specific method for step S1 is as follows: Step S11: Take the PDF file to be parsed as the target document, obtain the storage path of the target document, and specify the output directory to save the extracted images; if the output directory does not exist, it will be created automatically. Step S12: Call the document parsing interface based on the Unstructured framework to perform parsing operations on the target document, generating a collection of document elements of multiple types, including text fragments, tables, and image blocks; Step S13: Filter and classify the element types from the multi-type document element set in Step S12. Convert the elements that belong to the table type into string form or HTML format structured data and retain them. Convert the composite elements that belong to text fragments into plain text form and summarize them to form an intermediate dataset containing structured table data and block text content. At the same time, save the image blocks in order to the image output directory specified in Step S11. Step S14: Traverse the image output directory set in step S11, retrieve all image files with the extensions .jpg, .jpeg, or .png, arrange them according to the natural order of file names or the order of timestamps, and generate an ordered list of complete storage paths for image files.

3. The document parsing and retrieval enhancement generation method integrating a multimodal embedding model according to claim 2, characterized in that, In step S12, during parsing, the parsing parameters are configured as follows: Enable the image element extraction function to identify and separate embedded images in the document; Enable the table structure recognition function to restore the table's hierarchy and layout information; Set relevant thresholds for text content segmentation and segment the text according to the document's semantic structure using a title-based strategy.

4. The document parsing and retrieval enhancement generation method integrating a multimodal embedding model according to claim 3, characterized in that, The relevant thresholds are as follows: the maximum number of characters in a single text block is limited to 4000. When the cumulative number of characters reaches 3800, a new text block is triggered, and adjacent text blocks with a length of less than 2000 characters are merged.

5. The document parsing and retrieval enhancement generation method integrating a multimodal embedding model according to claim 2, characterized in that, The specific method for step S2 is as follows: Step S21: Initialize the multimodal embedding model; The multimodal embedding model is built based on the multimodal vectorization interface of the DashScope platform. The model name is configured as MULTI_MODAL_EMBEDDING_MODEL, and the corresponding API access key is set to enable remote service calls. The multimodal embedding model supports joint vector representation of text content and image files, and has three functional interfaces: text embedding, query embedding, and image embedding. Step S22: Input the segmented text and structured table data extracted in step S1 as a set of document text elements into the multimodal embedding model, and call its embed_documents interface to generate vector representations of the corresponding text elements one by one; wherein each segmented text or structured table data is encapsulated as an input object containing a text field, and after being encoded by the model, its output embedding vector is extracted to form a vector set of text elements. Step S23: For each local image in the ordered image path list generated in step S1, perform path validity verification to confirm that the file exists and is readable, and convert it into a local resource URI format that conforms to the requirements of the DashScope API; Step S24: Input the converted list of local image URIs into the embed_image interface of the multimodal embedding model; the model uploads the local images to the cloud OSS service through its internal mechanism and performs visual feature extraction; returns the high-dimensional embedding vector corresponding to each image; during the vector acquisition process, records the processing status code of each image and the values ​​of the first few dimensions of the vector for log tracking; finally, integrates all successfully encoded image vectors to form a vector set of image elements. Step S25: Input the set of text element vectors, the set of image element vectors, and the corresponding original element data into the vector database construction module. The vector storage manager establishes a vector index structure that supports multimodal retrieval. Generate a unified vector space representation that integrates text and image semantic information and store it in the multimodal vector knowledge base to complete the initial construction of the multimodal vector knowledge base.

6. The document parsing and retrieval enhancement generation method integrating a multimodal embedding model according to claim 5, characterized in that, In step S25, when establishing the vector index structure, the vector storage manager needs to synchronously record the association mapping relationship between text element vectors, image element vectors and original element data. This mapping relationship is stored in the form of key-value pairs, where the key is a unique identifier for the vector and the value contains the type, content and page number position information of the corresponding original element in the original PDF document.

7. The document parsing and retrieval enhancement generation method integrating a multimodal embedding model according to claim 1, characterized in that, The specific method for constructing step S3 of the multimodal retrieval chain based on the vector retrieval unit is as follows: Step S31: Call the RAG chain construction method based on the LangChain framework, input the retrieval tool and the specified large language model name, and perform the following operations: (1) Initialize the large language model instance, and generate the answer content based on the input large language model name through the semantic understanding and reasoning mechanism inside the model; (2) Construct a multimodal retrieval-processing pipeline, obtain a set of documents related to the user query through the retrieval tool, and pass the set of documents into the document classification and processing module; (3) The document classification and processing module classifies the retrieved multimodal document units and determines whether the document content type is text or image: if it is an image file read through an image path, it is scaled and saved to a temporary path; if it is text, the text content is extracted directly, and finally the dictionary structure data containing the image path list and the text list is output. Step S32: Construct a query-context mapping relationship, associate the user query with the image path list and text list processed in step S31, and form structured data containing three fields: question, text context, and image context path; Step S33: Format the structured data formed in step S32 using the prompt word function to generate prompt words that meet the input requirements of the large language model; The multimodal input interface of the large language model is called to process the prompt words. The processing logic is as follows: the model first parses the text context and image path in the prompt words, reads the corresponding image file through the image path and extracts visual features, and then performs joint reasoning by combining text semantics and visual features; finally, the output parser converts the model's returned result into string format to complete the response to the user's query.

8. The document parsing and retrieval enhancement generation method integrating a multimodal embedding model according to claim 1, characterized in that, In step S33, the template format of the prompt word function is to answer the question based on the following context: text context: {text_list}; image path: {image_path_list}; question: {user_query}.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the document parsing and retrieval enhancement generation method that integrates a multimodal embedding model as described in claim 1.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the document parsing and retrieval enhancement generation method that integrates a multimodal embedding model as described in claim 1.

Images