A knowledge embedding and retrieval method and system for multi-modal PDF documents

Abstract

The application discloses a kind of knowledge embedding and retrieval method and system of multi-modal PDF document, specifically relates to knowledge embedding and retrieval field, the present application uses multi-modal big model identification picture, table and other non-text elements in PDF, generates its position coordinate, converts picture into structured literal description and stores source URL;Second, the original text, picture description and URL are integrated into Markdown format, then, adopt the strategy of gradual blocking: each picture information is individually used as text block, and the remaining pure text is blocked according to character length;Subsequently, all text blocks are vector embedded, and the type is stored as metadata into vector database;Finally, in the retrieval stage, relevant text blocks are returned based on vector similarity;In the generation stage, single mode or multi-modal model generates accurate answer using text content and picture URL.

Description

Technical Field

[0001] This invention relates to the field of knowledge embedding and retrieval technology, and more specifically, to a method and system for knowledge embedding and retrieval of multimodal PDF documents. Background Technology

[0002] In existing technologies, RAG (Retrieval-Augmented Generation) systems, when processing multimodal PDF documents containing text and images, typically employ a large multimodal model to convert image content into text descriptions or use OCR to extract text information from the document before further processing, thus converting the multimodal PDF into plain text format. Subsequent text segmentation and vector embedding are then performed, and finally, the text blocks and their corresponding vectors are stored in a database for retrieval. Another approach avoids structured PDF conversion, instead directly inputting all pages of the entire PDF as images into the large multimodal model, which then directly understands the document content and generates answers during the inference phase. However, this approach cannot reuse pure language models that only support text input in retrieval enhancement scenarios, and each question-and-answer session requires a large number of image tokens corresponding to the entire document, resulting in excessive token consumption and potentially exceeding the context length limit of the large multimodal model. Regardless of the method used, the selection and generation performance of the large language generation model will still be significantly limited in the subsequent retrieval phase, mainly for the following two reasons: First, it's difficult to balance the degree of information retention. While multimodal models can preserve semantic information in image descriptions, visual information (such as color, font, and layout) is often easily lost. When user queries involve visual features, the system struggles to provide accurate answers. If developers attempt to ensure the model retains all visual information in image descriptions, the descriptions become overly lengthy and cluttered, making it difficult to strike a balance in retaining the appropriate level of visual information.

[0003] Second, image information is corrupted during the segmentation process. Text descriptions of images or tables may be split during text segmentation, resulting in incomplete information. Although this can be mitigated by using sliders or child-level searches to return to the parent level, it cannot be completely avoided and introduces contextual noise, affecting retrieval and generation performance. Summary of the Invention

[0004] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a method and system for knowledge embedding and retrieval of multimodal PDF documents to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A1: A multimodal model is used to identify non-text elements in multimodal PDF documents. Relative position coordinates are generated based on the four corners of the rectangular outline of the non-text elements. At the same time, the images are converted into text descriptions to preserve semantic information, and the source URL of the images is stored. A2: Integrate the original text from the multimodal PDF document, the image text descriptions obtained in A1, and the image source URLs, and output them in Markdown format; A3: Divide each image information in the Markdown format generated in A2 into a separate text block; integrate the remaining plain text content in the Markdown format except for the image information, and use a character splitter to divide it into blocks to obtain plain text blocks; A4: Perform vector embedding on all text blocks obtained in A3 to generate vectors corresponding to each text block; store the type of each text block as metadata, associated with the text block itself and its corresponding vector, in the vector database; A5: Receives user query requests and returns the text block with the highest relevance to the user query from the vector database based on vector similarity. During the generation phase, depending on whether a single-modal or multi-modal language model is invoked, it utilizes the text information in the returned text block and the image information / image description information corresponding to the image source URL to construct a detailed multi-modal context and generate an answer to the user query.

[0006] Preferably, in A1, a multimodal model integrating visual information understanding and semantic generation functions is used to perform full-domain parsing of the multimodal PDF document. This model scans the document page by page and uses a visual attention mechanism to identify non-text elements, including images and tables. For images, the model distinguishes between sub-types such as photos, charts, and diagrams. For tables, it identifies their border structure, cell distribution, and merge status. In the localization phase, a two-dimensional coordinate system is established with the top left corner of the PDF page as the origin. The pixel coordinates of the four vertices of the rectangular outline of each non-text element are extracted (e.g., ...). For the top left corner (The bottom right corner) uses a coordinate transformation algorithm to map pixel coordinates to normalized coordinates relative to the page size; For image elements, a dual-channel processing approach of "visual feature understanding - text generation" is used: the semantic information of the image is identified and understood through a multimodal large model, and then its autoregressive language generation capability is combined to generate a structured text description of the semantic information of the image. The description content includes core visual information. At the same time, the system automatically generates a unique identifier for each image, stores the image in a lossless format to a distributed file unit, and generates a source URL containing the storage path, access permissions and UUID. This URL is bound to the normalized coordinates of the image through an association table and stored to form a three-element mapping relationship of "location-content-access path".

[0007] Preferably, in A2, Markdown syntax is used as the carrier in the format conversion stage to construct a relational system of "original text-image semantics-visual resources"; firstly, the original text in the multimodal PDF is structurally sorted and converted into the corresponding Markdown format according to the inherent logic of the document: first-level headings are marked with "#", second-level headings are marked with "##", body paragraphs directly retain the original line breaks and indentation, ordered lists are marked with "1." and "2.", and unordered lists are marked with "-". When integrating the image elements output from the A1 stage, the Markdown standard image syntax is adopted. The "image description information" uses the structured text description generated in A1, and the "image source URL" embeds a complete link containing the UUID and access permissions. At the same time, the location information generated in A1 is implicitly linked through Markdown comments after the image syntax. The process of generating structured text descriptions is as follows: The A1 stage uses a multimodal model (such as BLIP-2) that integrates visual information understanding and semantic generation to scan each page of a PDF, identify non-text elements in the images, and distinguish between different image types such as photos, charts, and diagrams. The model uses a dual-channel processing approach of "visual feature understanding - text generation" to identify and understand the core semantic information and key visual features of images, and then combines this with autoregressive language generation capabilities to generate structured text descriptions. The process for generating and storing the image source URL is as follows: In step A1, after parsing the images, the system automatically generates a unique identifier (UUID) for each image and stores the images in a lossless format to a distributed file unit. Based on the image's storage path, access permission configuration, and the generated UUID, the system constructs a complete image source URL, which is unique and has exclusive access. This URL will be bound to the normalized relative position coordinates of the image in the PDF through an association table, forming a three-element mapping relationship of "location-content-access path", and finally embedded as the "image source URL" in the Markdown image syntax.

[0008] Preferably, in step A3, let D be the complete Markdown document generated by step A2; this document is a sequence of basic elements arranged in order, i.e. , where each element It can be one of three elements: text paragraph, heading, or image tag; The segmentation process first identifies and extracts all image tags; the image tag set P is defined as: ; For each identified image tag It is directly divided into separate text blocks and treated as an independent text block. Use functions This operation means: ; After extracting all image tags, the remaining plain text element sequence is obtained. Will All plain text elements are concatenated in their original order to form a continuous plain text string A, i.e. Then, A is divided into blocks using a character splitter. The splitter divides A into plain text blocks using a sliding window based on a preset block size L and overlap length O.

[0009] Preferably, in A4, a pre-trained text embedding model is used to process all text blocks obtained in A3. Vectorization is performed separately; let the embedding model be a function. Map a text string to a A 1-dimensional real vector for each text block in set B. Calculate the corresponding vector representation : in, Represented as a text block The string content; Whether it's an image-based text block or a plain text block, they are all uniformly converted into a point in a high-dimensional space through the same embedding model; The system adds metadata when storing each text block; for each text block Extract key attributes to construct a metadata object.

[0010] Preferably, in step A5, when a user query is received, the retrieval stage is initiated first; the same text embedding model as in step A4 is used to convert the user's query request in natural language into a query vector in a high-dimensional vector space. ,Right now Then, from the vector database constructed using the A4 steps, the knowledge fragments most relevant to the user's question are located; Specifically, the system initiates an approximate nearest neighbor search in the vector database, which is performed in a deep semantic space. The algorithm calculates the cosine similarity or inner product distance metric between the query vector and all stored text block vectors in the database, and quickly finds h sets of text blocks that are most semantically relevant to the query from the database. The text blocks are then sorted based on similarity scores, and the top-K text blocks that are semantically related to the user's query are finally selected. The specific calculation method is as follows: Based on the modality of the large model actually used by the user, the system automatically switches the returned image modality. If the user selected a multimodal large model when generating the question, then blocks with image attributes will only send the URL link, while blocks with text attributes will send the token in its original form, preserving as much multimodal image information as possible while saving unnecessary tokens. If a pure language large model is used, the large model will answer and operate based on the text description and image URL; depending on the context of the query, it will decide whether to enable metadata filtering; for example, when the user explicitly asks "Please show and explain the flowchart on page 3", the system will pre-limit the search scope to text blocks of type "image" and page number "3"; finally, the set of most relevant text blocks, as context information, will be passed to the next stage.

[0011] Technical effects and advantages of the present invention: This invention solves the problem of balancing information retention. By extracting and locating multimodal information, storing image source URLs, and combining this with Markdown format output, it achieves complete preservation of semantic and visual information. Users can directly access the image itself, avoiding information loss or redundancy. Secondly, it solves the problem of image information being destroyed during the segmentation process. This invention adopts a hierarchical text segmentation strategy, treating image information as a separate text block and integrating it with plain text content for segmentation, ensuring information integrity and reducing noise interference. Furthermore, this invention employs an automatic information retrieval modality switching mechanism, decoupling it from the user's use of the large model modality. This preserves all image information while fully utilizing multimodal information even in scenarios with limited model resources. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0013] Figure 2 This is a schematic diagram of the module connection of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Please see Figure 1 As shown, this invention provides a method for knowledge embedding and retrieval of multimodal PDF documents, including: A1: A multimodal model is used to identify non-text elements in multimodal PDF documents, including images and tables; relative position coordinates are generated based on the four corners of the rectangular outline of the non-text elements, while the images are converted into text descriptions to preserve semantic information, and the source URL of the images is stored; In A1, the BLIP-2 model, which integrates object detection and semantic generation, performs full-domain parsing of multimodal PDF documents. This model scans the document page by page and uses a visual attention mechanism to identify non-text elements, including images and tables. For images, the model distinguishes between sub-types such as photos, charts, and diagrams; for tables, it identifies their border structure, cell distribution, and merge status. In the localization phase, a two-dimensional coordinate system is established with the top left corner of the PDF page as the origin. The pixel coordinates of the four vertices of the rectangular outline of each non-text element are extracted (e.g., ...). For the top left corner (The bottom right corner) uses a coordinate transformation algorithm to map pixel coordinates to normalized coordinates relative to the page size; For image elements, a dual-channel processing approach of "visual feature understanding - text generation" is used: the semantic information of the image is identified and understood through a multimodal large model, and then its autoregressive language generation capability is combined to generate a structured text description of the semantic information of the image. The description content includes core visual information. At the same time, the system automatically generates a unique identifier for each image, stores the image in a lossless format to a distributed file unit, and generates a source URL containing the storage path, access permissions and UUID. This URL is bound to the normalized coordinates of the image through an association table and stored to form a three-element mapping relationship of "location-content-access path".

[0016] For table elements, in addition to generating the relative position coordinates of their rectangular outlines, the table structure recognition algorithm is used to parse the row and column relationships, convert the table content into structured text, and simultaneously record the table title, header information, and page number association in the document.

[0017] A2: Integrate the original text from the multimodal PDF document, the image text descriptions obtained in A1, and the image source URLs, and output them in Markdown format; In section A2, Markdown syntax is used as the carrier during the format conversion stage to construct a relational system of "original text - image semantics - visual resources". First, the original text in the multimodal PDF is structurally sorted and converted into the corresponding Markdown format according to the inherent logic of the document: first-level headings are marked with "#", second-level headings are marked with "##", body paragraphs directly retain the original line breaks and indentation, ordered lists are marked with "1." and "2.", and unordered lists are marked with "-". When integrating the image elements output from the A1 stage, the Markdown standard image syntax is adopted. The "image description information" uses the structured text description generated in A1, and the "image source URL" embeds a complete link containing the UUID and access permissions. At the same time, the location information generated in A1 is implicitly linked through Markdown comments after the image syntax. The process of generating structured text descriptions is as follows: The A1 stage uses a multimodal model (such as BLIP-2) that integrates visual information understanding and semantic generation to scan each page of a PDF, identify non-text elements in the images, and distinguish between different image types such as photos, charts, and diagrams. The model uses a dual-channel processing approach of "visual feature understanding - text generation" to identify and understand the core semantic information and key visual features of images, and then combines this with autoregressive language generation capabilities to generate structured text descriptions. The process for generating and storing the image source URL is as follows: In step A1, after parsing the images, the system automatically generates a unique identifier (UUID) for each image and stores the images in a lossless format to a distributed file unit. Based on the image's storage path, access permission configuration, and the generated UUID, the system constructs a complete image source URL, which is unique and has exclusive access. This URL will be bound to the normalized relative position coordinates of the image in the PDF through an association table, forming a three-element mapping relationship of "location-content-access path", and finally embedded as the "image source URL" in the Markdown image syntax.

[0018] For table elements processed in A1, if the table content has been converted into structured text, it is directly embedded into the corresponding text paragraph; if the table contains complex visual styles (such as merged cells, color annotations), the core data is first represented as a Markdown table, and then image syntax is added below the table. The URL of the table's visual screenshot is associated with the description to achieve a dual presentation of data structuring and visual traceability. In addition, to enhance the relevance of multimodal information, a cross-reference mechanism of "text-image-table" needs to be established during the integration process: when an image is mentioned in the original text, the Markdown syntax of that image is directly inserted at the corresponding position to ensure that the text and image positions match; when the text references table data, the table position is linked through anchor points at the reference point, and a unique ID attribute is added to the table; the final generated Markdown document is checked for grammatical correctness through a format validation tool to ensure that the logical integrity of the original text, the accessibility of image information, and the accuracy of table data are consistent.

[0019] A3: Divide each image information in the Markdown format generated in A2 into a separate text block; integrate the remaining plain text content in the Markdown format except for the image information, and use a character splitter to divide it into blocks to obtain plain text blocks; In step A3, let D be the complete Markdown document generated by step A2; this document is a sequence of basic elements arranged in order, i.e. , where each element It can be one of three elements: text paragraph, heading, or image tag; The segmentation process first identifies and extracts all image tags; the image tag set P is defined as: ; For each identified image tag It is directly divided into separate text blocks and treated as an independent text block. Use functions This operation means: ; After extracting all image tags, the remaining plain text element sequence is obtained. Will All plain text elements are concatenated in their original order to form a continuous plain text string A, i.e. Then, A is divided into blocks using a character splitter. The splitter divides A into plain text blocks using a sliding window based on a preset block size L and overlap length O. Let the partitioning function be... This function returns a sequence of text blocks. For the j-th text block, its content is a substring of string A, that is... ,in, Indicates the starting position. Indicates the end position; The specific calculation method is as follows: The specific calculation method is as follows: By employing a hierarchical block segmentation strategy, two clearly distinguishable sets of text blocks are ultimately obtained: Collection of image-based text blocks: ; Collection of plain text blocks: ; The final block division result B of the entire document D is the union of these two types of blocks, that is: A4: Perform vector embedding on all text blocks obtained in A3 to generate vectors corresponding to each text block; store the type of each text block as metadata, associated with the text block itself and its corresponding vector, in the vector database; In A4, a pre-trained text embedding model is used to process all text blocks obtained in A3. Vectorization is performed separately; let the embedding model be a function. Map a text string to a A 1-dimensional real vector for each text block in set B. Calculate the corresponding vector representation : in, Represented as a text block The string content; Whether it's an image-based text block or a plain text block, they are all uniformly converted into a point in a high-dimensional space through the same embedding model; The system adds metadata when storing each text block; for each text block Extract key attributes to construct a metadata object, i.e.: Type: Directly records whether the text block is of type "img", "text", or other types; Source block content: Stores the raw string of the text block. ; Other optional information: such as the page number and chapter information of this block in the original PDF; Subsequently, the system represents the text block as a vector. It is closely related to its corresponding metadata, forming a searchable unit. ; Finally, all searchable units Batch store into the vector database.

[0020] A5: Receives user query requests and returns the text block with the highest relevance to the user query from the vector database based on vector similarity. During the generation phase, depending on whether a single-modal or multi-modal language model is invoked, it utilizes the text information in the returned text block and the image information / image description information corresponding to the image source URL to construct a detailed multi-modal context and generate an answer to the user query. In step A5, when a user query is received, the retrieval phase begins. First, the same text embedding model as in step A4 is used to convert the user's query request in natural language into a query vector in a high-dimensional vector space. ,Right now Then, from the vector database constructed using the A4 steps, the knowledge fragments most relevant to the user's question are located; Specifically, the system initiates an approximate nearest neighbor search in the vector database, which is performed in a deep semantic space. The algorithm calculates the cosine similarity or inner product distance metric between the query vector and all stored text block vectors in the database, and quickly finds h sets of text blocks that are most semantically relevant to the query from the database. The text blocks are then sorted based on similarity scores, and the top-K text blocks that are semantically related to the user's query are finally selected. The specific calculation method is as follows: Based on the modality the user actually uses in the large model, the system automatically switches the returned image modality. If the user selected a multimodal large model when generating the question, then blocks with image attributes will only send the URL link, while blocks with text attributes will send the token in its original form, preserving as much multimodal image information as possible while saving unnecessary tokens. If a pure language large model is used, the large model answers and operates based on the text description and image URL; it determines whether to enable metadata filtering based on the query context; for example, when the user explicitly asks "Please show and explain the flowchart on page 3", the system pre-limits the search scope to text blocks of type "image" and page number "3"; finally, the set of most relevant text blocks, as context information, is passed to the next stage. The retrieved collection of text blocks Contextual hints are assembled into a large language model; if a unimodal language model is invoked, the model will directly read and understand it. The text block contains the string content; for image-based text blocks, the model uses the "description" section in Markdown format, i.e., the semantically rich image text description generated by the multimodal model, to understand the image content and generate an answer accordingly. If a multimodal language model is invoked, the system provides all text content to the model and actively parses the image source URLs contained in the image-type text blocks. The model directly accesses the original image pointed to by the URL through the API, using the image's pixel information along with the text information as context. This allows the model to combine visual details and text semantics to generate an answer.

[0021] Please see Figure 2 As shown, this invention provides a knowledge embedding and retrieval system for multimodal PDF documents, including: Multimodal Information Extraction and Localization Module: A1: Employs a multimodal model to identify non-text elements in multimodal PDF documents, generates relative position coordinates based on the four corners of the rectangular outline of the non-text elements, converts images into text descriptions to preserve semantic information, and stores the source URL of the images; Format conversion module: Integrates the original text from the multimodal PDF document, the image text descriptions obtained from the multimodal information extraction and location module, and the image source URLs, and outputs them in Markdown format; The hierarchical text segmentation module divides each image information in the generated Markdown format into a separate text block; it integrates the remaining plain text content in the Markdown format excluding image information, and uses a character splitter to segment it into plain text blocks. Vectorization and storage module: Vector embedding is performed on all text blocks obtained from the hierarchical text segmentation module to generate vectors corresponding to each text block; the type of each text block is used as metadata, associated with the text block itself and its corresponding vector, and stored in the vector database; The retrieval and adaptation generation module receives user query requests and returns the text blocks with the highest relevance to the user query from the vector database based on vector similarity. During the generation phase, depending on whether a single-modal or multi-modal language model is invoked, it utilizes the text information in the returned text blocks and the image information / image description information corresponding to the image source URL to construct a detailed multi-modal context and generate an answer to the user query. Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for knowledge embedding and retrieval in multimodal PDF documents, characterized in that... ,include: A1: A multimodal model is used to identify non-text elements in multimodal PDF documents. Relative position coordinates are generated based on the four corners of the rectangular outline of the non-text elements. At the same time, the images are converted into text descriptions to preserve semantic information, and the source URL of the images is stored. A2: Integrate the original text from the multimodal PDF document, the image text descriptions obtained in A1, and the image source URLs, and output them in Markdown format; A3: Divide each image information in the Markdown format generated in A2 into a separate text block; integrate the remaining plain text content in the Markdown format except for the image information, and use a character splitter to divide it into blocks to obtain plain text blocks; A4: Perform vector embedding on all text blocks obtained in A3 to generate vectors corresponding to each text block; store the type of each text block as metadata, associated with the text block itself and its corresponding vector, in the vector database; A5: Receives user query requests and returns the text block with the highest relevance to the user query from the vector database based on vector similarity. During the generation phase, depending on whether a single-modal or multi-modal language model is invoked, it utilizes the text information in the returned text block and the image information / image description information corresponding to the image source URL to construct a detailed multi-modal context and generate an answer to the user query.

2. The knowledge embedding and retrieval method for multimodal PDF documents according to claim 1, characterized in that... In A1, a multimodal model integrating visual information understanding and semantic generation functions performs full-domain parsing of multimodal PDF documents. This model scans the document page by page and uses a visual attention mechanism to identify non-text elements, including images and tables. For images, the model distinguishes between subcategories such as photos, charts, and diagrams; for tables, it identifies their border structure, cell distribution, and merge status. In the localization process, a two-dimensional coordinate system is established with the top left corner of the PDF page as the origin. The pixel coordinates of the four vertices of the rectangular outline of each non-text element are extracted, and the pixel coordinates are mapped to normalized coordinates relative to the page size through a coordinate transformation algorithm.

3. The knowledge embedding and retrieval method for multimodal PDF documents according to claim 1, characterized in that... In section A2, Markdown syntax is used as the carrier in the format conversion stage to construct a relational system of "original text-image semantics-visual resources"; firstly, the original text in the multimodal PDF is structurally sorted and converted into the corresponding Markdown format according to the inherent logic of the document; When integrating the image elements output from the A1 stage, the Markdown standard image syntax is adopted. The "image description information" uses the structured text description generated in A1, and the "image source URL" embeds a complete link containing the UUID and access permissions. At the same time, the location information generated in A1 is implicitly associated with the image syntax through Markdown comments.

4. The knowledge embedding and retrieval method for multimodal PDF documents according to claim 1, characterized in that... In step A3, let D be the complete Markdown document generated by step A2; this document is a sequence of basic elements arranged in order, i.e. , where each element It can be one of three elements: text paragraph, heading, or image tag; The segmentation process first identifies and extracts all image tags; the image tag set P is defined as: ; After extracting all image tags, the remaining plain text element sequence is obtained. Will All plain text elements are concatenated in their original order to form a continuous plain text string A, i.e. Then, A is divided into blocks using a character splitter. The splitter recursively divides A into plain text blocks based on a preset block size L and a sliding window size O.

5. The knowledge embedding and retrieval method for multimodal PDF documents according to claim 4, characterized in that... Let the partitioning function be... This function returns a sequence of text blocks. For the j-th text block, its content is a substring of string A, that is... ; By employing a hierarchical block segmentation strategy, two clearly distinguishable sets of text blocks are ultimately obtained: Collection of image-based text blocks: ; Collection of plain text blocks: ; The final block division result B of the entire document D is the union of these two types of blocks, that is: .

6. The knowledge embedding and retrieval method for multimodal PDF documents according to claim 1, characterized in that... In A4, a pre-trained text embedding model is used to process all text blocks obtained in A3. Vectorization is performed separately; let the operation of the embedding model on the text block be a function. Map a text string to a A 1-dimensional real vector for each text block in set B. Calculate the corresponding vector representation The system adds metadata when storing each text block; for each text block Extract key attributes to construct a metadata object.

7. The knowledge embedding and retrieval method for multimodal PDF documents according to claim 1, characterized in that... In step A5, when a user query is received, the retrieval phase begins. First, the same text embedding model as in step A4 is used to convert the user's query request in natural language into a query vector in a high-dimensional vector space. ,Right now Then, from the vector database constructed using the A4 steps, the knowledge fragments most relevant to the user's question are located; The specific approach is as follows: The system initiates an approximate nearest neighbor search in the vector database, and the search is performed in a deep semantic space; the algorithm calculates the cosine similarity or inner product distance metric between the query vector and all stored text block vectors in the database, and quickly finds h sets of text blocks that are most semantically relevant to the query from the database. The text blocks are then sorted based on similarity scores, and the top-K text blocks that are semantically related to the user's query are finally selected. The specific calculation method is as follows: ； Determine whether to enable metadata filtering based on the context of the query.

8. A knowledge embedding and retrieval system for multimodal PDF documents, using a knowledge embedding and retrieval method for multimodal PDF documents as described in any one of claims 1-7, characterized in that: Multimodal Information Extraction and Localization Module: A1: Uses a multimodal model to identify non-text elements in multimodal PDF documents, generates relative position coordinates based on the four corners of the rectangular outline of the non-text elements, converts the images into text descriptions to preserve semantic information, and stores the source URL of the images; Format conversion module: integrates the original text in the multimodal PDF document, the image text descriptions obtained from the multimodal information extraction and location module, and the image source URLs, and outputs them in Markdown format; The hierarchical text segmentation module divides each image information in the generated Markdown format into a separate text block; it integrates the remaining plain text content in the Markdown format excluding image information, and uses a character splitter to segment it into plain text blocks. Vectorization and storage module: Vector embedding is performed on all text blocks obtained in the hierarchical text segmentation module to generate vectors corresponding to each text block; the type of each text block is used as metadata, and after being associated with the text block itself and the corresponding vector, it is stored in the vector database. The retrieval and adaptation generation module receives user query requests and returns the text blocks with the highest relevance to the user query from the vector database based on vector similarity. During the generation phase, depending on whether a single-modal or multi-modal language model is invoked, the text information in the returned text blocks and the image information / image description information corresponding to the image source URL are used to construct a detailed context of multi-modal information and generate an answer to the user query.

Images