Document knowledge base retrieval method based on multi-modal information matching

Abstract

The invention discloses a document knowledge base retrieval method based on multi-modal information matching, and the method specifically comprises the following steps: firstly, constructing a cross-modal feature extraction framework by using a pre-trained multi-modal model JinaCLIP, carrying out feature coding on multi-modal entities such as texts, images, tables and the like in a document, generating diversified text descriptions through a text Prompt module (Text Prompt), and carrying out feature extraction on the multi-modal entities; and semantic expression of text input and task adaptability are enhanced. Then, an entity-level multi-modal retrieval data set D4LA-MIR is built on the basis of a document layout analysis result, text-text comparison loss or graph-text comparison loss is dynamically selected according to entity types through a modal self-adaptive comparison loss mechanism, and the cross-modal semantic alignment effect is optimized. And finally, calculating the cosine similarity of the text features and the visual features through an image-text matching module (Image-Text Matching, ITM), and carrying out weighted fusion in combination with the similarity between texts to obtain a final retrieval result. According to the method, the problems that traditional document retrieval only depends on text information, and non-text modal semantic association is difficult to capture are solved, and the retrieval precision and robustness in a document knowledge base scene are remarkably improved through multi-modal feature fusion and adaptive loss optimization.

Description

Technical Field

[0001] This application relates to the interdisciplinary field of computer vision and natural language processing, and in particular to a document knowledge base retrieval method based on multimodal information matching. It is applicable to document knowledge base retrieval scenarios that include multimodal entities such as text, images, and tables, such as academic literature databases, enterprise knowledge bases, and intelligent document management systems. Background Technology

[0002] Document knowledge base retrieval is a core task in the field of information retrieval, aiming to quickly locate and recall content highly relevant to user queries from large-scale document collections. Traditional document retrieval methods mainly rely on plain text information, with typical solutions including the BM25 algorithm based on keyword matching and the bge-large-v1.5 model based on text embedding similarity. These methods can only capture semantic associations of text modalities and are difficult to handle complex documents containing non-text modalities such as images, tables, and formulas. For example, when a user queries "a product structure table displayed in the document," traditional methods cannot associate the visual features of the table with the query semantics, resulting in a significant decrease in retrieval accuracy. While existing multimodal retrieval technologies (such as the original CLIP model) can achieve cross-modal matching, they are primarily geared towards natural image and plain text scenarios, lacking adaptation to document layout structure and entity relationships. On one hand, the pre-training data for natural image modalities differs significantly from the visual features of tables and charts in documents, resulting in poor feature transfer performance. On the other hand, public datasets (such as COCO and Flickr30K) are mostly designed for "image-text" pairing, which is insufficient to meet the "fine-grained entity-query" retrieval needs of document knowledge bases, leading to data scenario mismatch issues. Therefore, constructing a multimodal retrieval framework adapted to document scenarios has become a key challenge in improving the retrieval performance of document knowledge bases. Summary of the Invention

[0003] This application provides a document knowledge base retrieval method based on multimodal information matching. Through three core modules—multimodal feature extraction, dedicated dataset construction, and adaptive loss optimization—it addresses the technical pain points of traditional methods. The specific technical solution is as follows: In a first aspect, embodiments of this application provide a multimodal feature extraction and semantic enhancement method, the method comprising: The pre-trained multimodal model JinaCLIP is selected as the basic framework, and its pre-trained weights are used to initialize the image encoder and text encoder. JinaCLIP, based on CLIP, unifies the two retrieval tasks of "text-to-image search" and "text-to-text search" through larger-scale cross-modal training data (including some document-type data) and optimization strategies. It has stronger cross-modal semantic alignment capabilities and can directly adapt to the multimodal feature extraction needs of document scenarios. The document is analyzed for layout, and multimodal entities such as text, images, and tables are extracted. For text entities (such as paragraphs, headings, and lists), complete text content is extracted using OCR technology to ensure that no semantic information is lost. For non-text entities such as images and tables, accurate text descriptions are generated using VLLM (Large Language Model) (e.g., "A bar chart table showing product sales in 2023, including product name, quarterly sales, and year-on-year growth rate"). At the same time, the coordinate position information and modal type identifiers (such as "Table" and "Figure") of each entity in the document are retained to provide a basis for subsequent modal adaptive optimization. A text prompt module is designed to enhance semantic expression: It utilizes a large model to generate diverse prompt templates, with template design incorporating document context characteristics, such as "Based on the document content, the text paragraph related to {query} is" or "Describe the image / table features in the document corresponding to {query}," etc. The user's original query is substituted into the template, generating 3-5 semantic variations to semantically expand the query. The expanded query text and entity text are encoded using JinaCLIP's text encoder, and the entity visual information is encoded using an image encoder, resulting in consistent text and visual features, laying the foundation for cross-modal matching. Secondly, this application also provides a method for constructing a multimodal retrieval dataset for entity-level documents: Based on the layout results of the document layout analysis dataset (such as document region segmentation and entity type annotation), the original document is divided into multiple fine-grained patches: each patch corresponds to an independent multimodal entity in the document (such as a piece of text, a table, or an image), and retains the original image information (such as the pixel matrix of the table and the original resolution of the image), position information (such as the coordinate range in the document page), and text content (text entities are directly extracted, and non-text entities are described by generating descriptions through VLLM), ensuring the integrity and semantic independence of the patch. LLM is used to generate related query questions for the core information of each patch: for example, for the patch "a table containing sales figures for Q1-Q4 of 2023", the query "the table content in the document that records the sales figures for each quarter of 2023" is generated; for the patch "images showing the appearance of the product", the query "the descriptive information corresponding to the images in the document that show the appearance of the product" is generated. Finally, sample pairs of "multimodal entity patch - related query" are constructed to form the entity-level multimodal retrieval dataset D4LA-MIR for document knowledge base scenarios. The dataset was partitioned and distributed as follows: it was randomly divided into training and test sets at a ratio of 9:1, with a total of 26,744 samples (24,153 in the training set and 2,591 in the test set). The layout and category distribution of the training and test sets were kept uniform, covering 27 entity types, including ParaText (30.08% / 26.40%), OtherText (11.96% / 12.81%), RegionKV (10.60% / 12.27%), DocTitle (6.30% / 6.52%), Table (2.36% / 3.32%), and Figure (2.23% / 2.05%). This design avoided model bias caused by class imbalance and ensured the dataset's coverage and representativeness for document knowledge base scenarios. Thirdly, embodiments of this application also provide a modality adaptive contrastive loss mechanism and a similarity matching module: A modality-adaptive contrastive loss mechanism is designed to achieve modality-specific optimization: During the model training phase, the entity category is first determined by the modality type identifier of the entity (such as "Text", "Table", "Figure"). For text entities, a text-to-text contrastive loss is used—the cosine distance between the query text features and the text entity features is calculated, minimizing the feature distance between similar entities (query and matching text entities) and maximizing the feature distance between dissimilar entities (query and non-matching text entities). For non-text entities such as images and tables, an image-to-text contrastive loss is used—the cosine distance between the query text features and the visual features of non-text entities is calculated, optimizing the alignment effect between visual features and query semantics. The two losses are balanced by parameter weights. Design a high-precision similarity calculation process: For a given sample It contains the original image. Text content and user issues Entity-level multimodal retrieval in document knowledge base scenarios aims to address each user's question. From the sample set The most relevant content was recalled. To this end, the user question was presented in text format. Data augmentation using large models yields diverse representations, which are then integrated with the text content. via text encoder Encode them as features respectively and On the image end, the original image via image encoder Encoding as features Finally, the similarity matching module calculates the results separately. and as well as and The similarity is used to determine the search results. Before calculating the final similarity matrix, text and video features are processed using one-dimensional convolution and L2 norm normalization. The one-dimensional convolutional layer aligns and enhances the features, maintaining consistency between the two feature dimensions and providing richer content representations. The normalization layer standardizes the length of the feature vectors, making the similarity calculation focus more on directional differences rather than size differences, thereby improving the model's generalization ability and reducing overfitting. Fourthly, embodiments of this application also provide a multimodal training and testing framework for document entity information matching, the framework specifically including the following: The document is split into multiple modal entities, such as text, images, and tables, and the user query is expanded by the text prompt module. These are then input into JinaCLIP's image encoder and text encoder, respectively, to extract the visual features, text features, and query features of the multiple modal entities. One-dimensional convolutional layers are used to align textual and visual features and enhance their details. L2 norm normalization is then applied to standardize the feature vector length, eliminating the interference of feature scale differences on the matching results. By employing dot product operations with a dynamic temperature coefficient, the text-to-text matching similarity between query features and entity text features, and the image-to-text matching similarity between query features and entity visual features are calculated separately, resulting in a bimodal similarity score matrix. During model training, entity features and query features are input into the modality-adaptive contrastive loss module. The module dynamically selects between text-to-text and image-to-text contrastive losses based on the entity modality type, balancing the two types of losses through parameter weights to optimize cross-modal semantic alignment. A total loss function is constructed based on the similarity score matrix and the modality-adaptive contrastive loss. A DropOut strategy is introduced to randomly discard text prompt templates to avoid overfitting. The model parameters are iteratively optimized through backpropagation using the AdamW optimizer. During model inference, the text-to-text matching score and the image-to-text matching score are weighted and fused according to preset weights to generate a comprehensive similarity score. Top-N candidate results are recalled in descending order of score. Non-maximum suppression (NMS) is used to remove duplicate results with high overlap, and high-confidence results are selected as the final retrieval output. The entire process revolves around the precise extraction of multimodal features, adaptive optimization of submodal features, and fusion of multidimensional similarity. It achieves deep integration of textual and visual modal information, effectively improving the accuracy, robustness, and scene adaptability of document knowledge base retrieval. The technical solutions provided in this application embodiment may have the following beneficial effects: This application proposes a document knowledge base retrieval method based on multimodal information matching. This method, through a complete process design including a self-built entity-level multimodal retrieval dataset, precise multimodal feature extraction, modal adaptive optimization, and multi-dimensional similarity fusion, not only addresses the pain point of traditional pure text retrieval's inability to capture non-textual modal semantic relationships, but also overcomes the limitations of general multimodal methods in terms of insufficient document scenario adaptability and the lack of fine-grained entity support in existing datasets. This significantly improves the accuracy, robustness, and scenario adaptability of document knowledge base retrieval. Firstly, addressing the mismatch between existing datasets and the "fine-grained entity retrieval" requirements of document knowledge bases, a self-built entity-level multimodal retrieval dataset, D4LA-MIR, is constructed. Through fine-grained patch partitioning, precise pairing of "multimodal entity-association queries," and a uniform category distribution design, high-quality data support that fits real-world scenarios is provided for model training, filling the gap in multimodal retrieval datasets for document knowledge bases. Secondly, the multimodal entities (text, images, tables, etc.) derived from the document splitting, along with the user query expanded by the text prompt module, are input into the JinaCLIP encoder to extract high-quality visual, textual, and query features. Semantic expansion strengthens the correlation between the query and the entities. Next, a modal adaptive contrastive loss mechanism is used to dynamically select optimization strategies based on entity type, balancing text-to-text and image-to-text losses to achieve accurate cross-modal semantic alignment. Then, one-dimensional convolutional feature alignment, L2 norm normalization, and dynamic temperature coefficient dot product operations are used to eliminate feature scale interference and improve the discriminative power of similarity calculations. During model training, DropOut regularization and adaptive optimizer iteratively optimize parameters to enhance model generalization ability; during inference, weighted fusion of dual-modal scores and NMS deduplication ensure the accuracy and simplicity of retrieval results. Finally, through the dual empowerment of dataset construction and multi-module collaborative optimization, deep integration of textual and visual modal information is achieved, providing a data-supported and stable retrieval solution for the document knowledge base. Attached Figure Description Figure 1This is a structural block diagram of a document knowledge base retrieval method based on multimodal information matching provided in an embodiment of this application; Figure 2 This is a structural block diagram of a text prompt module based on a large model for data augmentation and semantic expansion, provided in an embodiment of this application. Figure 3 This is a structural block diagram of a pre-trained multimodal feature extractor for visual and textual feature extraction, provided in an embodiment of this application. Figure 4 This is a structural block diagram of a similarity matching module provided in an embodiment of this application; Detailed Implementation

[0004] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings. Exemplary embodiments will be described in detail here, examples of which are illustrated in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods consistent with some aspects of this application as detailed in the appended claims. In developing this application, the inventors discovered that both existing unimodal and multimodal retrieval methods have significant limitations when handling document knowledge base retrieval tasks. Unimodal methods rely solely on textual information for keyword matching or semantic embedding, completely failing to capture the semantic relationships between non-textual modalities such as images and tables in the document and the query, resulting in limited retrieval accuracy. Existing multimodal methods are primarily geared towards natural image and ordinary text scenarios, lacking adaptation to document layout structures and fine-grained entity relationships. Furthermore, publicly available datasets (such as COCO and Flickr30K) are mostly designed with "image-text" page-level pairings, making it difficult to meet the "fine-grained entity-query" retrieval needs of document knowledge bases, resulting in a mismatch between data scenarios and task requirements. In addition, existing methods only achieve cross-modal alignment through pre-training without designing specific optimization strategies for document scenarios, leading to poor semantic matching results for non-textual entities and difficulty in supporting high-quality document knowledge base retrieval. To address the aforementioned technical pain points, the inventors of this application have attempted an innovative design across the entire process, from dataset construction and feature extraction to loss optimization and similarity calculation. This involves building a self-constructed entity-level multimodal dataset tailored to the task scenario to solve the problem of insufficient data support; enhancing semantic expression through a text prompt module to improve the relevance of cross-modal matching; and achieving precise modality-specific optimization through a modality-adaptive contrastive loss mechanism. Ultimately, this combined approach significantly improves the accuracy and robustness of document knowledge base retrieval, demonstrating the inventors' creative efforts. To address related technical issues, this application provides a document knowledge base retrieval method based on multimodal information matching. The method includes: First, addressing the gaps in existing datasets, constructing the entity-level multimodal retrieval dataset D4LA-MIR—based on document layout analysis results, splitting the original document into fine-grained entity patches, directly extracting content from text entities, generating precise text descriptions for non-text entities using VLLM, and then generating related queries using LLM to form "multimodal entity-query" sample pairs. The training and test sets are divided at a 9:1 ratio to ensure even category distribution and sufficient sample size. Second, inputting the multimodal entities from the split document and the user query expanded by the text prompt module into JinaCLIP's image encoder and text encoder respectively, extracting visual features, text features, and query features, and enhancing the semantic expression of the query through diversified prompt templates. Next, the three types of features are input into the modality adaptive contrastive loss module. Based on the entity type, text-to-text or image-to-text contrastive loss is dynamically selected. The two types of losses are balanced through parameter weights to optimize cross-modal semantic alignment. Simultaneously, a one-dimensional convolutional layer completes feature dimension alignment and detail enhancement. Feature vectors are normalized using L2 norm, and dot product operations with a dynamic temperature coefficient are used to calculate the bimodal similarity score. During model training, a total loss function is constructed based on the similarity score matrix and the modality adaptive contrastive loss. A DropOut strategy is introduced to prevent overfitting, and the model parameters are iteratively optimized through backpropagation using the AdamW optimizer. During inference, the text-to-text matching score and the image-to-text matching score are weighted and fused. Top-N candidate results are recalled by score ranking, and non-maximum suppression (NMS) is used for deduplication to obtain the final retrieval results. As can be seen, this solution, through its innovative end-to-end process of "dedicated dataset construction + multimodal semantic enhancement + submodal adaptive optimization + precise similarity calculation," not only fully leverages the feature extraction advantages of pre-trained multimodal models but also specifically addresses the scenario adaptability issues of document knowledge base retrieval. It effectively compensates for the shortcomings of existing methods in non-textual modality utilization, data support, and cross-modal alignment, thereby improving the accuracy of fine-grained entity retrieval and enhancing the reliability and robustness of document knowledge base retrieval. The following is a description of some terms used in the embodiments of this application. JinaCLIP: A pre-trained multimodal model optimized based on CLIP, which unifies the "text-to-image" and "text-to-text" tasks through large-scale cross-modal training data, has stronger cross-modal semantic alignment capabilities, and is adapted to the multimodal feature extraction needs of document scenarios. Text Prompt module: By pre-setting diverse scenario templates, it expands the user's original query into multiple semantic variations, enhances the semantic expression and task adaptability of the query, and improves the relevance of cross-modal matching. Modality-adaptive contrastive loss: An optimization mechanism that dynamically selects the loss function based on the entity modality type (text / non-text). Text-to-text contrastive loss is used for text entities, and image-to-text contrastive loss is used for non-text entities. Weight balancing ensures balanced optimization across modalities. D4LA-MIR: This application presents a self-built entity-level multimodal retrieval dataset containing 26,744 samples covering 27 document entity types. It is presented in the form of "multimodal entity patch - related query" and is specifically adapted for fine-grained retrieval tasks of document knowledge bases. Non-maximum suppression (NMS): A post-processing technique used to deduplicate search results. By calculating the overlap (IoU) of candidate results, it retains high-confidence results and eliminates redundant and overlapping results, ensuring the simplicity and accuracy of the search output. Dynamic temperature coefficient: An adaptive parameter used for similarity calculation. Its value is dynamically adjusted according to the batch feature distribution, optimizing the output of dot product operation, improving the discriminative power of similarity scores, and reducing overfitting. The overall block diagram provided in the embodiments of this application will be described in detail below with reference to Figure 1. First, the semantics of the query are expanded using the text suggestion enhancement module. Text suggestion enhancement is a crucial preliminary step in improving the accuracy of cross-modal semantic matching. This application's embodiments construct a text suggestion module using a Large Language Model (LLM), designing diverse suggestion templates for document knowledge base retrieval scenarios, such as "The text paragraphs in the document related to {query} are" or "Describe the image / table features in the document corresponding to {query} are," etc. Substituting the user's original query into the template generates 3-5 semantic variations, achieving semantic expansion and task adaptation of the query. This process overcomes the limitations of traditional retrieval relying solely on a single query, providing richer semantic anchors for subsequent cross-modal matching through multi-perspective semantic expression. Secondly, feature extraction is performed using a pre-trained multimodal model. Pre-trained multimodal models possess powerful cross-modal semantic alignment capabilities, and this application selects JinaCLIP as the basic feature extraction framework. Based on CLIP, this model unifies "text-to-image" and "text-to-text" tasks through large-scale cross-modal training data and optimization strategies, directly adapting to the multimodal feature extraction needs of document scenarios. For multimodal entities such as text, images, and tables after document splitting, OCR technology is used to extract text entity content, and VLLM is used to generate text descriptions of non-text entities. Simultaneously, the expanded user query is input into the model, and visual features, text features, and query features of multimodal entities are extracted respectively. The feature extraction process fully utilizes the pre-trained weight advantages of JinaCLIP to ensure the alignment of text and visual modal features in the semantic space, laying the foundation for subsequent similarity calculations. Then, the extracted features are used to calculate similarity based on entity type. Similarity calculation is a core step in achieving multimodal retrieval. This embodiment uses a one-dimensional convolutional layer to align and enhance the details of text and visual features. L2 norm normalization standardizes the feature vector length, ensuring that similarity calculation focuses on differences in feature direction rather than size. Based on this, a dot product operation with a dynamic temperature coefficient is used to calculate the text-to-text similarity between query features and text entity features, and the image-to-text similarity between query features and non-text entity visual features. The dynamic temperature coefficient is dynamically adjusted according to the batch feature distribution, optimizing the dot product output, improving the discriminative power of the similarity score, and effectively supporting subsequent retrieval decisions. Finally, the loss function is dynamically selected during training, and the weighted score is used to recall the retrieval results during inference. This step achieves a closed-loop process for training and inference. During model training, the loss function is dynamically selected based on the entity modality type (text / non-text): text-to-text comparison loss is used for text entities to minimize the feature distance of similar texts and maximize the feature distance of dissimilar texts; image-to-text comparison loss is used for non-text entities such as images and tables to optimize the alignment between visual features and corresponding textual descriptions. The two losses are balanced by parameter weights, and a total loss function is constructed for backpropagation to optimize model parameters. A DropOut strategy is also introduced to prevent overfitting. During model inference, text-to-text matching scores and image-to-text matching scores are weighted and fused according to preset weights to generate a comprehensive similarity score. Top-N candidate results are recalled in descending order of score, and non-maximum suppression is used to remove duplicates of results with high overlap, ultimately outputting high-confidence retrieval results. The technical solutions provided in this application embodiment may have the following beneficial effects: This application proposes a document knowledge base retrieval method based on multimodal information matching. The scheme fully utilizes the semantic association information of multimodal entities in the document through a full-process design of text prompt semantic enhancement, pre-trained multimodal feature extraction, modality adaptive contrastive loss optimization and bimodal similarity fusion. At the same time, it leverages the cross-modal alignment capability of the pre-trained multimodal model, which significantly improves the accuracy and robustness of document knowledge base retrieval. In the semantic enhancement stage, a variety of prompt templates are generated using a large language model to semantically expand user queries, breaking through the semantic limitations of single queries and providing multi-perspective semantic anchors for cross-modal matching, thereby enhancing the semantic relevance between queries and document entities. In the feature extraction stage, the pre-trained multimodal model JinaCLIP is used to extract features from the document's multimodal entities and the enhanced query. By leveraging the model's cross-modal pre-trained weights, the alignment of textual and visual features in a unified semantic space is ensured, laying a high-quality feature foundation for subsequent similarity calculations. In the loss optimization and similarity calculation stages, a modality-adaptive contrastive loss mechanism is used to dynamically select text-to-text or image-to-text contrastive loss based on entity type, achieving precise optimization across modalities. At the same time, through one-dimensional convolutional feature alignment, L2 norm normalization, and dynamic temperature coefficient dot product operation, feature scale interference is eliminated, improving the discriminativeness and reliability of similarity scores. During the retrieval reasoning stage, text-to-text matching scores and image-to-text matching scores are weighted and fused, and non-maximum suppression (NMS) is used for deduplication to ensure the accuracy and simplicity of the retrieval results. The combination of offline feature encoding and online fast matching meets the real-time retrieval needs of large-scale document knowledge bases. In summary, this method, through the synergistic effect of multiple modules, fully leverages the complementarity of multimodal information in documents, achieving accurate cross-modal semantic matching. This innovative approach demonstrates significant advantages in improving the accuracy of document knowledge base retrieval and enhancing the model's ability to perceive multimodal entities, providing an efficient and feasible solution for the field of document knowledge base retrieval. It should be understood that although the steps have been described above, they are not necessarily to be performed in the order described above. Unless otherwise expressly stated herein, there is no strict order in which these steps are performed, and they may be performed in other orders. Any reasonable order is within the scope of protection of this application, and will not be described in detail here. Moreover, at least some steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps. Corresponding to the above method embodiments, this application also provides a structural block diagram of a text prompt module based on a large language model for data augmentation and semantic expansion, as shown in Figure 2: First, the user's original query input text suggestion module generates 3-5 semantic variants based on a preset document scenario template and the generation capabilities of a large language model. These variants expand the semantic expression of the query from different perspectives. For example, for the query "2023 product sales table", variants such as "Based on this document, query the table content that records the product sales in 2023" and "Describe the characteristics of the 2023 product sales table in the document" can be generated. Subsequently, the generated query variants and multimodal entities (text, images, tables, etc.) in the document are input into the pre-trained multimodal model, providing rich semantic input for cross-modal feature matching. This process fully utilizes the semantic understanding and generation capabilities of large language models, breaking through the semantic limitations of single queries, making subsequent cross-modal matching more targeted and comprehensive, and effectively improving the semantic relevance between multimodal entities and queries. In addition, this embodiment of the method also provides a structural block diagram for visual and textual feature extraction using a pre-trained multimodal feature extractor, as shown in Figure 3: Visual features refer to the features extracted from the original document image by the JinaCLIP image encoder, while text features refer to the features extracted from text entity content, non-text entity text descriptions, and user queries by the JinaCLIP text encoder. This feature extraction process is based on the JinaCLIP pre-trained model, which uses a ViT-B / 16 architecture for the image encoder and a Transformer architecture for the text encoder. Pre-trained weight initialization ensures cross-modal alignment of the extracted features. After analyzing the document layout, the extracted text entities are extracted using OCR technology, while non-text entities such as images and tables are described using VLLM. Simultaneously, the expanded user query is encoded using a text encoder. Ultimately, the model outputs consistent text and visual features (512 dimensions in this example), providing a unified semantic space for subsequent similarity calculations. This process fully leverages the feature extraction advantages of the pre-trained multimodal model, ensuring semantic alignment between text and visual modal features, which is fundamental to achieving accurate multimodal retrieval. Fourthly, embodiments of this application also provide a structural block diagram of the similarity matching module, see below for details. Figure 4 : Before calculating the final similarity matrix, text and video features are processed using one-dimensional convolution and L2 norm normalization. The one-dimensional convolutional layer aligns and enhances the features, maintaining consistency between the two feature dimensions and providing richer content representation. The normalization layer standardizes the length of the feature vectors, making the similarity calculation focus more on directional differences rather than magnitude differences, thereby improving the model's generalization ability and reducing overfitting. The similarity calculation process can be represented as: in, It is a dynamic temperature coefficient used to scale the output of the dot product operation to optimize model performance. It is a similarity matrix. and These are the normalized text and visual features, respectively. M represents the batch size during training and is 1 during testing; N represents the number of entities in the knowledge base; and D represents the feature dimension. Finally, refer to Figure 4 The similarity matrix obtained by the similarity matching module is used to complete the training and inference of the model. During model training, entity features and query features are input into the modality-adaptive contrastive loss module. The module dynamically selects between text-to-text and image-to-text contrastive losses based on the entity modality type, balancing the two types of losses through parameter weights to optimize cross-modal semantic alignment. A total loss function is constructed based on the similarity score matrix and the modality-adaptive contrastive loss. A DropOut strategy is introduced to randomly discard text prompt templates to avoid overfitting. The model parameters are iteratively optimized through backpropagation using the AdamW optimizer. The total loss function can be expressed as: in Represents the image-text similarity matrix. Let N represent the text-to-text similarity matrix, N represent the batch size, and M represent the set of entities containing the text. This represents the parameter weights, used to balance the two contrast losses. During model inference, the text-to-text matching score and the image-to-text matching score are weighted and fused according to preset weights to generate a comprehensive similarity score. Top-N candidate results are recalled in descending order of score. Non-maximum suppression (NMS) is used to remove duplicate results with high overlap, and high-confidence results are selected as the final retrieval output. This module achieves accurate matching between multimodal entities and queries through feature alignment, standardization, and dynamic similarity calculation, and is one of the core technical aspects of improving the retrieval accuracy of this patent. It should be understood that "multiple" as mentioned herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, the step numbers described herein only exemplify one possible execution order between steps. In some other embodiments, the steps may not be executed in the order shown in the numbers, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this. The above embodiments can also be combined arbitrarily, and the combination schemes will not be elaborated here. The above are merely exemplary embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application shall be included within the protection scope of this application.

Claims

1. A document knowledge base retrieval method based on multimodal information matching, characterized in that, This includes multimodal feature extraction and semantic enhancement methods, specifically: The pre-trained multimodal model JinaCLIP is selected as the basic framework, and its pre-trained weights are called to initialize the image encoder and text encoder. JinaCLIP is based on CLIP and unifies the "text-to-image" and "text-to-text" tasks through large-scale cross-modal training data and optimization strategies, and has cross-modal semantic alignment and retrieval capabilities. The document is analyzed for layout and broken down into multimodal entities such as text, images, and tables. Text entities are extracted using OCR technology, and non-text entities such as images and tables are generated with corresponding text descriptions using VLLM. At the same time, the position information and modal type identifier of each entity in the document are preserved. The design incorporates a text prompt module that utilizes a large model to generate diverse prompt templates (such as "According to the document, {query}"), combining user queries with document context information to achieve semantic expansion and task adaptation of text input. JinaCLIP's text encoder and image encoder are then used to encode the expanded text and entity visual information respectively, obtaining text and visual features that provide a feature foundation for cross-modal matching. This process fully leverages the cross-modal representation advantages of the JinaCLIP pre-trained model and the semantic enhancement capabilities of the text prompt module. It avoids the shortcomings of traditional pure text models in capturing visual modal semantics, and strengthens the semantic relevance between queries and document entities through contextualized prompt templates. This provides high-quality, strongly correlated feature support for subsequent cross-modal matching, effectively improving the feature representation accuracy of multimodal entities in document knowledge base scenarios.

2. The document knowledge base retrieval method based on multimodal information matching as described in claim 1, characterized in that, Including methods for constructing entity-level multimodal retrieval datasets: Based on the layout results of the document layout analysis dataset, the original document is divided into multiple fine-grained patches. Each patch retains the corresponding image information, location information, and text content. The text content is extracted from text entities using OCR and generated from images and table entities using VLLM. By using LLM to generate related query questions based on the core information of each patch, sample pairs of "multimodal entity patch-related query" are constructed, forming the entity-level multimodal retrieval dataset D4LA-MIR for document knowledge base scenarios. The dataset is divided into a training set and a test set in a 9:1 ratio, with a total number of samples of no less than 26,000. The layout categories of the training set and the test set are evenly distributed, covering at least 8 entity types such as ParaText, Table, Figure, and DocTitle, to meet the training and testing requirements of the fine-grained retrieval model. This process overcomes the limitations of existing retrieval datasets, which are mostly geared towards understanding natural images or page-level documents. Through fine-grained patch partitioning and precise "entity-query" matching, the dataset better meets the real needs of "fragment-level retrieval" in document knowledge bases. At the same time, the uniform class distribution and sufficient sample size avoid class bias problems during model training, providing a high-quality training data foundation with strong scenario adaptability for multimodal retrieval models, and ensuring the model's generalization ability in complex document scenarios.

3. The document knowledge base retrieval method based on multimodal information matching as described in claim 1, characterized in that, This includes modality-adaptive contrastive loss mechanisms and similarity calculation methods: During the model training phase, the loss function is dynamically selected based on the entity modality type: text-to-text comparison loss is used for text entities to minimize the feature distance of similar text entities and maximize the feature distance of dissimilar text entities; image-to-text comparison loss is used for non-text entities such as images and tables to optimize the alignment effect between the visual features of non-text entities and their corresponding text description features; the two losses are balanced by parameter weights to avoid the bias of a single loss in optimizing specific modal entities. In the similarity calculation stage, a one-dimensional convolutional layer is used to align and enhance text features and visual features, ensuring consistency between the two feature dimensions and enriching content representation. The feature vector length is standardized by L2 norm normalization, so that the similarity calculation focuses on the difference in feature direction rather than the difference in size. The feature similarity is calculated by dot product operation with dynamic temperature coefficient, and the image-text matching score and text-text matching score are obtained respectively. The dynamic temperature coefficient is used to optimize the output of dot product operation, improve the model's generalization ability and reduce overfitting. This process achieves "differentiated optimization" for different types of entities through a modal adaptive loss mechanism, solving the problem of insufficient optimization of non-text entities by the traditional unified loss function. At the same time, the feature alignment of one-dimensional convolution and the standardization processing of L2 norm normalization eliminate the impact of feature dimension differences and scale interference on similarity calculation. The dynamic temperature coefficient further improves the discriminativeness of similarity scores, ultimately ensuring the accuracy and reliability of cross-modal matching results.

4. The document knowledge base retrieval method based on multimodal information matching as described in claim 1, characterized in that, This includes the training and inference framework for the retrieval model, specifically: The multimodal entity patch from the D4LA-MIR dataset is input into the image encoder and text encoder of JinaCLIP, which output visual feature vectors and text feature vectors, respectively. The user query is augmented by the text prompt module to generate diversified query expressions, which are then input into the text encoder to obtain the query feature vector. During the training phase, the feature vectors are input into the image-text matching module (ITM) to calculate the similarity matrix. The total loss is calculated by combining the modality adaptive contrastive loss function, and the model parameters are optimized through backpropagation. The DropOut strategy is introduced to randomly discard text prompt templates to prevent the model from overfitting to fixed templates and further improve the generalization ability. During the inference phase, the similarity between the query feature vector and the feature vectors of multimodal entities in the document knowledge base is calculated. The image-text matching score and the text-text matching score are weighted and fused, and the Top-N results are recalled in descending order of the fusion score. Non-maximum suppression (NMS) is used to remove duplicates from the recalled results, and high-confidence results are selected as the final retrieval output. This process achieves a balance between retrieval performance and efficiency through "targeted optimization during the training phase + efficient matching during the inference phase": the DropOut strategy during training effectively avoids the risk of template overfitting and ensures the model's adaptability to diverse queries; the weighted fusion and NMS deduplication during inference combine the advantages of multimodal matching while eliminating redundant and low-quality results, ensuring the accuracy and simplicity of the retrieval results; at the same time, the combination of offline feature encoding and online fast matching meets the real-time retrieval needs of large-scale document knowledge bases.

Images