Dynamic resolution visual language model training method, detection method and device

Abstract

The application provides a dynamic resolution visual language model training method, a detection method and equipment. The method comprises: acquiring a multi-modal document data set and a plurality of query text feature vectors; training a preset deep learning model so that the deep learning model detects each document image to generate a plurality of visual area graphs of different resolutions corresponding to each document image respectively in the training process, extracts and fuses each visual area graph to obtain an image feature vector, calculates a target loss based on the image feature vector and the corresponding query text feature vector to update the model parameters of the deep learning model, and obtains a trained deep learning model; and the deep learning model is used as a dynamic resolution visual language model for retrieving a document page corresponding to the query text feature vector from a plurality of document images. The application can effectively solve the problem that the traditional method ignores the complex layout and visual details in the document due to the fixed resolution input, thereby limiting the retrieval accuracy.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and multimodal information retrieval, and in particular to a dynamic resolution visual language model training method, detection method and device. Background Technology

[0002] With the deepening of digital transformation, document processing is gradually evolving from a single text format to a multimodal format that includes images, tables, charts, formulas, and more. Traditional document retrieval systems mostly rely on optical character recognition (OCR) technology to extract text content and then perform text-based searches. However, OCR technology suffers from reduced accuracy and fails to preserve the document's visual structure and semantic relationships when handling complex layouts, multi-column layouts, handwritten text, slanted text, or low-resolution images. Furthermore, since non-text elements such as charts, flowcharts, and mathematical formulas often carry important information, relying solely on text-based retrieval can lead to information loss, thus affecting the completeness and accuracy of the search results.

[0003] In recent years, document retrieval methods based on visual language models have gradually emerged. However, existing methods mostly adopt a fixed-resolution input approach, which makes it difficult to simultaneously preserve global layout information and local detailed features under limited computational resources. Especially for documents containing dense tables, small-font text, and complex charts, using a fixed low-resolution input approach leads to blurred details, affecting the model's capture of key information; while using a fixed high-resolution input approach significantly increases the computational burden and reduces retrieval efficiency.

[0004] Furthermore, existing cross-modal alignment methods are mostly based on global feature matching, lacking modeling of spatial and semantic relationships between entities within a document (such as headings, paragraphs, charts, and tables), which limits the model's performance when understanding complex document structures. Therefore, how to achieve multi-granular, high-fidelity document visual feature extraction under limited computational resources and establish an effective image-text alignment mechanism has become a key challenge to improving the performance of multimodal document retrieval. Summary of the Invention

[0005] In view of this, embodiments of this application provide a dynamic resolution visual language model training method, detection method, and device to eliminate or improve one or more defects existing in the prior art.

[0006] The first aspect of this application provides a method for training a dynamic resolution visual language model, the method comprising: Obtain a multimodal document dataset and feature vectors from multiple query texts; A preset deep learning model is trained based on the feature vectors of each query text and the document images in the multimodal document dataset. During the training process, the deep learning model detects each document image to generate multiple visual region maps of different resolutions corresponding to each document image. The deep learning model extracts and fuses each visual region map to obtain an image feature vector. The deep learning model calculates the target loss based on the image feature vector and the corresponding query text feature vector to update the model parameters of the deep learning model, thus obtaining the trained deep learning model. This deep learning model serves as a dynamic resolution visual language model for retrieving the document page corresponding to the feature vector of query text from multiple document images. In some embodiments of this application, the deep learning model includes: A dynamic resolution image extraction layer is used to detect and segment semantic entities in the document image to output a visual representation structure map, visual region maps, and bounding box coordinates corresponding to each semantic entity. The visual representation structure map is composed of nodes and edges between nodes. The nodes are visual region maps, and the edges are spatial adjacency, semantic correlation, or logical membership relationships between visual regions. The visual region maps include a global map, a combined region map, and an entity region map. A visual feature refinement layer is used to extract the feature vectors corresponding to each of the visual region maps, align and fuse the feature vectors corresponding to each of the visual region maps based on the coordinates of each bounding box and the visualization representation structure map, so as to output an image feature vector; wherein, the feature vectors corresponding to each of the visual region maps include global map visual feature vectors, combined region visual feature vectors and entity region visual feature vectors; A cross-modal alignment layer is used to calculate the delayed interactive similarity between the image feature vector and the query text feature vector to obtain a similarity score, and to align the image feature vector and the query text feature vector using a symmetric contrast loss function to obtain a symmetric contrast loss. The symmetric contrast loss is combined with the losses of each auxiliary task to obtain the target loss, and the model parameters of the deep learning model are updated to output the trained deep learning model. The auxiliary tasks include image-text comparison task, image-guided text generation task, and image-text matching task.

[0007] In some embodiments of this application, the dynamic resolution image extraction layer includes: An image detection unit is used to detect and segment each semantic entity in the document image, and obtain the original region, bounding box coordinates, entity type and visual complexity score corresponding to each semantic entity. An image segmentation unit is used to divide each of the original region images into two types, entity regions and non-critical regions, based on the entity type and the visual complexity score, determine the resolution corresponding to the critical regions and the non-critical regions respectively, and enhance each of the original region images belonging to the non-critical regions. The image fusion unit is used to merge the original region images belonging to the same logic based on a preset logic to obtain a combined region image, and to construct a visual representation structure diagram stored in a graph structure.

[0008] In some embodiments of this application, the visual feature refinement layer includes: A visual feature encoding unit is used to extract the feature vectors corresponding to each of the visual region maps and retain the position information of each of the visual region maps to obtain a feature map sequence. The region feature fusion unit is used to align and combine the region visual feature vector and the entity region visual feature vector based on the feature map sequence and the coordinates of each bounding box, and using the global map visual feature vector as a reference. It then uses a convolutional network to align and combine the region visual feature vector and the entity region visual feature vector, and uses an attention mechanism to fuse the global map visual feature vector and the aligned combined region visual feature and entity region visual feature to obtain the document visual feature vector. The visual feature enhancement unit is used to update the feature vectors of each node in the visualization representation structure graph and the feature vectors of its neighboring nodes based on the visualization representation structure graph to enhance the document visual feature vector and obtain the image feature vector.

[0009] In some embodiments of this application, before training a preset deep learning model based on the feature vectors of each of the query texts and the document images in the multimodal document dataset, the method further includes: The dynamic resolution image extraction layer of the deep learning model is optimized using weighted cross-entropy loss and regression loss; The visual feature refinement layer of the deep learning model is trained based on a preset image-text pair dataset, while retaining the model parameters.

[0010] In some embodiments of this application, before obtaining the multimodal document dataset and multiple query text feature vectors, the method further includes: Collect documents of various document types and save each document as a document image of a preset resolution to construct a multimodal document dataset; Construct a document retrieval dataset, which includes query text and the document pages corresponding to the query text; The query text is segmented and encoded to obtain a feature vector of the query text.

[0011] A second aspect of this application provides a document retrieval method based on a dynamic resolution visual language model, the method comprising: Obtain the query text and document image library, segment the query text into words, and encode the query text to obtain the query text feature vector; The visual features of each document page in the document image library are extracted using a dynamic resolution visual language model to construct a hierarchical graph index; Based on the query text feature vector and the hierarchical graph index, a coarse search is performed on the document image library using a dynamic resolution visual language model to obtain each document page; wherein, the dynamic resolution visual language model is trained using the dynamic resolution visual language model training method described above. The document pages are comprehensively scored by combining multi-dimensional scores to generate a ranking result; wherein the multi-dimensional scores include visual similarity, text semantic matching degree, and layout structure consistency score.

[0012] In some embodiments of this application, after obtaining the various document pages, the method further includes: The document images and query text corresponding to each document page are input into a multimodal large language model to generate natural language results, and the regions of the results in the document are highlighted.

[0013] A third aspect of this application provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned dynamic resolution visual language model training method and / or document retrieval method based on the dynamic resolution visual language model.

[0014] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned dynamic resolution visual language model training method and / or document retrieval method based on the dynamic resolution visual language model.

[0015] The fifth aspect of this application provides a computer program product comprising a computer program that, when executed by a processor, implements the dynamic resolution visual language model training method and / or the document retrieval method based on the dynamic resolution visual language model described in the first aspect above.

[0016] This application provides a method for training a dynamic resolution visual language model. The method includes: acquiring a multimodal document dataset and multiple query text feature vectors; training a pre-defined deep learning model based on each query text feature vector and each document image in the multimodal document dataset, so that during the training process, the deep learning model detects each document image to generate multiple visual region maps of different resolutions corresponding to each document image, and extracts and fuses each visual region map to obtain an image feature vector, and calculates a target loss based on the image feature vector and the corresponding query text feature vector to update the model parameters of the deep learning model, thereby obtaining the trained deep learning model; using this deep learning model as a dynamic resolution visual language model for retrieving document pages corresponding to query text feature vectors from multiple document images, thus designing a novel document retrieval method based on a dynamic resolution visual language model. This framework uses document page images as a unified input to achieve a unified understanding and efficient retrieval of fine-grained visual features within document images. Through dynamic resolution image processing and structured visual feature extraction mechanisms, it adaptively adjusts the resolution of different regions of the image, ensuring computational efficiency while fully preserving fine-grained visual features such as charts, tables, and non-text elements. This addresses the performance bottleneck of traditional methods on complex documents, achieving an optimal balance between detail preservation and efficiency. Furthermore, through multi-granularity cross-modal alignment and multi-task joint optimization mechanisms, it achieves high-quality cross-modal feature alignment and retrieval, improving the accuracy and generalization ability of document retrieval. Simultaneously, the method in this application achieves significant performance improvements on multiple public document retrieval benchmarks, demonstrating strong generalization ability and practical value. It can be widely applied to various document-intensive application scenarios, such as intelligent document analysis, visual question answering, and cross-modal retrieval, providing users with efficient, accurate, and intelligent multimodal document retrieval services, significantly improving information acquisition efficiency and knowledge management levels.

[0017] Additional advantages, objectives, and features of this application will be set forth in part in the description which follows, and will in part become apparent to those skilled in the art upon review of the following description, or may be learned by practice of the application. The objectives and other advantages of this application can be realized and obtained by means of the structures specifically pointed out in the specification and drawings.

[0018] Those skilled in the art will understand that the purposes and advantages that can be achieved with this application are not limited to those specifically described above, and that the above and other purposes that this application can achieve will be more clearly understood from the following detailed description. Attached Figure Description

[0019] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, do not constitute a limitation thereof. The components in the drawings are not drawn to scale but are merely for illustrating the principles of this application. For ease of illustration and description of certain parts of this application, corresponding portions in the drawings may be enlarged, i.e., may appear larger relative to other components in an exemplary device actually manufactured according to this application. In the drawings: Figure 1 This is a flowchart illustrating a dynamic resolution visual language model training method according to an embodiment of this application.

[0020] Figure 2 This is a schematic diagram of the document retrieval method based on a dynamic resolution visual language model in an application example of this application.

[0021] Figure 3 This is a schematic diagram of the dynamic resolution image extraction module in an application example of this application.

[0022] Figure 4 This is a schematic diagram of the visual feature refinement module in an application example of this application.

[0023] Figure 5 This is a schematic diagram of the cross-modal alignment module in an application example of this application.

[0024] Figure 6 This is a graph of the loss function in an application example of this application. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and their descriptions are used to explain this application, but are not intended to limit it.

[0026] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the structures and / or processing steps closely related to the solution according to this application are shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0027] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0028] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0029] In the following description, embodiments of the present application will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0030] Therefore, to address the problem that traditional methods ignore complex layouts and visual details in documents due to fixed resolution input, thus limiting retrieval accuracy, this application provides a dynamic resolution visual language model training method, a document retrieval method based on a dynamic resolution visual language model, an electronic device, a computer-readable storage medium, and a computer program product, respectively, to improve the accuracy and generalization ability of document retrieval.

[0031] The following examples will provide a detailed description.

[0032] Based on this, embodiments of this application provide a dynamic resolution visual language model training method that can be executed by a dynamic resolution visual language model training device, see [link to relevant documentation]. Figure 1 The method specifically includes the following: Step 100: Obtain the multimodal document dataset and multiple query text feature vectors.

[0033] Specifically, the multimodal document dataset encompasses large-scale image-text pair data across various document types. The query text feature vector is obtained by processing the query text, which originates from the document retrieval dataset. The dataset construction process will be described in subsequent embodiments.

[0034] Step 200: Train a preset deep learning model based on each of the query text feature vectors and each document image in the multimodal document dataset. During the training process, the deep learning model detects each of the document images to generate multiple visual region maps of different resolutions corresponding to each document image. The deep learning model extracts and fuses each of the visual region maps to obtain image feature vectors. The deep learning model calculates the target loss based on the image feature vectors and the corresponding query text feature vectors to update the model parameters of the deep learning model, thus obtaining the trained deep learning model.

[0035] It should be noted that, firstly, the dynamic resolution image extraction layer intelligently analyzes and extracts multi-granular visual features from the input document image, abandoning the traditional single-resolution processing method and instead adopting a content-adaptive dynamic resolution strategy; secondly, the visual feature refinement layer encodes the extracted visual region maps of different resolutions into image feature vectors with strong semantic representation capabilities, and eliminates geometric deviations caused by image cropping, scaling, and deformation through a spatial alignment mechanism; finally, the cross-modal alignment layer constructs a deep semantic association between text query features and image features through joint training using a multi-task assisted learning framework, a delayed interaction strategy, and symmetric contrast loss, achieving cross-modal feature alignment to enhance the model's multimodal understanding capabilities and realize end-to-end model optimization. Detailed implementation methods will be described in subsequent embodiments.

[0036] Step 300: Use the deep learning model as a dynamic resolution visual language model for retrieving the document page corresponding to the feature vector of the query text from multiple document images.

[0037] It is understood that the deep learning model is configured to retrieve the document page corresponding to the query text feature vector from multiple document images.

[0038] As described above, the dynamic resolution visual language model training method provided in this application, through the design of a novel document retrieval framework based on a dynamic resolution visual language model, uses document page images as unified input to achieve a unified understanding and efficient retrieval of fine-grained visual features in document images. Through dynamic resolution image processing and structured visual feature extraction mechanisms, the resolution of different regions of the image is adaptively adjusted, ensuring computational efficiency while fully preserving fine-grained visual features such as charts, tables, and non-text elements. This solves the performance bottleneck of traditional methods on complex documents, achieving an optimal balance between detail preservation and efficiency. Furthermore, through multi-granularity cross-modal alignment and multi-task joint optimization mechanisms, high-quality cross-modal feature alignment and retrieval are achieved, improving the accuracy and generalization ability of document retrieval. Simultaneously, the method of this application achieves significant performance improvements on multiple publicly available document retrieval benchmarks, demonstrating strong generalization ability and practical value. It can be widely applied to various document-intensive application scenarios, such as intelligent document analysis, visual question answering, and cross-modal retrieval, providing users with efficient, accurate, and intelligent multimodal document retrieval services, significantly improving information acquisition efficiency and knowledge management levels.

[0039] To further improve the accuracy and generalization ability of document retrieval, in a dynamic resolution visual language model training method provided in this application embodiment, step 200 of the deep learning model specifically includes the following: Step 210: Dynamic resolution image extraction layer, used to detect and segment each semantic entity in the document image, so as to output the visual representation structure map corresponding to the document image, each visual region map, and the bounding box coordinates corresponding to each semantic entity; wherein, the visual representation structure map is composed of nodes and edges between nodes, the nodes are each visual region map, the edges are the spatial adjacency relationship, semantic correlation relationship or logical membership relationship between each visual region, and the visual region map includes a global map, a combined region map and an entity region map.

[0040] Step 220: Visual feature refinement layer, used to extract the feature vectors corresponding to each of the visual region maps, align and fuse the feature vectors corresponding to each of the visual region maps based on the coordinates of each bounding box and the visualization representation structure map, so as to output image feature vectors; wherein, the feature vectors corresponding to each of the visual region maps include global map visual feature vectors, combined region visual feature vectors and entity region visual feature vectors.

[0041] Step 230: Cross-modal alignment layer, used to calculate the delayed interactive similarity between the image feature vector and the query text feature vector to obtain a similarity score, and to align the image feature vector and the query text feature vector using a symmetric contrast loss function to obtain a symmetric contrast loss, and to combine the symmetric contrast loss with the losses of each auxiliary task to obtain the target loss, and to update the model parameters of the deep learning model to output the trained deep learning model; wherein, the auxiliary tasks include image-text comparison task, image-guided text generation task, and image-text matching task.

[0042] Specifically, the dynamic resolution image extraction layer sequentially performs global document analysis and entity region detection, adaptive resolution allocation and region cropping, and multi-level region organization and spatial relationship modeling on each semantic entity in the document image, finally outputting a visual representation structure map, a visual region map, and the bounding box coordinates of the semantic entities corresponding to the document image. The visual feature refinement layer, based on the visual region map, sequentially performs visual feature encoding, region feature alignment and fusion, and cross-regional context enhancement, finally outputting an image feature vector. The cross-modal alignment layer sequentially performs delayed interactive similarity calculation, symmetric contrastive learning and hard negative sample mining, and multi-task assisted learning framework training on the image feature vector and the query text feature vector.

[0043] It is understood that the visualization representation structure graph is stored in a graph structure, consisting of nodes and edges between nodes. The nodes are various visual region graphs, and the edges are spatial adjacency relationships, semantic related relationships, or logical membership relationships between various visual regions. The visual region graphs include: a global low-resolution overview graph, i.e., a global graph (used to capture the overall layout), a medium-granularity combined region graph, i.e., a combined region graph (used to understand the relationships between modules), and a high-resolution key entity graph, i.e., an entity region graph (used to extract detailed features).

[0044] It should also be noted that, considering the characteristics of document retrieval tasks, such as the large difference between query and document lengths and the need for multi-granularity alignment in semantic matching, a delayed interactive similarity calculation method can be used, as shown in the following formula: Where each p(j) represents a single-page image of the j-th document, and q(i) is the ith query question. and These are the multi-vector representations of queries and documents in the shared embedding space, respectively. and These represent the number of vectors in the query vector set and the document page embedded vector set, respectively. Specifically, instead of directly calculating the dot product similarity between the image feature vector and the query text feature vector, the query text feature vector is decomposed into multiple query sub-units (e.g., through the intermediate layer output of the query encoder), and the image feature vector is decomposed into multiple image sub-units. The similarity score is obtained by summing the bidirectional maximum similarity of all query sub-units and all image sub-units. This method allows deep learning models to perform flexible alignment at different semantic granularities, and is particularly suitable for matching scenarios involving long queries and complex documents. During training, a symmetric contrastive loss function can be used to simultaneously optimize the retrieval consistency in both the image-to-text and text-to-image directions. The loss function is defined as: in, and Normalized cross-entropy losses are applied in the image-to-text and text-to-image directions, respectively. In each training batch, not only are random negative samples used, but an online hard negative sample mining strategy is also employed to select documents most similar to positive samples from the same batch as hard negative samples, significantly improving the model's discriminative ability. The loss function incorporates temperature scaling parameters and marginal loss to enhance training stability and convergence efficiency. In addition to the main retrieval task, three auxiliary tasks are jointly trained to enhance the model's multimodal understanding capabilities. These multi-tasks can be jointly optimized through weighted losses, sharing underlying encoder parameters to achieve knowledge transfer and representation enhancement. It is understood that the auxiliary tasks include image-text comparison tasks (a binary classification task to determine whether an image and text match, enhancing cross-modal alignment), image-guided text generation tasks (generating descriptive text based on the input image, improving the model's ability to express visual content in a linguistic way), and image-text matching tasks (performing fine-grained matching of image regions and text fragments, enhancing local alignment capabilities).

[0045] To further improve the accuracy and generalization ability of document retrieval, in a dynamic resolution visual language model training method provided in this application embodiment, step 210 of the dynamic resolution image extraction layer specifically includes the following: Step 211: Image detection unit, used to detect and classify each semantic entity in the document image, and obtain the original region map, bounding box coordinates, entity type and visual complexity score corresponding to each semantic entity; Step 212: Image segmentation unit, used to divide each of the original region images into two types, entity region and non-critical region, based on the entity type and the visual complexity score, determine the resolution corresponding to the critical region and the non-critical region respectively, and enhance each of the original region images belonging to the non-critical region; Step 213: Image fusion unit, used to merge the original regions belonging to the same logic based on a preset logic to obtain a combined region map, and construct a visual representation structure map stored in a graph structure.

[0046] Specifically, the image detection unit performs global document analysis and entity region detection: it can use a visual detection model (i.e., a deep learning model) based on the Transformer architecture to perform a full-page scan of the document image, identify and locate all semantic entities in the image, including title areas, body paragraphs, table structures, statistical charts, mathematical formulas, illustrations, headers and footers, and column boundaries, etc. Each detected entity outputs its precise bounding box coordinates, entity type, and visual complexity score. The image-based sub-unit performs adaptive resolution allocation and region cropping: based on the entity type and visual complexity score, the processing resolution of each region is dynamically determined. At the same time, an edge-aware super-resolution algorithm can be introduced to intelligently enhance downsampled text regions, ensuring that text readability is not compromised by reduced resolution. The image fusion unit performs multi-level region organization and spatial relationship modeling: after extracting all original region maps, region merging and hierarchical construction operations are performed. Based on spatial adjacency and semantic consistency, original region maps belonging to the same logical unit (such as a table and its title, or a chart and its legend) can be merged into a combined region map, forming a medium-granularity visual unit, and constructing a visual representation structure map from the overall page view to local detailed regions.

[0047] The critical areas are those containing dense small-font text, complex table borders, and fine chart curves. For critical areas, a high-resolution (e.g., 600 DPI or higher) processing strategy is adopted to directly crop the corresponding areas of the original image without loss. The non-critical areas are those containing large areas of blank background, simple decorative elements, or low information content. For non-critical areas, a low-resolution (e.g., 72 DPI) downsampling processing is adopted to significantly reduce the computational load.

[0048] To further improve the accuracy and generalization ability of document retrieval, in a dynamic resolution visual language model training method provided in this application embodiment, the visual feature refinement layer in step 220 specifically includes the following: Step 221: Visual feature encoding unit, used to extract the feature vectors corresponding to each of the visual region maps, and retain the position information of each of the visual region maps to obtain a feature map sequence; Step 222: Region feature fusion unit, used to align and combine region visual feature vectors and entity region visual feature vectors based on the feature map sequence and the coordinates of each bounding box, and using the global map visual feature vector as a reference, using a convolutional network to align and combine the global map visual feature vectors and the aligned combined region visual features and entity region visual features, and using an attention mechanism to fuse the global map visual feature vectors and the aligned combined region visual features and entity region visual features to obtain the document visual feature vector; Step 223: Visual feature enhancement unit, used to update the feature vectors of each node in the visualization representation structure graph and the feature vectors of its neighboring nodes based on the visualization representation structure graph to enhance the document visual feature vector and obtain the image feature vector.

[0049] Specifically, the visual feature encoding unit can use a pre-trained large-scale visual Transformer model, i.e., a deep learning model, as the basic encoder, such as Vision Transformer-Large / 16 (ViT-L / 16). It independently encodes all visual region maps (including global maps, combined region maps, and entity region maps), extracting the visual feature vector of each visual region map. The feature vectors corresponding to each visual region map include the global map visual feature vector, the combined region visual feature vector, and the entity region visual feature vector. During the encoding process, the model retains the spatial location information of the region and outputs a sequence of feature maps with two-dimensional positional encoding.

[0050] The visual feature enhancement unit aligns and fuses regional features: To address the scale inconsistency and spatial misalignment issues between regions of different resolutions, using the global image visual feature vector as a spatial reference, a deformable convolutional network can predict the offset of each combined region visual feature vector and entity region visual feature vector relative to the bounding box coordinates. Local features are spatially resampled to achieve spatial alignment of multi-resolution features. All aligned regional features are then concatenated along the channel dimension for feature fusion (a multi-layer cross-modal attention module can be used) to generate a unified, dense, and geometrically consistent document visual feature vector. The visual feature enhancement unit performs context enhancement across regions: To further enhance the model's understanding of the document structure, a graph neural network can be introduced to perform message passing and feature aggregation on the visual representation structure graph. The features of each node and the features of its neighboring nodes are updated through multiple rounds of interaction, ultimately resulting in an enhanced visual feature, i.e., an image feature vector, that integrates global layout context and local semantic association.

[0051] To further improve the accuracy and generalization ability of document retrieval, in a dynamic resolution visual language model training method provided in this application embodiment, before training a preset deep learning model based on each of the query text feature vectors and each document image in the multimodal document dataset in step 200, the following content is specifically included: Step 110: Optimize the dynamic resolution image extraction layer of the deep learning model using weighted cross-entropy loss and regression loss; Step 120: Train the visual feature refinement layer of the deep learning model based on the preset image-text pair dataset, and retain the model parameters.

[0052] Specifically, a general document entity detection model, i.e., a deep learning model, is pre-trained using an object detection dataset, such as the Public Layout Dataset (PubLayNet) or the Table Detection and Recognition Dataset (TableBank). The model can use a Swin Transformer as its backbone network, outputting the bounding box coordinates, entity type, and visual complexity score for each entity. Visual complexity can be calculated based on features such as complexity within multimodal data regions. During training, a combination of weighted cross-entropy loss and regression loss (such as the Intersection over Union (IoU) loss function) can be used for joint optimization to ensure a balance between detection accuracy and speed. The base encoder of the visual language model (i.e., the deep learning model) is pre-trained on large-scale image-text pair datasets, such as the Large-scale Artificial Intelligence Open Network 5 Billion Dataset (LAION-5B) and the COYO 700 Million Image-Text Pair Dataset (COYO-700M). A joint pre-training strategy of masked image modeling and masked language modeling can be employed to enable the model to learn to recover overall semantics from local visual cues, enhancing fine-grained understanding. After pre-training, the visual encoder parameters are frozen, and only the decoder and alignment module are fine-tuned to adapt to document retrieval tasks.

[0053] To further improve the accuracy and generalization ability of document retrieval, in the dynamic resolution visual language model training method provided in this application embodiment, before step 100, before obtaining the multimodal document dataset and multiple query text feature vectors, the following content is specifically included: Step 010: Collect documents of various document types and save each document as a document image of a preset resolution to construct a multimodal document dataset; Step 020: Construct a document retrieval dataset, which includes query text and the document pages corresponding to the query text; Step 030: Segment and encode the query text to obtain the query text feature vector.

[0054] Specifically, this involves collecting large-scale image-text pair data covering various document types, including academic papers (arXiv PDFs, scanned journal articles), business documents (annual reports, product manuals, financial statements), presentations (images exported from PowerPoint slides), technical documents (API documentation, user guides), and historical archives (scanned books, manuscripts), etc. Each document page is saved as an RGB image in a three-channel color space, with a recommended resolution of at least 1000×1000 pixels. Simultaneously, a corresponding text description is collected or generated for each page, including the page title, abstract, manually annotated keywords, and automatically generated descriptive text. For training the retrieval task, it is necessary to construct paired data of query text and relevant document pages. Paragraphs are randomly extracted from the documents as query text, and the pages containing them are considered positive samples. Large language models (such as the fourth-generation generative pre-trained transformer GPT-4o) can be used to generate diverse questions as query text based on the page content. Alternatively, question-answer pairs from publicly available document question-answering datasets (such as the Document Visual Question Answering dataset DocVQA and the Slide Visual Question Answering dataset SlideVQA) can be used, with the question as the query text and the page containing the answer as the positive sample. Each query text corresponds to at least one positive sample page and 10-20 negative sample pages (from the same document set but unrelated pages).

[0055] It should also be noted that the constructed document images and query text require data augmentation and preprocessing. Perspective transformation, brightness adjustment, Gaussian noise, and blurring are randomly applied to the document images to simulate distortion and noise in real-world photography. Synonym replacement, sentence structure transformation, and random word deletion are performed on the query text to improve the robustness of the deep learning model to language variations. During training, pages that are visually similar to positive samples but semantically unrelated are dynamically identified as difficult negative samples to enhance the discriminative boundary of the deep learning model.

[0056] Based on the above embodiments of the dynamic resolution visual language model training method, this application also provides an embodiment of a document retrieval method based on a dynamic resolution visual language model, which specifically includes the following: Step 400: Obtain the query text and document image library, segment the query text and encode it to obtain the query text feature vector; Step 500: Use a dynamic resolution visual language model to extract the visual features of each document page in the document image library to construct a hierarchical graph index; Step 600: Based on the query text feature vector and the hierarchical graph index, a coarse search is performed on the document image library using a dynamic resolution visual language model to obtain each document page; wherein, the dynamic resolution visual language model is trained by the dynamic resolution visual language model training method. Step 700: Combine the multi-dimensional scores to comprehensively score each document page and generate a ranking result; wherein, the multi-dimensional scores include visual similarity, text semantic matching degree and layout structure consistency score.

[0057] Specifically, the trained dynamic resolution visual language model is applied to online retrieval scenarios in large-scale document repositories, addressing engineering challenges such as high-dimensional vector indexing, real-time retrieval latency, and multimodal answer generation. An efficient vector similarity search library can be used to offline encode the visual features of all page images in the document repository and construct a hierarchical, navigable small-world graph index (HWGraph Index). This supports millisecond-level nearest neighbor search for millions of documents, and the index supports dynamic addition, deletion, and updates to adapt to real-time changes in the document repository. Search libraries such as the efficient similarity search and clustering library (Facebook AI SimilaritySearch, FAISS) are used. The retrieval process consists of two stages: First, a coarse-grained retrieval is performed using the dynamic resolution visual language model, returning Top-K candidate pages; second, the candidate pages are fine-grained re-ranked, comprehensively considering multiple dimensions such as visual similarity, text semantic matching, and layout consistency to generate the final ranking results.

[0058] It is understandable that the dynamic resolution visual language model is used to perform a coarse search on the user-specified document image library (i.e., the dataset corresponding to the training side, where the training set must be in the same format) based on the hierarchical graph index to obtain each document page.

[0059] It should be noted that the maximum protection scope of the document retrieval method is the same as that of the training side, and does not distinguish between indexes.

[0060] To further improve the accuracy and generalization ability of document retrieval, in the dynamic resolution visual language model training method provided in this application embodiment, after step 600 and after obtaining each document page, the following content is specifically included: Step 800: Input the document image and query text corresponding to each document page into the multimodal large language model to generate natural language results and highlight the area of ​​the results in the document.

[0061] Specifically, the dynamic resolution visual language model not only supports document retrieval but can also be combined with large-scale language models to achieve end-to-end visual question answering. Given query text, the dynamic resolution visual language model retrieves relevant pages and inputs the page images along with the query into a multimodal large-scale language model, such as the Qwen Vision-Language Model (Qwen-VL) or Microsoft's third-generation Phi series Multimodal Vision-Language Model (Phi-3-Vision), to generate a natural language answer based on visual evidence and highlight the source region of the answer in the document.

[0062] To further illustrate the embodiments of the above-described dynamic resolution visual language model training method and the document retrieval method based on the dynamic resolution visual language model, this application also provides a specific application example of the document retrieval method based on the dynamic resolution visual language model. Specifically, it includes the following: This application aims to construct an efficient, accurate, and generalizable multimodal document retrieval system to overcome the problems of visual information loss, layout structure destruction, and insufficient fine-grained semantic understanding in traditional document retrieval methods based on optical character recognition (OCR) when processing documents with complex layouts. It also addresses the contradiction between the detail blurring and low computational efficiency of existing visual language models due to fixed-resolution input. This application proposes a novel document retrieval framework based on a dynamic resolution visual language model, particularly suitable for handling multimodal document retrieval tasks involving complex layouts, charts, tables, and non-text elements. This framework utilizes four core technologies: multi-level dynamic resolution image processing, structured visual feature extraction, multi-granular cross-modal alignment, and multi-task joint optimization. These technologies enable unified understanding and efficient retrieval of text, charts, tables, formulas, and layout information in document images. A schematic diagram of its structure is shown below. Figure 2 As shown.

[0063] I. Model Training and Optimization The pre-trained deep learning model is integrated with the dynamic resolution image extraction module (i.e., the visual feature refinement layer), the cross-modal alignment module (i.e., the cross-modal alignment layer), and the cross-modal alignment module (i.e., the cross-modal alignment layer). A schematic diagram of the dynamic resolution image extraction module is shown below. Figure 3 As shown in the diagram, the structure of the visual feature refinement module is as follows: Figure 4 As shown in the diagram, the cross-modal alignment module has the following structural schematic. Figure 5 As shown, the loss function graph is as follows: Figure 6 As shown. Among them, in Figure 4In this context, (x, y) represents the coordinates of the endpoints of the region bounding box, r represents the embedding vector, ViT represents the visual transformer, and RoI represents the region of interest. This represents the image region. In Figure 5 middle, Representing image feature vectors, `h` represents the feature vector of the query text, and `h` represents the dimension-aligned feature vector. Figure 6 In this context, Image-Text Contrast Learning (ITC), Image-Guided Text Generation (ITG), and Image-Text Matching (ITM) are used. I and T represent images and text, respectively; positive and negative represent positive and negative samples, respectively; logit represents the probability of matching an image with text; and T_mask represents the text sequence after masking. Figures 2-4 The document images in the text are for illustrative purposes only. The specific text content recorded in the document images does not affect the understanding of the technical solution, and therefore has been blurred.

[0064] End-to-end fine-tuning is performed on the constructed document retrieval dataset. It can be configured with the AdamW (Adaptive Moment Estimation with Decoupled Weight Decay) optimizer, an initial learning rate of 0.00002, and a batch size of 64. Linear warm-up and cosine annealing learning rate scheduling can be used. The multi-task loss weights are: retrieval contrast loss weight 1.0, image guided generation loss weight 0.5, and image-text matching loss weight 0.3. Training is performed on two A100 GPUs for approximately 50 training epochs, requiring about half a day. Every 5 epochs, the retrieval accuracy normalized depreciation cumulative gain (top 5), recall (top 10), question answering accuracy, and cross-modal matching score are evaluated on the validation set.

[0065] To adapt to edge devices or low-latency scenarios, knowledge distillation techniques can be used to compress large models into lightweight versions. A pre-trained dynamic resolution visual language model can be used as a teacher model to train a small student model (such as the Mobile Vision Transformer (MobileViT) network or the backbone of the Efficient Network (EfficientNet) series of convolutional neural network models). Distillation objectives include matching the logits of the output layer log regression values, matching intermediate feature maps, and matching attention distributions. The resulting lightweight model is approximately 1 / 10 the size of the original model, with a 5x improvement in inference speed and accuracy loss controlled within 3%.

[0066] II. System Integration and Deployment 1. Document Index Building Service Develop an offline index building pipeline, taking a document repository directory as input and outputting high-efficiency vector index files. The process includes: Automatically traverse the document directory, recognize PDF, PPT, Word and other formats, and convert them into PNG images; The visual feature vector (512-dimensional) of each page image is extracted using a trained dynamic resolution visual language model. We use the FAISS vector similarity search library to build an approximate nearest neighbor search structure based on inverted file index and product quantization (IVF-PQ index, which balances search accuracy and memory usage). The index file is stored on a high-performance SSD, supporting distributed loading and parallel querying.

[0067] 2. Real-time search API service Develop a RESTful API service based on the FastAPI framework, providing the following endpoints: / index: Accepts new document uploads, automatically extracts features and updates the index; / search: Accepts text queries and returns Top-K related pages (including page thumbnails, similarity scores, and source document information); / ask: Accepts natural language questions and returns text answers based on search results with highlighted visual evidence; / batch_search: Supports batch query processing, improving throughput.

[0068] The service deployment uses the Docker Container Platform (Docker) for containerization, supports horizontal scaling of the Kubernetes container orchestration system, and can achieve more than 100 queries per second (QPS) per node.

[0069] 3. User Interface and Visualization Develop a web-based interactive interface with the following functionalities: Document library browsing and categorization management; Multimodal search input box (supports mixed text and image queries); Search results are displayed in pagination and can be filtered by relevance, time, and source. Clicking on the results page allows you to view the original document, highlight the matching area, and display relevant text snippets. Search history and user feedback collection (relevant / irrelevant tags).

[0070] III. Performance Verification and Evaluation 1. Benchmarking and Comparative Experiments Evaluate the system performance comprehensively on the following public datasets: Document Visual Question Answering (DocVQA): A benchmark for document visual question answering, testing a model's ability to answer questions based on document images; Slide Visual Question Answering (SlideVQA): A presentation question-and-answer dataset that assesses understanding of slide content; Infographics Visual Question Answering (InfographicsVQA): An infographic question answering dataset that tests the understanding of charts and images; Chart Question Answering (ChartQA) dataset: A chart question answering dataset used to evaluate data extraction and reasoning abilities; Multimodal Long Document Understanding Benchmark (MMLongBench-Doc): A multimodal long document understanding benchmark that tests the comprehensive analysis of multi-page documents.

[0071] The comparison methods include: traditional optical character recognition (OCR) + text retrieval, such as BM25 ranking (Best Matching 25, BM25), dense passage retrieval (DPR) models, general multimodal models, such as contrastive language-image pretraining (CLIP), bootstrapping language-image pretraining (BLIP), and specialized document retrieval models, such as column-wise patch alignment for layout-aware image-text retrieval (ColPali), and document similarity encoder (DSE). Evaluation metrics include the top 5 normalized discounted cumulative gain (nDCG@5), the top 10 recall (Recall@10), precision, and the harmonic mean of precision and recall (F1 Measure, F1 score).

[0072] 2. Ablation Experiment Analysis To verify the contribution of each module, the following ablation experiment was designed: Remove dynamic resolution processing and switch to a fixed resolution (224×224). Remove the region alignment module and directly stitch together multi-scale features; Remove the delayed interaction mechanism and replace it with global vector dot product; Remove multi-task training and use only contrastive learning loss; Remove contextual enhancements from graph neural networks.

[0073] Experimental results show that dynamic resolution and delayed interaction contribute the most to the performance improvement (8.2% and 6.7% respectively), while multi-task training improves performance by 4.5%.

[0074] 3. Real-world deployment testing Deployment and testing were conducted in real-world scenarios such as the company's internal knowledge base, digital library, and enterprise document management system. Knowledge base retrieval: Tested in a database of over 100,000 technical documents, the average retrieval accuracy rate was 87.3%, a 34% improvement over the original keyword-based system; Digitization of historical archives: Tested in an ancient book scan image database, it still maintains a retrieval accuracy of over 75% for blurry, tilted, and handwritten documents; Customer service automation: Integrated into the customer service system, it automatically retrieves answers from product manuals, achieving a 91% accuracy rate and reducing human intervention by 60%.

[0075] 4. Efficiency and Scalability Testing Feature extraction time per document page: 50ms for GPU (Graphics Processing Unit), 300ms for CPU (Central Processing Unit); The size of a million-document index is approximately 20GB (after compression). Single retrieval latency (including feature extraction + index query): average 120ms; The system supports a maximum document library size: theoretically unlimited, but in actual testing, the retrieval latency for tens of millions of documents is less than 1 second; Concurrency processing capability: A single server supports 200+ concurrent queries.

[0076] IV. Maintenance and Continuous Optimization 1. Model Iterative Update Mechanism Establish a model performance monitoring and automatic update process: Collect user feedback data (click-through rate, dwell time, relevant tags); Regularly fine-tune the model on new data to keep it adaptable to emerging document types; We adopted an A / B testing framework to gradually release improved versions.

[0077] 2. Domain-Adaptive Customization We offer customized training services tailored to the specific needs of different industries. Medical documentation: Enhances understanding of medical charts and medical record forms; Legal documents: Optimize the recognition of clauses, citations, and signatures; Engineering drawings: Enhance the ability to analyze technical drawings and schematic diagrams.

[0078] 3. Privacy and Security Protection Supports local deployment, ensuring that sensitive documents do not leave the intranet environment; Provides a document desensitization module to automatically mask private information such as ID card numbers and phone numbers; Implement fine-grained access control to ensure that users can only retrieve documents within their authorized scope.

[0079] Through the complete process described above, the dynamic resolution visual language model of this invention can be successfully deployed in various document-intensive application scenarios, providing users with efficient, accurate, and intelligent multimodal document retrieval services, and significantly improving information acquisition efficiency and knowledge management level.

[0080] The significant advantages of the application examples in this application are reflected in: Breaking the limitations of fixed resolution: By using a dynamic resolution mechanism, the optimal balance between detail preservation and efficiency is achieved with limited computing resources, thus solving the performance bottleneck of traditional methods on complex documents.

[0081] Deep integration of structure and semantics: Through hierarchical region organization and graph neural network modeling, it captures the visual layout structure and deep semantic relationships of a document, achieving a true "visual-structure-semantic" three-in-one understanding.

[0082] Highly scalable and versatile: The model does not depend on specific document types or templates and can be generalized to various document formats such as academic papers, business reports, technical manuals, presentations, and historical archives, and has strong domain adaptability.

[0083] End-to-end integrated design: From the input of original document images to the output of final search results, the entire process is automated, eliminating the need for manual annotation or rule configuration, and significantly reducing deployment and maintenance costs.

[0084] Superior performance: Achieves leading performance on multiple authoritative benchmarks, with retrieval accuracy 10-15 percentage points higher than the best existing methods.

[0085] This application also provides an electronic device, which may include a processor, a memory, a receiver, and a transmitter. The processor is used to execute the dynamic resolution visual language model training method and / or the document retrieval method based on the dynamic resolution visual language model mentioned in the above embodiments. The processor and the memory can be connected via a bus or other means, taking a bus connection as an example. The receiver can be connected to the processor and the memory via wired or wireless means.

[0086] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0087] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the dynamic resolution visual language model training method and / or the document retrieval method based on the dynamic resolution visual language model in the embodiments of this application. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory, thereby implementing the dynamic resolution visual language model training method and / or the document retrieval method based on the dynamic resolution visual language model in the above method embodiments.

[0088] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0089] The one or more modules are stored in the memory, and when executed by the processor, they execute the dynamic resolution visual language model training method and / or the document retrieval method based on the dynamic resolution visual language model in the implementation embodiment.

[0090] In some embodiments of this application, the user equipment may include a processor, a memory, and a transceiver unit. The transceiver unit may include a receiver and a transmitter. The processor, memory, receiver, and transmitter may be connected via a bus system. The memory is used to store computer instructions, and the processor is used to execute the computer instructions stored in the memory to control the transceiver unit to send and receive signals.

[0091] As one implementation method, the functions of the receiver and transmitter in this application can be implemented by transceiver circuits or dedicated transceiver chips, and the processor can be implemented by dedicated processing chips, processing circuits or general-purpose chips.

[0092] As another implementation approach, the server provided in this application embodiment can be implemented using a general-purpose computer. That is, the program code implementing the processor, receiver, and transmitter functions is stored in memory, and the general-purpose processor implements the processor, receiver, and transmitter functions by executing the code in memory.

[0093] This application also provides a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the steps of the aforementioned dynamic resolution visual language model training method and / or document retrieval method based on the dynamic resolution visual language model. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0094] This application also provides a computer program product, specifically comprising a computer program that, when executed by a processor, implements the steps of the dynamic resolution visual language model training method and / or the document retrieval method based on the dynamic resolution visual language model mentioned in the foregoing embodiments.

[0095] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. The programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave.

[0096] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0097] In this application, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0098] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to the embodiments of this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for training a dynamic resolution visual language model, characterized in that, The method includes: Obtain a multimodal document dataset and feature vectors from multiple query texts; A preset deep learning model is trained based on the feature vectors of each query text and the document images in the multimodal document dataset. During the training process, the deep learning model detects each document image to generate multiple visual region maps of different resolutions corresponding to each document image. The deep learning model extracts and fuses each visual region map to obtain an image feature vector. The deep learning model calculates the target loss based on the image feature vector and the corresponding query text feature vector to update the model parameters of the deep learning model, thus obtaining the trained deep learning model. This deep learning model is used as a dynamic resolution visual language model for retrieving the document page corresponding to the feature vector of query text from multiple document images. The deep learning model includes: A dynamic resolution image extraction layer is used to detect and segment semantic entities in the document image to output a visual representation structure map, visual region maps, and bounding box coordinates corresponding to each semantic entity. The visual representation structure map is composed of nodes and edges between them. Each node represents a visual region map, and each edge represents a spatial adjacency, semantic correlation, or logical membership relationship between visual regions. The visual region maps include a global map, a combined region map, and an entity region map. The original region maps obtained by detecting and segmenting semantic entities in the document image are divided into two types: key regions and non-key regions. High-resolution processing is used for key regions, while low-resolution downsampling processing is used for non-key regions. A visual feature refinement layer is used to extract the feature vectors corresponding to each of the visual region maps, align and fuse the feature vectors corresponding to each of the visual region maps based on the coordinates of each bounding box and the visualization representation structure map, so as to output an image feature vector; wherein, the feature vectors corresponding to each of the visual region maps include global map visual feature vectors, combined region visual feature vectors and entity region visual feature vectors; A cross-modal alignment layer is used to calculate the delayed interactive similarity between the image feature vector and the query text feature vector to obtain a similarity score, and to align the image feature vector and the query text feature vector using a symmetric contrast loss function to obtain a symmetric contrast loss. The symmetric contrast loss is combined with the losses of each auxiliary task to obtain the target loss, and the model parameters of the deep learning model are updated to output the trained deep learning model. The auxiliary tasks include image-text comparison task, image-guided text generation task, and image-text matching task.

2. The dynamic resolution visual language model training method according to claim 1, characterized in that, The dynamic resolution image extraction layer includes: An image detection unit is used to detect and segment each semantic entity in the document image, and obtain the original region map, bounding box coordinates, entity type and visual complexity score corresponding to each semantic entity. An image segmentation unit is used to divide each of the original region images into two types, key regions and non-key regions, based on the entity type and the visual complexity score, determine the resolution corresponding to the key regions and the non-key regions respectively, and enhance each of the original region images belonging to the non-key regions. The image fusion unit is used to merge the original region images belonging to the same logic based on a preset logic to obtain a combined region image, and to construct a visual representation structure diagram stored in a graph structure.

3. The dynamic resolution visual language model training method according to claim 1, characterized in that, The visual feature refinement layer includes: A visual feature encoding unit is used to extract the feature vectors corresponding to each of the visual region maps and retain the position information of each of the visual region maps to obtain a feature map sequence. The region feature fusion unit is used to align and combine the region visual feature vector and the entity region visual feature vector based on the feature map sequence and the coordinates of each bounding box, and using the global map visual feature vector as a reference. It then uses a convolutional network to align and combine the region visual feature vector and the entity region visual feature vector, and uses an attention mechanism to fuse the global map visual feature vector and the aligned combined region visual feature and entity region visual feature to obtain the document visual feature vector. The visual feature enhancement unit is used to update the feature vectors of each node in the visualization representation structure graph and the feature vectors of its neighboring nodes based on the visualization representation structure graph to enhance the document visual feature vector and obtain the image feature vector.

4. The dynamic resolution visual language model training method according to claim 1, characterized in that, Before training the preset deep learning model based on the feature vectors of each query text and the document images in the multimodal document dataset, the method further includes: The dynamic resolution image extraction layer of the deep learning model is optimized using weighted cross-entropy loss and regression loss; The visual feature refinement layer of the deep learning model is trained based on a preset image-text pair dataset, while retaining the model parameters.

5. The dynamic resolution visual language model training method according to claim 1, characterized in that, Before obtaining the multimodal document dataset and multiple query text feature vectors, the method further includes: Collect documents of various document types and save each document as a document image of a preset resolution to construct a multimodal document dataset; Construct a document retrieval dataset, which includes query text and the document pages corresponding to the query text; The query text is segmented and encoded to obtain a feature vector of the query text.

6. A document retrieval method based on a dynamic resolution visual language model, characterized in that, The method includes: Obtain the query text and document image library, segment the query text into words, and encode the query text to obtain the query text feature vector; The visual features of each document page in the document image library are extracted using a dynamic resolution visual language model to construct a hierarchical graph index; Based on the query text feature vector and the hierarchical graph index, a coarse search is performed on the document image library using a dynamic resolution visual language model to obtain each document page; wherein, the dynamic resolution visual language model is trained by the dynamic resolution visual language model training method according to any one of claims 1 to 5; The document pages are comprehensively scored by combining multi-dimensional scores to generate a ranking result; wherein the multi-dimensional scores include visual similarity, text semantic matching degree, and layout structure consistency score.

7. The document retrieval method based on a dynamic resolution visual language model according to claim 6, characterized in that, After obtaining the various document pages, the process also includes: The document images and query text corresponding to each document page are input into a multimodal large language model to generate natural language results, and the regions of the results in the document are highlighted.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the dynamic resolution visual language model training method as described in any one of claims 1 to 5, and / or implements the document retrieval method based on the dynamic resolution visual language model as described in any one of claims 6 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the dynamic resolution visual language model training method as described in any one of claims 1 to 5, and / or implements the document retrieval method based on the dynamic resolution visual language model as described in any one of claims 6 to 7.

Images