Document retrieval method and device, equipment, storage medium and computer program product

Abstract

The invention relates to the technical field of computers, and discloses a document retrieval method, device and equipment, a storage medium and a computer program product.The method comprises the steps that in response to a document retrieval request, the document retrieval request is converted into a problem retrieval vector, and the vector similarity between the problem retrieval vector and a document image retrieval vector is calculated, the document image retrieval vector is obtained by analyzing a document into a document image set and encoding the document image set, and a document retrieval result corresponding to the document retrieval request is generated according to the vector similarity; according to the document retrieval method, the document is analyzed into the document picture set in advance, the document picture set is encoded to obtain the document image retrieval vector, and the document retrieval is performed through the problem retrieval vector and the document image retrieval vector, so that the document retrieval in the form of the document image is realized; therefore, more space structure information, fine-grained information and context information can be reserved, and the document retrieval precision is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a document retrieval method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] Document retrieval refers to matching user queries with relevant documents in a given corpus. It is central to many wide-ranging industrial applications and can function as a standalone ranking system (search engine) or as part of a more complex information extraction or retrieval-augmented generation (RAG) + document understanding question answering system.

[0003] However, relevant document retrieval methods typically perform document retrieval in text form, that is, process the document content into text paragraphs for retrieval. This results in the loss of a lot of spatial structure information and fine-grained information (such as footnotes, citations, text layout, charts, tables and images in the document). At the same time, the fragmentation of document text paragraphs also leads to the loss of a large amount of contextual information, affecting the accuracy of document retrieval. Summary of the Invention

[0004] The main objective of this application is to provide a document retrieval method, apparatus, device, storage medium, and computer program product, aiming to solve the technical problem that related document retrieval methods lose a lot of spatial structure information, fine-grained information, and contextual information of documents, thus affecting the accuracy of document retrieval.

[0005] To achieve the above objectives, this application provides a document retrieval method, which includes: In response to a document retrieval request, the document retrieval request is transformed into a question retrieval vector; Calculate the vector similarity between the question retrieval vector and the document image retrieval vector, wherein the document image retrieval vector is obtained by parsing the document into a document image set and encoding the document image set; The document retrieval results corresponding to the document retrieval request are generated based on the vector similarity.

[0006] Optionally, before converting the document retrieval request into a question retrieval vector in response to the document retrieval request, the method further includes: The document dataset is transformed into a document image set, and the images in the document image set are encoded using a preset multimodal large model to obtain document image retrieval vectors. A local document library is constructed based on the document image retrieval vectors; Accordingly, calculating the vector similarity between the question retrieval vector and the document image retrieval vector includes: Calculate the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the local document library.

[0007] Optionally, the preset multimodal large model includes a visual encoder, a multilayer perceptron, and a large language model. The step of encoding the images in the document image set using the preset multimodal large model to obtain document image retrieval vectors includes: The visual encoder is used to encode the visual features of the images in the document image set to obtain visual features; The visual features are projected onto the text semantic space using the multilayer perceptron to obtain the projected features. The large language model is used to extract image features with fused semantics from the projected features; The image features of the fused semantics are converted into document image retrieval vectors using the multilayer perceptron.

[0008] Optionally, constructing a local document library based on the document image retrieval vector includes: Obtain the document image and basic document information corresponding to the document image retrieval vector; The document image retrieval vector, the document image, and the basic document information are associated and stored to construct a local document library.

[0009] Optionally, before converting the document dataset into a document image set and encoding the images in the document image set using a preset multimodal large model to obtain the document image retrieval vector, the method further includes: Construct a training dataset, wherein the training dataset includes training data pairs consisting of query statements and document images; The initial multimodal large model is trained based on the training dataset to obtain the preset multimodal large model.

[0010] Optionally, constructing the training dataset includes: Obtain a web document and split each page of the web document into document images; A visual language model is used to generate corresponding query statements for each page of images, and a training dataset is constructed based on the query statements and the document images.

[0011] Optionally, before calculating the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the local document library, the method further includes: Obtain user actions and determine whether the user needs to build a user document library based on the user actions; Accordingly, calculating the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the local document library includes: If it is not necessary to build a user document library, then calculate the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the local document library.

[0012] Optionally, after obtaining user actions and determining whether the user needs to build a user document library based on the user actions, the method further includes: If a user document library needs to be built, then the user document library is built based on the documents uploaded by users; Calculate the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the user document library.

[0013] Optionally, the step of converting the document retrieval request into a question retrieval vector in response to the document retrieval request includes: In response to a document retrieval request, the query question input by the user is extracted from the document retrieval request; The query question is transformed into a question retrieval vector by using a pre-set multimodal large model.

[0014] Optionally, the preset multimodal large model includes a multilayer perceptron and a large language model, and the step of transforming the query question into a question retrieval vector through the preset multimodal large model includes: The query question is input into the large language model, and the textual semantic features of the query question are extracted through the large language model. The semantic features of the text are converted into question retrieval vectors using a multilayer perceptron.

[0015] Optionally, generating the document retrieval result corresponding to the document retrieval request based on the vector similarity includes: The required number of search results to be returned from the document retrieval request is extracted from the user's input. The document retrieval results corresponding to the document retrieval request are generated based on the vector similarity and the required number of retrieval results returned.

[0016] Furthermore, to achieve the above objectives, this application also proposes a document retrieval device, which includes: The vector conversion module is used to convert a document retrieval request into a question retrieval vector in response to the document retrieval request. The similarity calculation module is used to calculate the vector similarity between the question retrieval vector and the document image retrieval vector, wherein the document image retrieval vector is obtained by parsing the document into a document image set and encoding the document image set; The result generation module is used to generate document retrieval results corresponding to the document retrieval request based on the vector similarity.

[0017] Optionally, the document retrieval device further includes: The local library construction module is used to convert the document dataset into a document image set, and encode the images in the document image set using a preset multimodal large model to obtain document image retrieval vectors; and construct a local document library based on the document image retrieval vectors. Accordingly, the similarity calculation module is also used to calculate the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the local document library.

[0018] Optionally, the preset multimodal large model includes a visual encoder, a multilayer perceptron, and a large language model. The local library construction module is further configured to: encode visual features of images in the document image set using the visual encoder to obtain visual features; project the visual features onto the text semantic space using the multilayer perceptron to obtain projected features; extract fused semantic image features from the projected features using the large language model; and convert the fused semantic image features into document image retrieval vectors using the multilayer perceptron.

[0019] Optionally, the local library construction module is further configured to obtain the document image and basic document information corresponding to the document image retrieval vector; and to associate and store the document image retrieval vector, the document image, and the basic document information to construct a local document library.

[0020] Optionally, the document retrieval device further includes: The model training module is used to construct a training dataset, wherein the training dataset includes training data pairs consisting of query statements and document images; and to train an initial multimodal large model based on the training dataset to obtain a preset multimodal large model.

[0021] Optionally, the model training module is further configured to acquire online documents and split each page of the online documents into document images; generate corresponding query statements for each page image using a visual language model; and construct a training dataset based on the query statements and the document images.

[0022] In addition, to achieve the above objectives, this application also proposes a document retrieval device, which includes a memory, a processor, and a document retrieval program stored in the memory and executable on the processor, the document retrieval program being configured to implement the document retrieval method as described above.

[0023] In addition, to achieve the above objectives, this application also proposes a storage medium storing a document retrieval program, which, when executed by a processor, implements the document retrieval method as described above.

[0024] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a document retrieval program that, when executed by a processor, implements the document retrieval method as described above.

[0025] One or more technical solutions proposed in this application have at least the following technical effects: This application discloses a method for responding to a document retrieval request by converting the request into a question retrieval vector and calculating the vector similarity between the question retrieval vector and the document image retrieval vector. The document image retrieval vector is obtained by parsing the document into a document image set and encoding the document image set. Based on the vector similarity, the document retrieval result corresponding to the document retrieval request is generated. Since this application pre-parses the document into a document image set and encodes the document image set to obtain the document image retrieval vector, and performs document retrieval using the question retrieval vector and the document image retrieval vector, it achieves document retrieval in the form of document images. This preserves more spatial structure information, fine-grained information, and contextual information, thereby improving the accuracy of document retrieval. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the first embodiment of the document retrieval method of this application; Figure 2 This is a flowchart illustrating the second embodiment of the document retrieval method of this application; Figure 3 This is a flowchart illustrating the third embodiment of the document retrieval method of this application; Figure 4 This is a detailed flowchart of an embodiment of the document retrieval method of this application; Figure 5 This is a diagram illustrating the multimodal large-scale model architecture of a document retrieval method according to an embodiment of this application. Figure 6 This is a specific interface diagram of an embodiment of the document retrieval method of this application; Figure 7 This is a specific interface diagram of an embodiment of the document retrieval method of this application; Figure 8This is a schematic diagram of the module structure of the document retrieval device according to an embodiment of this application; Figure 9 This is a schematic diagram of the device structure of the hardware operating environment involved in the document retrieval method in this application embodiment.

[0029] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0030] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0031] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0032] Document retrieval refers to matching user queries with relevant documents in a given corpus. It is central to many wide-ranging industrial applications, functioning as a standalone ranking system (search engine) or as part of more complex information extraction or retrieval-augmented generation (RAG) + document understanding question-answering systems. As a core component, the performance of document retrieval significantly impacts the user experience of many applications and is a major focus of the industry. However, currently, document retrieval is primarily performed in text form, processing document content into text paragraphs for searching. While this method can solve many text-related retrieval problems, it loses much of the document's spatial structure and fine-grained information (such as footnotes, citations, text layout, charts, tables, and images). Furthermore, the fragmentation of document text paragraphs leads to a significant loss of contextual information, affecting the accuracy of document retrieval.

[0033] Therefore, to overcome the above-mentioned shortcomings, this application provides a solution comprising: responding to a document retrieval request, converting the document retrieval request into a question retrieval vector, calculating the vector similarity between the question retrieval vector and the document image retrieval vector, wherein the document image retrieval vector is obtained by parsing the document into a document image set and encoding the document image set, and generating the document retrieval result corresponding to the document retrieval request based on the vector similarity; since this application pre-parses the document into a document image set and encodes the document image set to obtain the document image retrieval vector, and performs document retrieval by comparing the question retrieval vector and the document image retrieval vector, it realizes document retrieval in the form of document images, thereby preserving more spatial structure information, fine-grained information and contextual information, and improving the accuracy of document retrieval.

[0034] It should be noted that the executing entity in this embodiment can be a document retrieval device with data processing, network communication and program running functions, such as a computer, server, etc., or other electronic devices that can achieve the same or similar functions. This embodiment does not limit this.

[0035] Based on this, embodiments of this application provide a document retrieval method, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the document retrieval method of this application.

[0036] In the first embodiment, the document retrieval method includes: Step S10: In response to the document retrieval request, the document retrieval request is converted into a question retrieval vector.

[0037] It should be understood that, in order to address the difficulty of directly comparing natural language queries with document images, this embodiment transforms document retrieval requests into question retrieval vectors, enabling these vectors to accurately represent the semantics of the query. A document retrieval request can refer to a user-initiated instruction to obtain target document information, including the user's query question (e.g., "What is the title of my graduation thesis?") and the required number of search results to be returned (i.e., top-k, such as returning the first or third most relevant results), which serves as the starting condition for triggering subsequent retrieval processes. A question retrieval vector can be a numerical vector generated by encoding the query question in the user's document retrieval request, accurately representing the semantic information of the query question.

[0038] In the specific implementation, after receiving the user's retrieval request containing the query question and top-k, the query question in natural language form is transformed into a machine-computable question retrieval vector. Specifically, the question text can be converted into a token, the question text features can be generated by a Large Language Model (LLM), and then the final question retrieval vector can be converted by a Multi-layer Perceptron (MLP) to ensure that the vector accurately matches the question semantics.

[0039] Step S20: Calculate the vector similarity between the question retrieval vector and the document image retrieval vector, wherein the document image retrieval vector is obtained by parsing the document into a document image set and encoding the document image set.

[0040] It's understandable that a document image retrieval vector can refer to a numerical vector generated through a document parsing-image encoding process. First, documents in formats like PDF, DOC, and TXT are parsed into a set of document images containing complete spatial structure (such as text layout and footnote positions) and fine-grained information (such as charts, tables, and images). Then, this document image set is encoded, preserving key non-textual information from the document. A document image set can refer to a collection of images generated after parsing the original document (such as financial reports, papers, or documents). Each image corresponds to a page or a complete information unit of the document, forming the basis for generating document image retrieval vectors. Compared to text paragraphs, it avoids contextual fragmentation and spatial information loss. Vector similarity can refer to a quantitative indicator (such as cosine similarity) used to measure the correlation between the query retrieval vector and the document image retrieval vector. It determines the relevance between the query question and the document image by calculating the spatial distance between the two vectors, serving as the core basis for subsequent ranking.

[0041] In the specific implementation, the vector calculation logic is called to compare the spatial distance between the question retrieval vector and all document image retrieval vectors in the library, and output the quantified vector similarity (i.e., relevance score). Since the document image retrieval vector retains the document spatial structure and fine-grained information, it avoids the relevance judgment bias caused by optical character recognition (OCR) errors or text fragmentation in traditional text retrieval.

[0042] Step S30: Generate the document retrieval results corresponding to the document retrieval request based on the vector similarity.

[0043] It should be understood that document retrieval results can refer to the final output generated according to the user's top-k requirements after sorting by vector similarity. This output includes the top k most relevant document images and their corresponding original document information (such as document name and source). The document images retain complete spatial structure and fine-grained information, which can directly support downstream tasks (such as RAG question answering).

[0044] In the specific implementation, the document images corresponding to the document image retrieval vectors are sorted from high to low according to the vector similarity score. Combined with the top-k parameters (such as top-1, top-3) in the user's search request, the top k most relevant document images are selected. At the same time, the corresponding original document information (such as document name, storage path) is associated to form a complete document retrieval result and return it to the user.

[0045] In this embodiment, the document is pre-parsed into a document image set and encoded to obtain a document image retrieval vector. The document is then retrieved by combining the question retrieval vector with the document image retrieval vector, thereby enabling document retrieval in the form of document images. This allows for the preservation of more spatial structure information, fine-grained information, and contextual information, thus improving the accuracy of document retrieval.

[0046] Reference Figure 2 , Figure 2 This is a flowchart illustrating the second embodiment of the document retrieval method of this application, based on the above. Figure 1 The first embodiment shown illustrates a second embodiment of the document retrieval method of this application.

[0047] In the second embodiment, before step S10, the method further includes: Step S01: Convert the document dataset into a document image set, and encode the images in the document image set using a preset multimodal large model to obtain document image retrieval vectors.

[0048] It should be understood that, in order to achieve the fusion of visual and semantic information and enable the generated document image retrieval vector to accurately represent the core information of the document, in this embodiment, the images in the document image set are encoded by a preset multimodal large model to obtain the document image retrieval vector, and a local document library is constructed based on the document image retrieval vector, and the vector similarity between the question retrieval vector and the document image retrieval vectors in the local document library is calculated.

[0049] The document dataset refers to the original collection of documents used to build the retrieval library, including various formats such as PDF, DOC, and TXT, and covering a wide range of types such as financial reports, papers, documents, books, and exam papers. It serves as the basic data source for generating the document image set. The pre-trained multimodal large model refers to a pre-trained multimodal model specifically designed for document image retrieval (such as the document retrieval multimodal large model). Its architecture includes a visual encoder (Vision Transformer, ViT), a large language model (LLM), and a multi-layer perceptron (MLP), which can process image and text information simultaneously, enabling document image encoding and question encoding.

[0050] In the specific implementation, the input document dataset (including PDF, DOC and other formats) is first parsed and processed, and each document is split into independent images according to pages or information units to form a document image set (e.g., "Computer Science and Technology Graduation Thesis.pdf" is parsed into "Graduation Task Page Image", "Acknowledgement Page Image", and "Main Text Page Image"). Then, a preset multimodal large model is called to encode the images in the document image set to obtain the document image retrieval vector.

[0051] Step S02: Construct a local document library based on the document image retrieval vector.

[0052] It should be understood that a local document library can refer to a database built around document image retrieval vectors, which not only stores the encoded vectors, but also associates them with the corresponding document images and original document information (such as document name, format, and storage path).

[0053] In the specific implementation, the encoded document image retrieval vector is used as the core index and associated with the corresponding document image and the original document's metadata (such as document name, size, and format) for storage, forming a structured local document library.

[0054] Accordingly, step S20 includes: Step S20': Calculate the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the local document library.

[0055] In the specific implementation, the document images corresponding to the document image retrieval vectors in the local document library are sorted from high to low according to the vector similarity score. Combined with the top-k parameters (such as top-1, top-3) in the user's search request, the top k most relevant document images are selected. At the same time, the corresponding original document information (such as document name, storage path) is associated to form a complete document search result and return it to the user.

[0056] This embodiment encodes images in the document image set using a pre-set multimodal large model to obtain document image retrieval vectors. Based on these vectors, a local document library is constructed. The similarity between the question retrieval vector and the document image retrieval vectors in the local document library is calculated, thereby enabling the fusion of visual and semantic information. This allows the generated document image retrieval vectors to accurately represent the core information of the document.

[0057] Furthermore, to improve the accuracy of document image retrieval vectors, the preset multimodal large model includes a visual encoder, a multilayer perceptron, and a large language model. The step of encoding images in the document image set using the preset multimodal large model to obtain document image retrieval vectors includes: encoding visual features of images in the document image set using the visual encoder to obtain visual features; projecting the visual features onto the text semantic space using the multilayer perceptron to obtain projected features; extracting image features with fused semantics from the projected features using the large language model; and converting the image features with fused semantics into document image retrieval vectors using the multilayer perceptron.

[0058] It should be understood that a visual encoder (ViT) can refer to the core component in a multimodal document retrieval model responsible for extracting visual information from document images. It captures spatial structural details (such as text layout, chart position, and table row and column distribution) and fine-grained visual features (such as font style and image pixel information) within document images, and is a key module for transforming document images into machine-processable features. Visual features can refer to the feature data generated by the visual encoder after encoding individual images in a document image set. These features quantify the visual attributes (such as text position, image outline, and table borders) and basic content information of the image, serving as the foundational feature form for subsequent fusion with text semantics. A multilayer perceptron (MLP) can refer to a neural network structure containing input, hidden, and output layers, responsible for feature transformation and spatial alignment. It projects visual features into the text semantic space and converts the image features used for semantic fusion into document image retrieval vectors, achieving format adaptation and dimensional unification for different types of features. The text semantic space can refer to the feature space upon which a Large Language Model (LLM) relies to process text information. Features in this space accurately represent the semantic meaning of the text (e.g., the semantic features corresponding to "thesis title"). Projecting visual features into this space enables cross-modal alignment of "visual information - text semantics," laying the foundation for subsequent semantic fusion. Projected features refer to the feature data generated after visual features are transformed by an MLP. Their dimensions are consistent with the feature dimensions of the text semantic space, and they retain the core visual information of the document image (e.g., table structure, chart trends). They also possess the ability to interact with text features, serving as a bridge connecting visual and semantic information. The Large Language Model (LLM) can refer to the core component responsible for semantic processing in a multimodal document retrieval model. Trained on massive amounts of Chinese and English text data (including document retrieval-related corpora), it can understand the semantic logic of the text and extract and fuse text semantic information from projected features, generating features that combine visual details and semantic connotations. Image features with fused semantics can refer to feature data generated by LLM after projected feature processing. It includes both visual structural information of the document image (such as the location and font size of the "title" in the "graduation task book page") and semantic information of the corresponding content, achieving a deep combination of "visual and semantic".

[0059] In the specific implementation, firstly, visual features are extracted through a visual encoder: a single document image from the document image set (such as the "graduation task page image" of a "graduation thesis for computer science and technology major") is input into the ViT visual encoder. ViT captures the spatial structure (such as the "title" text being located in the center at the top of the page and the "guiding ideology" paragraph being located at the bottom) and visual details (such as the "title" font being bolded and the page containing the "XXXX College" school name logo) in the image by segmenting the image into multiple image blocks and calculating the attention relationship between the blocks. Finally, it outputs visual features that can quantitatively represent this information.

[0060] Secondly, visual features are projected into the text semantic space using MLP: Since visual features and text features belong to different spaces and cannot directly interact semantically, MLP is called to perform linear transformation and non-linear mapping on the visual features generated in the first step, adjusting the dimensions of the visual features to be consistent with the semantic feature dimensions when LLM processes text. At the same time, the feature distribution is optimized to ensure that the projected features can retain the core visual information (such as the position of the title on the "graduation task book page") and adapt to the feature interaction rules of the text semantic space, thus generating the projected features.

[0061] Secondly, the LLM extracts semantic image features: the projected features are input into the pre-trained LLM (which is trained on 500,000 Chinese and English query-document image pairs, covering multiple document scenarios such as papers and financial reports). The LLM combines its own semantic understanding ability to identify and fuse the semantic information of the document images from the projected features. For example, from the projected features of the "graduation task page", the core semantics such as "graduation thesis title" are extracted and deeply combined with the original visual structural information (title position, paragraph layout) to generate semantic image features.

[0062] Finally, the image features of the fused semantics are converted into document image retrieval vectors using MLP: To meet the requirements of subsequent vector similarity calculation (which requires vectors of fixed dimensions and uniform format), MLP is called again to process the image features of the fused semantics. Through feature compression and optimization in the hidden layer, the high-dimensional fused features are converted into numerical vectors of fixed dimensions (such as 512-dimensional or 1024-dimensional). This vector contains both the visual structural information of "graduation task page" and the semantic information of "graduation thesis title is AAAA", ultimately forming a document image retrieval vector that can be used for retrieval.

[0063] Furthermore, in order to improve the model performance of the preset multimodal large model, before step S01, the method further includes: constructing a training dataset, wherein the training dataset includes training data pairs consisting of query statements and document images; and training the initial multimodal large model based on the training dataset to obtain the preset multimodal large model.

[0064] It is understandable that the training dataset can refer to a dataset containing a large amount of labeled information specifically constructed for training multimodal large models. Its core is the correspondence between "query statements" and "document images," aiming to provide the model with a sample foundation for learning the "association between query intent and document images," adapting to Chinese and English languages, multiple document types, and single / multi-page query scenarios. Document images can refer to image data in the training dataset paired with query statements, parsed from original documents (such as financial reports, papers, documents, books, and exam papers) in formats such as PDF, DOC, and TXT. These images fully preserve the document's spatial structure information (such as text layout and footnote positions) and fine-grained information (such as charts, tables, and images), serving as the core input for the model to learn "document content representation." Training data pairs refer to the basic units in the training dataset, consisting of a query statement and one (or multiple related) document images. These are categorized as relevant or irrelevant pairs. Relevant pairs allow the query statement to be found in the corresponding document image (e.g., "What is the title of my graduation thesis?" and a document image containing the thesis title), while irrelevant pairs have no corresponding relationship (e.g., "What is the title of my graduation thesis?" and a document image containing an internet service agreement). These pairs are used to train the model to distinguish relevance. The initial multimodal large model refers to a model framework that has not been trained, possesses a basic multimodal processing architecture, but lacks document retrieval capabilities. Its architecture includes a visual encoder (ViT), a large language model (LLM), and a multilayer perceptron (MLP). ViT processes image information, LLM processes text information, and MLP handles feature transformation and spatial alignment. It needs to learn the association patterns between query statements and document images through the training dataset to acquire retrieval capabilities.

[0065] In the specific implementation, the English dataset is constructed as follows: the total scale is approximately 120,000 training data pairs, which come from two sources: first, open-source academic datasets, such as DocVQA (Document Visual Question Answering Dataset), arXiVQA (arxiv Paper Question Answering Dataset), and InfoVQA (Information-based Document Question Answering Dataset), directly extracting "query statement - document image" pairs from the datasets; second, synthetic datasets, which obtain a large number of English PDF documents (covering academic papers, technical reports, etc.) from the Internet, first parsing the documents into document image sets, and then using a visual language model (VLM) to generate query statements that match the image content (such as generating "What is the experimental conclusion shown in this chart?" for a paper's chart), forming synthetic training data pairs.

[0066] Chinese dataset construction: The total scale is approximately 380,000 training data pairs. The data source comes from Chinese documents downloaded from multiple platforms, covering types such as financial reports, academic papers, book chapters, exam papers, invoices, product manuals, career planning books, etc. (e.g., documents related to "Computer Science and Technology Graduation Thesis" and "Physical Education Teacher Career Planning Book"). These Chinese documents are first parsed into document image sets, and then adapted query statements are generated through VLM (e.g., generating "What is the title of this graduation thesis" for a graduation thesis assignment page). At the same time, it takes into account both single-page queries (e.g., "What is the amount in a certain invoice?") and multi-page queries (e.g., "On which pages is the quarterly revenue data shown in a certain financial report?"), and finally forms Chinese "query statement-document image" training data pairs. Dataset integration: The English dataset (120,000) and the Chinese dataset (380,000) were merged to form a training dataset with a total size of 500,000, ensuring the diversity of data types and query scenarios and providing a guarantee for the model's generalization ability.

[0067] Model input processing: The document images from the training data pairs are input into the ViT visual encoder of the initial model. After encoding, MLP projection onto the text semantic space, LLM extraction of image features for fused semantics, and then MLP conversion into document image candidate vectors; at the same time, the paired query statements are input into the initial model. After token conversion, LLM extraction of text features, and MLP conversion, the query statement vector is obtained.

[0068] Contrastive learning training: A contrastive learning loss function is used as the training objective. For relevant pairs (such as "What is the title of my graduation thesis?" and document images containing the thesis title), the model parameters are adjusted through the loss function to make the distance between the query vector and the corresponding document image candidate vector closer in the vector space. For irrelevant pairs (such as "What is the title of my graduation thesis?" and document images containing internet service agreements), the loss function is used to make the distance between the two greater, thereby allowing the model to learn the association between query intent and document image content.

[0069] Model optimization and validation: During training, the parameters of the ViT, LLM, and MLP components of the model are continuously adjusted iteratively. The model accuracy is validated through the business document RAG test set (including real-world scenario samples such as "thesis title retrieval"). When the model's "query-document image" matching accuracy is significantly higher than that of traditional text retrieval solutions and its generalization ability meets the standards in Chinese and English and multi-document type scenarios, training is stopped. The model at this point is the preset multimodal large model.

[0070] Furthermore, to enhance the diversity and adaptability of the training data, the construction of the training dataset includes: acquiring online documents and splitting each page of the online documents into document images; generating corresponding query statements for each page image using a visual language model, and constructing a training dataset based on the query statements and the document images.

[0071] It should be understood that a Visual Language Model (VLM) can refer to a pre-trained model that has the ability to understand image content and generate natural language. It is used to generate corresponding query statements based on the content of document images. It can accurately identify key information in document images (such as titles, table data, and chart conclusions) and convert them into natural language questions that conform to users' real search habits (such as generating "What is the title of this graduation thesis" for an image containing a thesis title). It is a key tool to supplement the diversity of training data.

[0072] In practice, based on training requirements, network documents that match the scenario are selected and downloaded from the network platform. Priority is given to documents that cover multiple types and fields. For Chinese documents, the focus is on collecting financial reports (corporate annual financial reports), academic papers (such as computer science graduation theses), documents (invoices, contracts), book chapters (textbook content), exam papers (exam questions), career planning books (such as career planning books for physical education teachers), etc. For English documents, the focus is on downloading academic PDFs (such as papers on the arXiv platform), technical reports, etc., to ensure that the document types are highly adapted to the subsequent retrieval business scenarios.

[0073] Downloaded online documents are parsed and processed using corresponding parsing tools (such as PDF parsing tools and document format conversion tools) for different formats such as PDF, DOC, and TXT. Each page is converted into a standard format image file (i.e., document image) using "page" as the smallest unit. During the splitting process, the original page's spatial layout (such as centered title and paragraph indentation) and fine-grained elements (such as table borders and chart legends) are strictly preserved to ensure the integrity of the document image information.

[0074] The individual document images, after being split, are input one by one into a pre-trained Visual Language Model (VLM). The VLM captures key information in the images through its image recognition module (such as the "title" in the "graduation assignment page image"), and then combines it with a natural language generation module to generate query statements strongly related to the image content. For the "graduation assignment page image," the VLM can generate "What is the title of the graduation thesis corresponding to this document?" or "What is the name of the graduating institution mentioned on this page?" For document images containing tables (such as the revenue table in a financial report), it can generate "What is the revenue amount in Q1 2023 in this table?" The VLM also takes into account both single-page queries (focusing on the content of a single image) and multi-page related queries (such as "On which pages in this thesis is the content related to 'local area network topology' reflected?", which requires linking multiple document images), ensuring the diversity and practicality of the query statements.

[0075] The query statements generated by VLM are paired with corresponding document images to form relevant training data pairs (e.g., "What is the title of the graduation thesis?" paired with "Image of a page from the graduation assignment book"). Simultaneously, to improve the model's ability to distinguish relevance, irrelevant training data pairs are also constructed (e.g., "What is the title of the graduation thesis?" paired with "Image of an Internet service agreement document"). Finally, these synthetic data pairs are integrated with open-source academic datasets (e.g., "query statement-document image" pairs from DocVQA and arXiVQA). The Chinese synthetic data pairs include 380,000 Chinese dataset entries, and the English synthetic data pairs include 120,000 English dataset entries, together forming a training dataset of 500,000 entries, providing sufficient samples for the initial training of the multimodal large model.

[0076] Furthermore, in order to provide complete data support for the generation of subsequent search results, step S02 includes: obtaining the document image and basic document information corresponding to the document image retrieval vector; associating and storing the document image retrieval vector, the document image, and the basic document information to construct a local document library.

[0077] Understandably, basic document information refers to metadata associated with the original document, covering the document's core identifiers and management attributes. Specifically, it includes the document name, file format, file size, storage path, and document type (such as papers, financial reports, and documents). It is important supplementary information for distinguishing different documents and assisting users in identifying search results.

[0078] In the specific implementation, the generation of document image retrieval vectors relies on the document images parsed from the original document. Therefore, during the encoding process, a one-to-one correspondence between "document image retrieval vectors and document images" is recorded. Simultaneously, basic document information, including document name, file size, format, and storage path, is extracted from the original document's file attributes. This basic information is then bound to the corresponding document images and document image retrieval vectors using association identifiers, ensuring data consistency across the three. To avoid issues such as "vector without corresponding image" or "image without basic information" during subsequent searches, the completeness of the three types of data is verified. This includes checking whether the document image retrieval vector is of a fixed dimension (e.g., 512 dimensions), whether the document image is clear and retains complete page information, and whether the basic document information covers the core fields of "name-format-size-path." If any missing information is found (e.g., an empty document name), a data processing error message is returned until the missing information is obtained before proceeding to the next stage.

[0079] Furthermore, in order to satisfy users with personalized search needs (such as enterprise users searching for internal documents) and also adapt to users who only need general search (such as students searching for public papers) and improve user experience, before step S20', the method further includes: obtaining user operations and determining whether the user needs to build a user document library based on the user operations; correspondingly, the calculation of the vector similarity between the question search vector and the document image search vectors in the local document library includes: if it is not necessary to build a user document library, then calculating the vector similarity between the question search vector and the document image search vectors in the local document library.

[0080] It should be understood that user actions refer to interactive behaviors initiated by users on the document retrieval system to express retrieval needs or select functions. These include, but are not limited to, entering query questions (such as "What is the title of my graduation thesis?"), setting the number of top-k search results, clicking the "Upload Document" button (for building a personal library), and selecting the "Use Default Library for Retrieval" option. These actions are the core basis for the system to determine whether a user needs to build a user document library, directly reflecting the user's retrieval scenario preferences (general retrieval / personalized retrieval). A user document library can refer to personalized documents uploaded by users themselves (such as personal graduation theses, internal company reports, and exclusive learning materials). This database, constructed according to the process of "document parsing into an image set → encoding into a document image retrieval vector → associated storage," is exclusively accessible to the user and caters to their retrieval needs for private or specific documents. It differs from the local document library pre-built by the business side. The local document library can refer to a structured database pre-built by the business side containing a full range of general-purpose document data. Document sources include publicly available financial reports, academic papers, book chapters, documents, and manuals. The library stores associated data of "document image retrieval vector - document image - document basic information," which is directly used by users who have not built their own personal databases and serves as the core data carrier for general retrieval scenarios.

[0081] In the specific implementation, the user's interactive behavior on the search interface is captured in real time, including but not limited to: (1) core search operations (entering the query question, setting the top-k); (2) library selection operations (clicking the "upload document" button, checking the "use my document library" option, uploading a local document file package); (3) default operations (entering only the question without selecting any library, not uploading any document). These operation signals are converted into system-recognizable instructions (such as "upload document instruction" and "no library selection instruction"). Based on the acquired operation instructions, if the user only enters the query question (such as "What is the title of the graduation thesis?"), sets the top-k (such as top-1), does not perform any document upload or personal library selection operations, or actively selects the "use default document library" option, it is determined that there is no need to build a user document library, and the system directly enters the vector similarity calculation stage of general search. The entire judgment process does not require manual intervention and is automatically executed by the system according to preset rules to ensure response efficiency.

[0082] Furthermore, in order to meet personalized retrieval needs, after obtaining user operations and determining whether the user needs to build a user document library based on the user operations, the method further includes: if a user document library needs to be built, then building the user document library based on the documents uploaded by the user; and calculating the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the user document library.

[0083] It is understandable that user-uploaded documents can refer to original document resources submitted by users to build their own document library. The formats include PDF, DOC, TXT, etc. The content is mostly private information of users or information that needs to be searched in a specific way. It is the basic data source for building user document libraries. It needs to be processed by parsing, encoding and other methods to be transformed into searchable vector and image data.

[0084] In its implementation, the system receives user-uploaded documents, calls a document parsing tool, and converts each page of the document into a standard-format document image, using "page" as the smallest unit. During parsing, the spatial structure and fine-grained information of the original document are strictly preserved, forming a user-specific document image set. The parsed user document image set is then input into a pre-defined multimodal large model, generating vectors according to the encoding process: first, visual features of the images are extracted using ViT, then projected onto the text semantic space using MLP, followed by semantic information fusion using LLM, and finally converted into a fixed-dimensional (e.g., 512-dimensional) document image retrieval vector using MLP. Basic information about the user-uploaded documents (such as document name, file format, file size, and storage path) is extracted, and a unique identifier is used to bind "document image retrieval vector - corresponding document image - document basic information," storing this information in the user-specific database (i.e., the user document library). Access permissions are set (only the user can retrieve this information), completing the construction of the user document library.

[0085] Reference Figure 3 , Figure 3 This is a flowchart illustrating the third embodiment of the document retrieval method of this application. Based on the above embodiments, a third embodiment of the document retrieval method of this application is proposed.

[0086] In the third embodiment, step S10 includes: Step S101: In response to the document retrieval request, extract the query question input by the user from the document retrieval request.

[0087] It should be understood that, in order to ensure semantic accuracy and improve retrieval precision and efficiency, in this embodiment, the query question input by the user is extracted from the document retrieval request, and the query question is transformed into a question retrieval vector through a preset multimodal large model.

[0088] In the specific implementation, the document retrieval request is first structured and parsed to identify two types of information in the request: one is the core need expressed by the user in natural language (i.e., the potential query question, such as "What is the title of my graduation thesis?"), and the other is auxiliary retrieval parameters (i.e., top-k, such as "top-1" and "top-3"). The attributes of the two types of information are distinguished through preset semantic recognition rules (e.g., statements containing interrogative words such as "what is", "how much", and "where" are query questions, and content in the format of "top-XX" is a quantity parameter). After eliminating interference from auxiliary parameters (top-k), the core need statement is determined as the query question. For example, if the user submits a document retrieval request of "Query 'What is the title of my graduation thesis?', return top-1 results", the system will extract "What is the title of my graduation thesis?" as the query question after parsing, while temporarily storing the top-1 parameters for subsequent result generation. This ensures that the extracted query question accurately reflects the user's search target and avoids misjudgment of needs caused by parameter interference.

[0089] Step S102: Transform the query question into a question retrieval vector using a preset multimodal large model.

[0090] In a specific implementation, the query question can be transformed into a question retrieval vector by pre-setting a multimodal large model. This can be achieved by inputting the query question into a large language model and extracting the textual semantic features of the query question through the large language model; or by using a multilayer perceptron to convert the textual semantic features into a question retrieval vector.

[0091] For example, the extracted query question (such as "What is the title of my graduation thesis?") is first input into the model's codebook component. The codebook breaks down the natural language statement into the smallest language units that the model can process (i.e., tokens, such as "graduation", "thesis", "title", and "what"), completing the initial conversion from natural language to the model's input format and preparing for subsequent semantic extraction. The tokenized text is then input into the model's LLM (Large Language Model). Based on its semantic understanding capabilities learned from the training data, the LLM identifies the core semantics of the query question. For example, it extracts the two key semantic elements "graduation thesis" and "title" from "What is the title of my graduation thesis?", ignoring interrogative particles like "what", and generates text semantic features that represent the user's core needs, ensuring that the features accurately match the target information in the document. The text semantic features generated by LLM are input into the MLP (Multilayer Perceptron) component of the model. MLP transforms high-dimensional, unstructured semantic features into fixed-dimensional (e.g., 512-dimensional) numerical vectors, i.e., question retrieval vectors, through linear transformation and nonlinear mapping. The dimension of this vector is consistent with that of the document image retrieval vector (generated by the same model), ensuring that vector similarity calculation can be performed subsequently. At the same time, the vector format can be adapted to fast numerical operations, improving retrieval efficiency.

[0092] This embodiment extracts the user-input query question from the document retrieval request and transforms the query question into a query retrieval vector through a preset multimodal large model, thereby ensuring semantic accuracy and improving retrieval precision and efficiency.

[0093] In the third embodiment, step S30 includes: Step S301: Extract the required number of search results returned by the user input from the document retrieval request.

[0094] It should be understood that, in order to accurately capture users' preferences for the number of results and avoid redundancy caused by defaulting to a fixed number of results, this embodiment first extracts the user's input requirement for the number of search results to be returned from the document search request, and then generates the document search results corresponding to the document search request based on vector similarity and the requirement for the number of search results to be returned. The requirement for the number of search results to be returned can refer to the user-specified parameter for the number of most relevant documents to be returned, extracted from the document search request. It can be presented in "top-k" format (k is a positive integer, such as top-1, top-3), where "top" indicates "ranked highly" and "k" represents the specific number, directly determining the final number of search results returned and serving as the core basis for subsequent result filtering.

[0095] Step S302: Generate the document retrieval results corresponding to the document retrieval request based on the vector similarity and the required number of retrieval results returned.

[0096] In the specific implementation, the document retrieval request submitted by the user is first fully parsed to identify two types of core information in the request: one is the query question in natural language (such as "What is the title of my graduation thesis?"), and the other is the quantity requirement parameter (which can be presented in "top-k" format, such as "return top-1" or "require the first 3 results"). Using preset semantic matching rules (such as recognizing keyword formats like "top-XX", "first XX", or "return XX results", where "XX" is a number), the attributes of the quantity parameter and the query question are distinguished. After eliminating interference from the query question, the content matching the quantity parameter format is converted into the standard "top-k" form. For example, if a user submits a request "Query 'What is the title of my graduation thesis?', please return the first most relevant document", "1" will be extracted as the k value and converted into a standard "top-1" quantity requirement. Simultaneously, the reasonableness of the k value is verified (it must be a positive integer and not exceed the number of valid documents in the document library to avoid invalid requests such as "k=100 but there are only 50 data entries in the library"). If the k value is abnormal, the user is prompted to correct it to ensure that the extracted quantity requirement conforms to the retrieval logic.

[0097] The similarity between the previously calculated question retrieval vector and the document image retrieval vector is used to sort all document images corresponding to the document image retrieval vectors from highest to lowest similarity. For example, in the local document library, the vector corresponding to "Graduation Task Book Page Image" has a similarity of 0.92 (highest) to the question vector "What is the title of the graduation thesis?", the vector similarity of "Acknowledgments Page Image" is 0.65, and the vector similarity of "Internet Service Agreement Image" is 0.1. The sorted order is Graduation Task Book Page Image > Acknowledgments Page Image > Internet Service Agreement Image. Based on the "top-k" extraction requirement, the top k document images are extracted from the sorted document image list. For example, if the requirement is "top-1", only the top-ranked "Graduation Task Book Page Image" is retained; if the requirement is "top-3", the first three images (Graduation Task Book Page, Acknowledgments Page, and Internet Service Agreement Image) are retained, excluding subsequent images with lower similarity, ensuring that the result quantity perfectly matches the user's needs. By using the unique association identifier of the document image retrieval vector, the basic document information corresponding to the filtered document images (such as the document name, format, and size associated with "graduation task book page image") is retrieved. The document images and basic document information are integrated into a structured search result, which is then returned to the user, ensuring that the user can not only intuitively view the document content but also quickly identify the document identity.

[0098] This embodiment first extracts the user's input requirement for the number of search results returned from the document retrieval request, and then generates the document retrieval results corresponding to the document retrieval request based on vector similarity and the requirement for the number of search results returned. This can accurately capture the user's preference for the number of results and avoid redundancy caused by returning a fixed number of results by default.

[0099] For ease of understanding, please refer to Figure 4 This explanation is provided, but does not limit the scope of this application. Figure 4 This is a detailed flowchart of an embodiment of the document retrieval method of this application, as an example. Figure 4 In China, document retrieval methods mainly consist of the following two parts: 1. Database Construction (Building a document and image database): (1) Parse and process document data such as pdf, doc, and txt into document image sets; (2) Use the trained multimodal retrieval model to encode document images into image retrieval vectors; (3) Input the encoded document image data into the database to build a local document image database.

[0100] 2. Search Application Section: (1) The user inputs a question and the search returns the top-k number of results. The system receives the question and uses a trained multimodal search model to encode the input question into a question search vector. (2) Determine whether the user wants to build their own user document database. If the user wants to build their own user database, then build the user document database according to the database building process, and then perform relevance scoring based on the user document database; if the user does not build their own database, then directly use the business's local document database (containing all document data) to perform relevance scoring. (3) Based on the input top-k and scoring results, sort and select the top-k most relevant document images; (4) Return the top-k search results to the user, including the top-k document images and documents and other related information.

[0101] For ease of understanding, please refer to Figure 5 This explanation is provided, but does not limit the scope of this application. Figure 5 This is a diagram illustrating the multimodal large-scale model architecture of a document retrieval method according to an embodiment of this application, as an example. Figure 5 The model architecture and training method are as follows: 1. Training Data: The dataset contains approximately 500,000 rich Chinese and English documents of various types. The English dataset contains approximately 120,000 query-document image pairs, consisting of an open-source academic dataset (DocVQA, arXiVQA, and InfoVQA, etc.) and a synthetic dataset (composed of PDF documents crawled from the web, expanded with questions generated using VLM). The Chinese dataset contains approximately 380,000 query-document image pairs, collected from multiple sites, including financial reports, papers, articles, books, exam papers, invoices, reports, instructions, and other richly varied document types. Query-document image pairs were then generated using VLM, with queries ranging from single-page to multi-page.

[0102] 2. Model Architecture: The overall architecture of the multimodal large-scale document retrieval model is as follows: Figure 5 As shown, the entire training process employs an end-to-end approach. Document images are encoded using VIT, then image tokens are projected onto the text token semantic space using MLP, document image features are generated using LLM, and finally converted into document image retrieval vectors using MLP. Query text is converted into tokens using a codebook, then query text features are directly generated using LLM, and finally converted into query retrieval vectors using MLP. Finally, contrastive learning loss is used for supervised learning, ensuring that queries are close to relevant document images and far from irrelevant ones.

[0103] For ease of understanding, please refer to Figure 6 and Figure 7 This explanation is provided, but does not limit the scope of this application. Figure 6and Figure 7 This is a specific interface diagram of one embodiment of the document retrieval method of this application. As an example, if the user builds their own user database, they need to upload a document dataset to build the database. If they do not need to build their own user database, they do not need to upload a document dataset and can directly use the local document database by default. Then, the user inputs a question (query) and top-k (the retrieval returns the k most relevant document images). Finally, the model encodes the question (query), calculates scores and ranks them, and returns the retrieval results based on the top-k parameters. The yellow box shows the user providing a document set to build the user retrieval database; the red box shows the user inputting a question (query) and top-k; the green box shows the retrieval results returned according to the top-k requirements, with the answer document page in the top 1 position.

[0104] It should be noted that the data collection, tag management, rule setting, and push decision-making processes involved in this application are designed to work with other technical features to solve technical problems. They do not involve or support any illegal activities. Any data processing that may violate laws and regulations (such as unauthorized collection of privacy data, generation of discriminatory tags, setting unfair rules, or pushing illegal information) is not within the scope of protection of this application's technical solution. Of course, user data in this application will be encrypted, anonymized, or de-identified before storage to ensure data security. The above examples are only for understanding this application and do not constitute a limitation on the document retrieval method of this application. Any simple modifications based on this technical concept are within the scope of protection of this application.

[0105] This application also provides a document retrieval device; please refer to [reference needed]. Figure 8 The document retrieval device includes: Vector conversion module 10 is used to convert a document retrieval request into a question retrieval vector in response to a document retrieval request; The similarity calculation module 20 is used to calculate the vector similarity between the question retrieval vector and the document image retrieval vector, wherein the document image retrieval vector is obtained by parsing the document into a document image set and encoding the document image set; The result generation module 30 is used to generate document retrieval results corresponding to the document retrieval request based on the vector similarity.

[0106] The document retrieval device provided in this application, employing the document retrieval method in the above embodiments, can solve the technical problem that related document retrieval methods lose a lot of spatial structure information, fine-grained information, and contextual information of documents, thus affecting the accuracy of document retrieval. Compared with the prior art, the beneficial effects of the document retrieval device provided in this application are the same as those of the document retrieval method provided in the above embodiments, and other technical features in the document retrieval device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.

[0107] This application provides a document retrieval device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the document retrieval method in Embodiment 1 above.

[0108] The following is for reference. Figure 9 The diagram illustrates a structural schematic of a document retrieval device suitable for implementing embodiments of this application. The document retrieval device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 9 The document retrieval device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0109] like Figure 9As shown, the document retrieval device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in ROM (Read Only Memory) 1002 or a program loaded from storage device 1003 into RAM (Random Access Memory) 1004. RAM 1004 also stores various programs and data required for the operation of the document retrieval device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via bus 1005. Input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touch screens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, LCDs (Liquid Crystal Displays), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the document retrieval device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show document retrieval devices with various systems, it should be understood that implementing or having all of the systems shown is not required. More or fewer systems may be implemented alternatively.

[0110] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0111] The document retrieval device provided in this application, employing the document retrieval method described in the above embodiments, can solve the technical problem that related document retrieval methods lose a lot of spatial structure information, fine-grained information, and contextual information of documents, thus affecting the accuracy of document retrieval. Compared with the prior art, the beneficial effects of the document retrieval device provided in this application are the same as those of the document retrieval method provided in the above embodiments, and other technical features in this document retrieval device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0112] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0113] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0114] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, which are used to execute the document retrieval method in the above embodiments.

[0115] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described document retrieval method. This solves the technical problem that related document retrieval methods often lose a lot of spatial structure information, fine-grained information, and contextual information, thus affecting the accuracy of document retrieval. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the document retrieval method provided in the above embodiments, and will not be repeated here.

[0116] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the document retrieval method described above.

[0117] The computer program product provided in this application can solve the technical problem that related document retrieval methods lose a lot of spatial structure information, fine-grained information, and contextual information of documents, thus affecting the accuracy of document retrieval. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as the beneficial effects of the document retrieval method provided in the above embodiments, and will not be repeated here.

[0118] The above description is only a part of the embodiments of this application and does not limit the scope of this application. All equivalent structural transformations made under the technical concept of this application and using the content of this application specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of this application.

[0119] This application discloses A1, a document retrieval method, the document retrieval method comprising: In response to a document retrieval request, the document retrieval request is transformed into a question retrieval vector; Calculate the vector similarity between the question retrieval vector and the document image retrieval vector, wherein the document image retrieval vector is obtained by parsing the document into a document image set and encoding the document image set; The document retrieval results corresponding to the document retrieval request are generated based on the vector similarity.

[0120] A2. The document retrieval method as described in A1, before converting the document retrieval request into a question retrieval vector in response to the document retrieval request, further includes: The document dataset is transformed into a document image set, and the images in the document image set are encoded using a preset multimodal large model to obtain document image retrieval vectors. A local document library is constructed based on the document image retrieval vectors; Accordingly, calculating the vector similarity between the question retrieval vector and the document image retrieval vector includes: Calculate the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the local document library.

[0121] A3. The document retrieval method as described in A2, wherein the preset multimodal large model includes a visual encoder, a multilayer perceptron, and a large language model, and the step of encoding the images in the document image set using the preset multimodal large model to obtain document image retrieval vectors includes: The visual encoder is used to encode the visual features of the images in the document image set to obtain visual features; The visual features are projected onto the text semantic space using the multilayer perceptron to obtain the projected features. The large language model is used to extract image features with fused semantics from the projected features; The image features of the fused semantics are converted into document image retrieval vectors using the multilayer perceptron.

[0122] A4. The document retrieval method as described in A2, wherein constructing a local document library based on the document image retrieval vector includes: Obtain the document image and basic document information corresponding to the document image retrieval vector; The document image retrieval vector, the document image, and the basic document information are associated and stored to construct a local document library.

[0123] A5. The document retrieval method as described in A2, before converting the document dataset into a document image set and encoding the images in the document image set using a preset multimodal large model to obtain the document image retrieval vector, further includes: Construct a training dataset, wherein the training dataset includes training data pairs consisting of query statements and document images; The initial multimodal large model is trained based on the training dataset to obtain the preset multimodal large model.

[0124] A6. The document retrieval method as described in A5, wherein constructing the training dataset includes: Obtain a web document and split each page of the web document into document images; A visual language model is used to generate corresponding query statements for each page of images, and a training dataset is constructed based on the query statements and the document images.

[0125] A7. The document retrieval method as described in A2, before calculating the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the local document library, further includes: Obtain user actions and determine whether the user needs to build a user document library based on the user actions; Accordingly, calculating the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the local document library includes: If it is not necessary to build a user document library, then calculate the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the local document library.

[0126] A8. The document retrieval method as described in A7, after obtaining user operations and determining whether the user needs to build a user document library based on the user operations, further includes: If a user document library needs to be built, then the user document library is built based on the documents uploaded by users; Calculate the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the user document library.

[0127] A9. The document retrieval method as described in any one of A1 to A8, wherein the step of converting the document retrieval request into a question retrieval vector in response to the document retrieval request includes: In response to a document retrieval request, the query question input by the user is extracted from the document retrieval request; The query question is transformed into a question retrieval vector by using a pre-set multimodal large model.

[0128] A10. The document retrieval method as described in A9, wherein the preset multimodal large model includes a multilayer perceptron and a large language model, and the step of transforming the query question into a question retrieval vector through the preset multimodal large model includes: The query question is input into the large language model, and the textual semantic features of the query question are extracted through the large language model. The semantic features of the text are converted into question retrieval vectors using a multilayer perceptron.

[0129] A11. The document retrieval method as described in any one of A1 to A8, wherein generating the document retrieval result corresponding to the document retrieval request based on the vector similarity includes: The required number of search results to be returned from the document retrieval request is extracted from the user's input. The document retrieval results corresponding to the document retrieval request are generated based on the vector similarity and the required number of retrieval results returned.

[0130] This application also discloses B12, a document retrieval device, the document retrieval device comprising: The vector conversion module is used to convert a document retrieval request into a question retrieval vector in response to the document retrieval request. The similarity calculation module is used to calculate the vector similarity between the question retrieval vector and the document image retrieval vector, wherein the document image retrieval vector is obtained by parsing the document into a document image set and encoding the document image set; The result generation module is used to generate document retrieval results corresponding to the document retrieval request based on the vector similarity.

[0131] B13. The document retrieval device as described in 12, further comprising: The local library construction module is used to convert the document dataset into a document image set, and encode the images in the document image set using a preset multimodal large model to obtain document image retrieval vectors; and construct a local document library based on the document image retrieval vectors. Accordingly, the similarity calculation module is also used to calculate the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the local document library.

[0132] B14. The document retrieval device as described in B13, wherein the preset multimodal large model includes a visual encoder, a multilayer perceptron, and a large language model, and the local library construction module is further configured to: encode visual features of images in the document image set using the visual encoder to obtain visual features; project the visual features onto the text semantic space using the multilayer perceptron to obtain projected features; extract fused semantic image features from the projected features using the large language model; and convert the fused semantic image features into document image retrieval vectors using the multilayer perceptron.

[0133] B15. In the document retrieval device described in B13, the local library construction module is further configured to obtain the document image and basic document information corresponding to the document image retrieval vector; and to associate and store the document image retrieval vector, the document image, and the basic document information to construct a local document library.

[0134] B16. The document retrieval device as described in B13, further comprising: The model training module is used to construct a training dataset, wherein the training dataset includes training data pairs consisting of query statements and document images; and to train an initial multimodal large model based on the training dataset to obtain a preset multimodal large model.

[0135] B17. In the document retrieval device as described in B16, the model training module is further configured to acquire online documents and split each page of the online documents into document images; generate corresponding query statements for each page image using a visual language model; and construct a training dataset based on the query statements and the document images.

[0136] This application also discloses C18, a document retrieval device, the document retrieval device comprising: a memory, a processor, and a document retrieval program stored in the memory and executable on the processor, wherein the document retrieval program, when executed by the processor, implements the document retrieval method as described above.

[0137] This application also discloses D19, a storage medium storing a document retrieval program, which, when executed by a processor, implements the document retrieval method as described above.

[0138] This application also discloses E20, a computer program product including a document retrieval program, which, when executed by a processor, implements the document retrieval method as described above.

Claims

1. A document retrieval method, characterized in that, The document retrieval method includes: In response to a document retrieval request, the document retrieval request is transformed into a question retrieval vector; Calculate the vector similarity between the question retrieval vector and the document image retrieval vector, wherein the document image retrieval vector is obtained by parsing the document into a document image set and encoding the document image set; The document retrieval results corresponding to the document retrieval request are generated based on the vector similarity.

2. The document retrieval method as described in claim 1, characterized in that, Before converting the document retrieval request into a question retrieval vector in response to the document retrieval request, the method further includes: The document dataset is transformed into a document image set, and the images in the document image set are encoded using a preset multimodal large model to obtain document image retrieval vectors. A local document library is constructed based on the document image retrieval vectors; Accordingly, calculating the vector similarity between the question retrieval vector and the document image retrieval vector includes: Calculate the vector similarity between the question retrieval vector and the image retrieval vectors of each document in the local document library.

3. The document retrieval method as described in claim 2, characterized in that, The preset multimodal large model includes a visual encoder, a multilayer perceptron, and a large language model. The step of encoding images in the document image set using the preset multimodal large model to obtain document image retrieval vectors includes: The visual encoder is used to encode the visual features of the images in the document image set to obtain visual features; The visual features are projected onto the text semantic space using the multilayer perceptron to obtain the projected features. The large language model is used to extract image features with fused semantics from the projected features; The image features of the fused semantics are converted into document image retrieval vectors using the multilayer perceptron.

4. The document retrieval method as described in claim 2, characterized in that, The construction of a local document library based on the document image retrieval vector includes: Obtain the document image and basic document information corresponding to the document image retrieval vector; The document image retrieval vector, the document image, and the basic document information are associated and stored to construct a local document library.

5. The document retrieval method as described in claim 2, characterized in that, Before converting the document dataset into a document image set and encoding the images in the document image set using a preset multimodal large model to obtain the document image retrieval vector, the method further includes: Construct a training dataset, wherein the training dataset includes training data pairs consisting of query statements and document images; The initial multimodal large model is trained based on the training dataset to obtain the preset multimodal large model.

6. The document retrieval method as described in claim 5, characterized in that, The construction of the training dataset includes: Obtain a web document and split each page of the web document into document images; A visual language model is used to generate corresponding query statements for each page of images, and a training dataset is constructed based on the query statements and the document images.

7. A document retrieval device, characterized in that, The document retrieval device includes: The vector conversion module is used to convert a document retrieval request into a question retrieval vector in response to the document retrieval request. The similarity calculation module is used to calculate the vector similarity between the question retrieval vector and the document image retrieval vector, wherein the document image retrieval vector is obtained by parsing the document into a document image set and encoding the document image set; The result generation module is used to generate document retrieval results corresponding to the document retrieval request based on the vector similarity.

8. A document retrieval device, characterized in that, The document retrieval device includes: a memory, a processor, and a document retrieval program stored in the memory and executable on the processor, wherein the document retrieval program, when executed by the processor, implements the document retrieval method as described in any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a document retrieval program, which, when executed by a processor, implements the document retrieval method as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a document retrieval program, which, when executed by a processor, implements the document retrieval method as described in any one of claims 1 to 6.

Images