Question and answer method and device based on multi-modal retrieval enhancement, storage medium and processor

Abstract

The invention discloses a question answering method and device based on multi-modal retrieval enhancement, a storage medium and a processor. According to the scheme, a question text is obtained; determining a plurality of target blocks corresponding to the question text through a double-flow cross-modal attention searcher, and determining a plurality of target pages corresponding to the plurality of target blocks; generating answer information according to the question text, the plurality of target blocks and the plurality of target pages; the answer information comprises answer content and answer position coordinates. Compared with the prior art that a multi-modal retrieval enhancement generation method has fundamental limitation in the aspect of understanding and utilizing deep information and structured information of a document, and the limitation leads to inaccurate and non-traceable answers generated by a language model, the multi-modal retrieval enhancement generation method has obvious advantages.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a question-answering method, device, storage medium, and processor based on multimodal retrieval enhancement. Background Art

[0002] Retrieval-Augmented Generation (RAG) is an artificial intelligence technology that combines information retrieval and text generation. It uses the retrieval component to search for document fragments related to the input query from a large document collection or knowledge base, and then uses these fragments as context to guide the generation component to generate answers. It is used to improve the accuracy and relevance of language models when answering questions or generating content, and reduce the "hallucinations" or inaccurate answers that may be produced by language models without external information support.

[0003] In recent years, multimodal RAG has emerged to handle complex documents containing graphs and charts. Existing multimodal RAG methods typically render document pages into images and retrieve the rendered images using a pre-trained image-text model. For example, the Open CLIP model is used to map text and images into the same embedding space, allowing them to be compared. However, existing multimodal RAG methods have fundamental limitations in understanding and utilizing the deep and structured information of documents: (1) semantic and spatial splitting; (2) shallow associations in the retrieval mechanism; and (3) information distortion and untraceability of the generation process. These limitations result in inaccurate and untraceable responses generated by language models.

[0004] Faced with the many problems existing in the existing multimodal RAG methods, how to improve the interpretability and traceability of the answers generated by the language model while ensuring the accuracy of the answers generated by the language model is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] Based on the above problems, the present application provides a question-answering method, device, storage medium and processor based on multimodal retrieval enhancement, with the aim of improving the interpretability and traceability of the answers generated by the language model while ensuring the accuracy of the answers generated by the language model.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] In a first aspect, the present application provides a question-answering method based on multimodal retrieval enhancement, the method comprising:

[0008] Get the question text;

[0009] Determining, through a dual-stream cross-modal attention retriever, multiple target blocks corresponding to the question text, and determining multiple target pages corresponding to the multiple target blocks;

[0010] Answer information is generated according to the question text, the multiple target blocks and the multiple target pages; the answer information includes answer content and answer location coordinates.

[0011] Optionally, determining multiple target blocks corresponding to the question text and determining multiple target pages corresponding to the multiple target blocks by using a dual-stream cross-modal attention retriever includes:

[0012] The dual-stream cross-modal attention retriever includes a text encoder, a Transformer and a cross-modal attention mechanism;

[0013] Using the text encoder, convert the question text into a query vector;

[0014] Encoding initial fusion position vectors of multiple blocks in each page of a vector database using the Transformer to obtain target fusion position vectors of the multiple blocks; the target fusion position vector of a first block is the initial fusion position vector after enhancing the semantic representation of the first block; the first block is any block among the multiple blocks;

[0015] Using the cross-modal attention mechanism, the correlation between the target fusion position vectors of the multiple blocks and the query vector is calculated, and multiple blocks that meet the preset conditions are selected as multiple target blocks according to the correlation sorting, and multiple pages corresponding to the multiple target blocks are used as multiple target pages. Optionally, the method for obtaining the initial fusion position coding sequence of the block includes:

[0016] Using a layout analysis model to divide a page into multiple blocks, obtaining category information and position coordinate information of each block, and generating a category vector and a position coordinate vector for each block based on the category information and position coordinate information of each block;

[0017] Extract the text content of each block and generate a content semantic vector for each block;

[0018] The content semantic vector, category vector and position coordinate vector of each block are combined to obtain the initial fused position vector of each block.

[0019] Optionally, generating answer information according to the question text, the multiple target blocks, and the multiple target pages includes:

[0020] A visual pointer network is introduced to frame and mark the multiple target blocks in red, thereby obtaining multiple target blocks that are framed and marked in red;

[0021] Based on the question text, the multiple target pages and the multiple target blocks framed and marked in red, an answer is generated and the target block position coordinates corresponding to the answer are output.

[0022] Optionally, the optimization method of the dual-stream cross-modal attention retriever includes:

[0023] Optimizing the correlation calculation process between the target fusion position vector of each block and the query vector through the region alignment loss function;

[0024] Optimizing the correlation calculation process between the query vector and the vector representation of the page in the vector database by comparing the loss function; the vector representation of the page is composed of the target fusion position vectors of multiple blocks;

[0025] Based on the contrast loss function and the region alignment loss function, the dual-stream cross-modal attention retriever is optimized by adopting a multi-task joint optimization method.

[0026] A second aspect of the present application provides a question-answering device based on multimodal retrieval enhancement, the device comprising:

[0027] Question text acquisition module, used to obtain question text;

[0028] A retrieval module, configured to determine, through a dual-stream cross-modal attention retriever, a plurality of target blocks corresponding to the question text, and a plurality of target pages corresponding to the plurality of target blocks;

[0029] The answer generation module is used to generate answer information according to the question text, the multiple target blocks and the multiple target pages; the answer information includes answer content and answer location coordinates.

[0030] Optionally, the retrieval module is specifically configured to:

[0031] The dual-stream cross-modal attention retriever includes a text encoder, a Transformer and a cross-modal attention mechanism;

[0032] The text encoder is used to convert the question text into a query vector;

[0033] The Transformer is configured to encode initial fusion position vectors of multiple blocks in each page of a vector database to obtain target fusion position vectors of the multiple blocks; the target fusion position vector of a first block is the initial fusion position vector after enhancing the semantic representation of the first block; the first block is any block among the multiple blocks;

[0034] The cross-modal attention mechanism is used to calculate the correlation between the target fusion position vectors of the multiple blocks and the query vector using the cross-modal attention mechanism, select multiple blocks that meet preset conditions as multiple target blocks according to the correlation sorting, and use multiple pages corresponding to the multiple target blocks as multiple target pages.

[0035] Optionally, the answer generation module is specifically configured to:

[0036] A visual pointer network is introduced to frame and mark the multiple target blocks in red, thereby obtaining multiple target blocks that are framed and marked in red;

[0037] Based on the question text, the multiple target pages and the multiple target blocks framed and marked in red, an answer is generated and the target block position coordinates corresponding to the answer are output.

[0038] A third aspect of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the question-answering method based on multimodal retrieval enhancement as provided in any implementation of the first aspect.

[0039] A fourth aspect of the present application provides a processor for running a computer program, which, when running, executes the question-answering method based on multimodal retrieval enhancement provided in any implementation of the first aspect.

[0040] Compared with the existing technology, this application has the following beneficial effects:

[0041] The question-answering method based on multimodal retrieval enhancement provided in this application uses a dual-stream cross-modal attention searcher to determine multiple target blocks corresponding to the question text and multiple target pages corresponding to the multiple target blocks. The dual-stream cross-modal attention searcher achieves block-level matching, guiding the answer generation process to focus on key information and improving the accuracy of the answer. Answer information is generated based on the question text, the multiple target blocks, and the multiple target pages. The generated answer information includes not only the answer content but also the location coordinates of the answer, which facilitates user verification and enhances the interpretability of the answer. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1A flowchart of a question-answering method based on multimodal retrieval enhancement provided in an embodiment of the present application;

[0044] Figure 2 A flowchart of another question-answering method based on multimodal retrieval enhancement provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of the structure of a question-answering device based on multimodal retrieval enhancement provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] As described above, current multimodal RAG approaches have fundamental limitations in understanding and utilizing the deep and structured information of documents: (1) semantic and spatial fragmentation; (2) shallow associations in retrieval mechanisms; and (3) information distortion and untraceability in the generation process. These limitations result in inaccurate and untraceable responses generated by language models.

[0047] In view of the above problems, the inventors have proposed a question-answering method, device, storage medium and processor based on multimodal retrieval enhancement after research, which obtains question text; determines multiple target blocks corresponding to the question text through a dual-stream cross-modal attention retriever, and determines multiple target pages corresponding to the multiple target blocks; generates answer information based on the question text, the multiple target blocks and the multiple target pages; the answer information includes answer content and answer location coordinates.

[0048] In order to help those skilled in the art better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0049] See also Figure 1 , which is a flow chart of a question-answering method based on multimodal retrieval enhancement provided by an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0050] S101. Obtain question text.

[0051] Questions asked by users in natural language format, for example, “What is the net profit in the third quarter of 2024?”

[0052] S102. Determine multiple target blocks corresponding to the question text and multiple target pages corresponding to the multiple target blocks through a dual-stream cross-modal attention retriever.

[0053] The dual-stream cross-modal attention retriever uses a dual-stream architecture, consisting of two parallel information processing streams: the query stream and the document stream. In the query stream, a text encoder is used to encode the question text and convert it into a query vector. In the document stream, a Transformer is used to encode the initial fusion position vectors of all blocks on each page, enhancing the semantic representation of each block and obtaining the target fusion position vector for each block. Using the cross-modal attention mechanism, the correlation between the target fusion position vector of each block and the query vector is calculated. Multiple blocks that meet the preset conditions are selected as multiple target blocks based on the correlation sorting, and the multiple pages corresponding to the multiple target blocks are selected as multiple target pages.

[0054] Through the dual-stream cross-modal attention retriever, it is possible to effectively find the specific block that is most likely to contain the answer from all blocks, achieve fine-grained matching from questions to document content, and improve the accuracy of answers.

[0055] S103: Generate answer information according to the question text, the multiple target blocks, and the multiple target pages.

[0056] The answer information includes the answer content and the answer location coordinates.

[0057] The question text, multiple target blocks, and multiple target pages are fed into the multimodal large model. The large multimodal model performs focused analysis based on the target blocks provided by the dual-stream cross-modal attention retriever. This allows for a more accurate understanding of the content of target blocks on the page, such as table cells, chart titles, and paragraphs, improving the accuracy of the answers. When the large multimodal model outputs the answer, it also returns the specific location of the answer, facilitating user verification and enhancing the interpretability and credibility of the answer.

[0058] The question-answering method based on multimodal retrieval enhancement provided in the embodiments of the present application uses a dual-stream cross-modal attention searcher to determine multiple target blocks corresponding to the question text and multiple target pages corresponding to the multiple target blocks. The dual-stream cross-modal attention searcher achieves block-level matching, guiding the answer generation process to focus on key information and improving the accuracy of the answer. Answer information is generated based on the question text, the multiple target blocks, and the multiple target pages. The generated answer information includes not only the answer content but also the location coordinates of the answer, facilitating user verification and enhancing the interpretability of the answer.

[0059] On the basis of the above embodiments, in order to further improve a question-answering method based on multimodal retrieval enhancement, the step of determining multiple target blocks corresponding to the question text and determining multiple target pages corresponding to the multiple target blocks through a dual-stream cross-modal attention retriever is refined.

[0060] See also Figure 2 , which is a flowchart of another question-answering method based on multimodal retrieval enhancement provided by an embodiment of the present application. Figure 2 As shown, the method includes the following steps:

[0061] S201. Obtain question text.

[0062] Questions asked by users in natural language format, for example, “What is the net profit in the third quarter of 2024?”

[0063] S202. Utilize the text encoder in the dual-stream cross-modal attention retriever to convert the question text into a query vector.

[0064] There is no restriction on the selection of text encoder.

[0065] S203. Using the Transformer in the dual-stream cross-modal attention retriever, encode the initial fusion position vectors of multiple blocks in each page in the vector database to obtain the target fusion position vectors of the multiple blocks.

[0066] The target fusion position vector of the first block is the initial fusion position vector after enhancing the semantic representation of the first block; the first block is any block among the multiple blocks.

[0067] The vector database includes vector representations of multiple pages; the vector representation of each page includes initial fusion position vectors of multiple blocks.

[0068] As a powerful sequence model, the Transformer can process block vectors and capture the dependencies between blocks through a self-attention mechanism, thereby generating a richer page-level representation. This not only considers the information of a single block, but also its relationship with other blocks on the page. The page vector representation encoded by the Transformer contains more comprehensive and in-depth semantic information.

[0069] In one feasible implementation, a method for obtaining a fusion position vector of a block includes:

[0070] A layout analysis model is used to divide a page into multiple blocks to obtain category information and position coordinate information of each block. Based on the category information and position coordinate information of each block, a category vector and position coordinate vector of each block are generated.

[0071] Extract the text content of each block and generate a content semantic vector for each block.

[0072] The content semantic vector, category vector and position coordinate vector of each block are combined to obtain the initial fused position vector of each block.

[0073] Collect documents in formats such as PDF and Word, render each page as a page image, and input the page image into a pre-trained layout analysis model, such as LayoutLMv3 or YOLO. The layout analysis model logically segments the page image to obtain the category and location coordinates of each block. The block categories include text, title, table, or image.

[0074] The text content of each block is extracted through micro-OCR or directly from the source file, and a text encoder such as BERT is used to generate a content semantic vector for the block.

[0075] The content semantic vector of the block, the type code of the block and the normalized position coordinate vector are concatenated or weighted summed to form the final fused position code of the block.

[0076] The text content, category features and location information within the block are integrated into a unified vector representation.

[0077] S204. Utilize the cross-modal attention mechanism in the dual-stream cross-modal attention retriever to calculate the correlation between the target fusion position vectors of the multiple blocks and the query vector, select multiple blocks that meet preset conditions as multiple target blocks according to the correlation sorting, and use multiple pages corresponding to the multiple target blocks as multiple target pages.

[0078] The preset conditions are used to filter the target blocks, and the pages that meet the preset conditions refer to the K blocks that are most relevant to the question.

[0079] In one possible implementation:

[0080] All blocks are sorted in descending order according to their relevance scores, and the K blocks with the highest scores are selected as target blocks.

[0081] The page information is extracted from the selected target block, and the page corresponding to the target block is used as the target page.

[0082] In one feasible implementation, the optimization method of the dual-stream cross-modal attention retriever includes:

[0083] The correlation calculation process between the target fusion position vector of each block and the query vector is optimized through the region alignment loss function.

[0084] By comparing the loss function, the correlation calculation process between the query vector and the vector representation of the page in the vector database is optimized.

[0085] Based on the contrast loss function and the region alignment loss function, the dual-stream cross-modal attention retriever is optimized by adopting a multi-task joint optimization method.

[0086] Region alignment loss is used to construct a supervisory signal based on the coordinate information of manually annotated question and correct answer blocks. This optimizes the cross-modal attention weight map, resulting in a higher response to the block containing the correct answer. This achieves a precise mapping from question to specific document blocks, meeting the requirements of high-precision retrieval. Furthermore, the attention weight map driven by region alignment loss has a clear physical meaning and can be used to generate a relevance heat map for each block in the document. This supports front-end visualization, such as highlighting the answer region in a PDF or image document, enhancing the interpretability and visualization capabilities of question and answer.

[0087] There is no limitation on the region alignment loss function, such as Dice Loss or Focal Loss.

[0088] Contrastive loss is used to supervise the overall semantic matching between questions and pages, ensuring that the matching score of relevant pages is higher than that of irrelevant pages. This enables the model to effectively identify pages related to questions and filter out irrelevant pages, significantly improving the retrieval accuracy and recall capabilities in large-scale document repositories.

[0089] The region alignment loss ensures block-level accuracy, and the contrast loss ensures page-level matching. Through the joint optimization of these two losses, the retrieval accuracy and recall ability of question and answer are improved, the fine-grained positioning from question to answer block is achieved, and the interpretability and visualization output capabilities of the answer are enhanced.

[0090] This step calculates the relevance between each block and the question, accurately locating the most relevant content area within the document, significantly improving the accuracy of answer matching. The search results not only include the final target page but also the specific block containing the answer, providing highly interpretable output and a reliable basis for subsequent answer generation.

[0091] S205: Generate answer information according to the question text, the multiple target blocks, and the multiple target pages.

[0092] The answer information includes the answer content and the answer location coordinates.

[0093] In one possible implementation:

[0094] A visual pointer network is introduced to frame and mark the multiple target blocks in red, thereby obtaining multiple target blocks that are framed and marked in red;

[0095] Based on the question text, the multiple target pages and the multiple target blocks framed and marked in red, an answer is generated and the target block position coordinates corresponding to the answer are output.

[0096] The introduction of a visual pointer network allows for the generation of answers while also outputting the coordinates of the image region of interest, making the answer generation process explainable and traceable. Furthermore, the front-end display component uses the returned block location coordinates to draw a red border on the image, visually indicating the answer location, reducing user search time and improving user-friendliness.

[0097] Another question-answering method based on multimodal retrieval enhancement provided by an embodiment of the present application uses a dual-stream cross-modal attention searcher to determine multiple target blocks corresponding to the question text and multiple target pages corresponding to the multiple target blocks. The dual-stream cross-modal attention searcher achieves block-level matching, guiding the answer generation process to focus on key information and improving the accuracy of the answer. Answer information is generated based on the question text, the multiple target blocks, and the multiple target pages. The generated answer information includes not only the answer content but also the location coordinates of the answer, facilitating user verification and enhancing the interpretability of the answer.

[0098] Based on the question-answering method based on multimodal retrieval enhancement introduced in the previous embodiment, the present application also provides a question-answering device based on multimodal retrieval enhancement. Figure 3 Figure 2 is a schematic diagram of the structure of the device. Figure 3 As shown, the question-answering device based on multimodal retrieval enhancement includes:

[0099] The question text acquisition module 301 is used to acquire the question text.

[0100] The retrieval module 302 is used to determine multiple target blocks corresponding to the question text and multiple target pages corresponding to the multiple target blocks through a dual-stream cross-modal attention retriever.

[0101] The answer generation module 303 is configured to generate answer information according to the question text, the multiple target blocks, and the multiple target pages; the answer information includes answer content and answer location coordinates.

[0102] Optionally, the retrieval module is specifically configured to:

[0103] The dual-stream cross-modal attention retriever includes a text encoder, a Transformer and a cross-modal attention mechanism;

[0104] The text encoder is used to convert the question text into a query vector;

[0105] The Transformer is configured to encode initial fusion position vectors of multiple blocks in each page of a vector database to obtain target fusion position vectors of the multiple blocks; the target fusion position vector of a first block is the initial fusion position vector after enhancing the semantic representation of the first block; the first block is any block among the multiple blocks;

[0106] The cross-modal attention mechanism is used to calculate the correlation between the target fusion position vectors of the multiple blocks and the query vector using the cross-modal attention mechanism, select multiple blocks that meet preset conditions as multiple target blocks according to the correlation sorting, and use multiple pages corresponding to the multiple target blocks as multiple target pages.

[0107] Optionally, the answer generation module is specifically configured to:

[0108] A visual pointer network is introduced to frame and mark the multiple target blocks in red, thereby obtaining multiple target blocks that are framed and marked in red;

[0109] Based on the question text, the multiple target pages and the multiple target blocks framed and marked in red, an answer is generated and the target block position coordinates corresponding to the answer are output.

[0110] Optionally, the method for obtaining the fusion position coding sequence of the block includes:

[0111] Using a layout analysis model to divide a page into multiple blocks, obtaining category information and position coordinate information of each block, and generating a category vector and a position coordinate vector for each block based on the category information and position coordinate information of each block;

[0112] Extract the text content of each block and generate a content semantic vector for each block;

[0113] The content semantic vector, category vector and position coordinate vector of each block are combined to obtain the initial fused position vector of each block.

[0114] Optionally, the optimization method of the dual-stream cross-modal attention retriever includes:

[0115] Optimizing the correlation calculation process between the target fusion position vector of each block and the query vector through the region alignment loss function;

[0116] Optimizing the correlation calculation process between the query vector and the vector representation of the page in the vector database by comparing the loss function; the vector representation of the page is composed of the target fusion position vectors of multiple blocks;

[0117] Based on the contrast loss function and the region alignment loss function, the dual-stream cross-modal attention retriever is optimized by adopting a multi-task joint optimization method.

[0118] In addition, an embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, the question-answering method based on multimodal retrieval enhancement as described in any of the method embodiments is implemented.

[0119] In addition, an embodiment of the present application also provides a processor for running a computer program, which, when running, executes the question-answering method based on multimodal retrieval enhancement as described in any implementation of the aforementioned method embodiment.

[0120] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components indicated as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0121] The above is merely one specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A question answering method based on multimodal retrieval enhancement, characterized in that: include: Get the question text; Determining, through a dual-stream cross-modal attention retriever, multiple target blocks corresponding to the question text, and determining multiple target pages corresponding to the multiple target blocks; generating answer information according to the question text, the plurality of target blocks, and the plurality of target pages; The answer information includes answer content and answer location coordinates.

2. The method according to claim 1, characterized in that The method of determining multiple target blocks corresponding to the question text and multiple target pages corresponding to the multiple target blocks by using a dual-stream cross-modal attention retriever includes: The dual-stream cross-modal attention retriever includes a text encoder, a Transformer and a cross-modal attention mechanism; Using the text encoder, convert the question text into a query vector; Encoding initial fusion position vectors of multiple blocks in each page of a vector database using the Transformer to obtain target fusion position vectors of the multiple blocks; the target fusion position vector of a first block is the initial fusion position vector after enhancing the semantic representation of the first block; the first block is any block among the multiple blocks; The cross-modal attention mechanism is used to calculate the correlation between the target fusion position vectors of the multiple blocks and the query vector, and multiple blocks that meet the preset conditions are selected as multiple target blocks according to the correlation sorting, and multiple pages corresponding to the multiple target blocks are used as multiple target pages.

3. The method according to claim 2, characterized in that The method for obtaining the initial fusion position vector of the block includes: Using a layout analysis model to divide a page into multiple blocks, obtaining category information and position coordinate information of each block, and generating a category vector and a position coordinate vector for each block based on the category information and position coordinate information of each block; Extract the text content of each block and generate a content semantic vector for each block; The content semantic vector, category vector and position coordinate vector of each block are combined to obtain the initial fused position vector of each block.

4. The method according to claim 1, wherein Generating answer information according to the question text, the multiple target blocks, and the multiple target pages includes: A visual pointer network is introduced to frame and mark the multiple target blocks in red, thereby obtaining multiple target blocks that are framed and marked in red; Based on the question text, the multiple target pages and the multiple target blocks framed and marked in red, an answer is generated and the target block position coordinates corresponding to the answer are output.

5. The method according to claim 2, characterized in that The optimization method of the dual-stream cross-modal attention retriever includes: Optimizing the correlation calculation process between the target fusion position vector of each block and the query vector through the region alignment loss function; Optimizing the correlation calculation process between the query vector and the vector representation of the page in the vector database by comparing the loss function; the vector representation of the page is composed of the target fusion position vectors of multiple blocks; Based on the contrast loss function and the region alignment loss function, the dual-stream cross-modal attention retriever is optimized by adopting a multi-task joint optimization method.

6. A question-answering device based on multimodal retrieval enhancement, characterized in that: include: Question text acquisition module, used to obtain question text; A retrieval module, configured to determine, through a dual-stream cross-modal attention retriever, a plurality of target blocks corresponding to the question text, and a plurality of target pages corresponding to the plurality of target blocks; an answer generation module, configured to generate answer information according to the question text, the plurality of target blocks, and the plurality of target pages; The answer information includes answer content and answer location coordinates.

7. The device according to claim 6, characterized in that The retrieval module is specifically used for: The dual-stream cross-modal attention retriever includes a text encoder, a Transformer and a cross-modal attention mechanism; The text encoder is used to convert the question text into a query vector; The Transformer is used to encode the initial fusion position vectors of multiple blocks in each page of the vector database to obtain the target fusion position vectors of the multiple blocks; first The target fusion position vector of the block is the initial fusion position vector after enhancing the semantic representation of the first block; The first block is any block among the multiple blocks; The cross-modal attention mechanism is used to calculate the correlation between the target fusion position vectors of the multiple blocks and the query vector using the cross-modal attention mechanism, select multiple blocks that meet preset conditions as multiple target blocks according to the correlation sorting, and use multiple pages corresponding to the multiple target blocks as multiple target pages.

8. The device according to claim 6, characterized in that The answer generation module is specifically used for: A visual pointer network is introduced to frame and mark the multiple target blocks in red, thereby obtaining multiple target blocks that are framed and marked in red; Based on the question text, the multiple target pages and the multiple target blocks framed and marked in red, an answer is generated and the target block position coordinates corresponding to the answer are output.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the program is executed by the processor, the question-answering method based on multimodal retrieval enhancement as described in any one of claims 1 to 6 is implemented.

10. A processor, characterized in that: Used to run a computer program, which, when running, executes the question-answering method based on multimodal retrieval enhancement according to any one of claims 1 to 6.

Images