Visual question answering method and apparatus based on multiple document images

Abstract

The embodiment of the specification provides a visual question answering method and device based on multiple document images, which is executed by using an LVLM, the LVLM comprising a multi-modal encoding model and an LLM. The method comprises: obtaining, by the multi-modal encoding model, respective image representations corresponding to the multiple document images and a text representation corresponding to a question text; inputting the respective image representations and the text representation into a text supervised scorer to obtain respective relevance scores of the multiple document images with respect to the question text; selecting, from the multiple document images, a plurality of document images belonging to a first category; the first category falls into a highest relevance score interval among multiple intervals divided according to the relevance scores; for any document image of the first category, inputting the document image into a resolution enhancer to obtain an image representation of a document image with a resolution improved; and inputting the respective image representations and the text representation into the LLM to obtain a reply text with respect to the multiple document images and the question text.

Description

TECHNICAL FIELD

[0001] One or more embodiments of the present specification relate to the field of computers, and in particular, to a method and apparatus for visual question answering based on multiple document images. BACKGROUND

[0002] A document image refers to an image rich in more textual content. For example, medical examination reports, financial statements, contracts, academic papers, etc. It can be understood that the document image can only contain textual content, or can contain both textual content and graphical content. Visual question answering (VQA) is a task of generating an answer according to an image and a related question.

[0003] In the visual question answering task based on multiple document images, it has its particularity. On the one hand, the information density in the document image is much higher than that in the natural image. Natural images usually contain simple scenes, objects or actions, while document images contain a large amount of text, symbols, charts, etc. On the other hand, document images are usually more in number than natural images. For example, a physical examination report, a financial statement or a contract document can contain 20 pages or even more content, each of which has independent visual text information.

[0004] For the above reasons, in the visual question answering task based on multiple document images, the generated answer is often inaccurate or the processing efficiency is low. SUMMARY

[0005] One or more embodiments of the present specification describe a method and apparatus for visual question answering based on multiple document images, which can improve the accuracy of generating answers and has high processing efficiency.

[0006] In a first aspect, a method for visual question answering based on multiple document images is provided, which is executed by a large vision language model (LVLM), the LVLM comprising a multi-modal encoding model and a large language model (LLM), and the method comprising:

[0007] obtaining, by the multi-modal encoding model, respective image representations corresponding to the multiple document images and a text representation corresponding to the question text;

[0008] inputting the respective image representations and the text representation into a text supervised scorer to obtain respective relevance scores of the multiple document images with respect to the question text;

[0009] selecting, from the multiple document images, a number of document images belonging to a first category; the first category falls into a highest relevance score interval among a plurality of intervals divided according to the relevance scores.

[0010] For any first category of document image, input the resolution enhancer to obtain an image representation of the document image with enhanced resolution;

[0011] Input each image representation and text representation into the LLM to obtain a reply text for the plurality of document images and the question text.

[0012] In one embodiment, the multi-modal encoding model comprises a visual encoder and a text encoder, and the obtaining, by the multi-modal encoding model, each image representation corresponding to each of the plurality of document images and a text representation corresponding to the question text comprises:

[0013] Inputting each of the plurality of document images into the visual encoder to obtain each image representation corresponding to each of the plurality of document images;

[0014] Inputting the question text into the text encoder to obtain the text representation corresponding to the question text.

[0015] In one embodiment, the image representation comprises a block feature vector corresponding to each image block of the document image, and the text representation comprises a word feature vector corresponding to each wordpiece of the question text; and the inputting, into the text supervision scorer, each image representation and text representation to obtain each relevance score of the plurality of document images to the question text comprises:

[0016] Determining an attention score between any image block and any wordpiece according to each block feature vector and each word feature vector;

[0017] Performing first pooling processing on each attention score to obtain a first pooled score between any image block and the question text;

[0018] Performing second pooling processing on the first pooled score between each image block and the question text to obtain a second pooled score between any document image and the question text;

[0019] Performing normalization processing on the second pooled score between each document image and the question text to obtain each relevance score of the plurality of document images to the question text.

[0020] Further, the determining an attention score between any image block and any wordpiece according to each block feature vector and each word feature vector comprises:

[0021] Inputting each block feature vector and each word feature vector into a first attention layer of the LLM to obtain the attention score between any image block and any wordpiece.

[0022] In one embodiment, the resolution enhancer comprises a visual encoder; for any document image of the first category, inputting the document image into the resolution enhancer to obtain an image representation of the document image with enhanced resolution, comprising:

[0023] splitting the document image into a plurality of sub-images;

[0024] inputting any sub-image into the visual encoder to obtain a sub-image representation corresponding to the sub-image;

[0025] splicing the sub-image representation corresponding to each sub-image and the image representation of the document image to obtain the image representation of the document image with enhanced resolution.

[0026] In one embodiment, the method further comprises:

[0027] determining a document image with a relevance score falling into a first interval range as a document image of the second category; the first interval range is the lowest interval of a plurality of intervals divided according to the relevance score;

[0028] for any document image of the second category, inputting the document image into the information compressor to obtain an image representation of the document image with compressed image size.

[0029] Further, the image representation comprises a block feature vector corresponding to each image block of the document image; and for any document image of the second category, inputting the document image into the information compressor to obtain an image representation of the document image with compressed image size, comprising:

[0030] for any document image of the second category, inputting the document image into the information compressor to determine whether the relevance score of the document image falls into a first sub-interval or a second sub-interval; the first sub-interval and the second sub-interval constitute the first interval range, and any value in the first sub-interval is smaller than any value in the second sub-interval;

[0031] if the information compressor determines that the relevance score of the document image falls into the first sub-interval, discarding all block feature vectors of the document image;

[0032] if the information compressor determines that the relevance score of the document image falls into the second sub-interval, discarding part of the block feature vectors of the document image.

[0033] Further, the method further comprises:

[0034] determining a plurality of document images other than the first category and the second category as document images of the third category;

[0035] for any document image of the third category, keeping the image representation of the document image unchanged.

[0036] In a second aspect, a visual question answering device based on a plurality of document images is provided, which is executed by a large visual language model (LVLM), the LVLM comprising a multi-modal encoding model and a large language model (LLM), and the device comprising:

[0037] an encoding unit configured to obtain, by the multi-modal encoding model, respective image representations corresponding to the plurality of document images and a text representation corresponding to the question text;

[0038] a scoring unit configured to input the respective image representations and the text representation obtained by the encoding unit into a text supervised scorer to obtain respective relevance scores of the plurality of document images with respect to the question text;

[0039] a classification unit configured to select, from the plurality of document images, a number of document images belonging to a first category, the first category falling into a highest relevance score interval among a plurality of intervals divided according to the relevance scores obtained by the scoring unit;

[0040] an enhancement unit configured to, for any document image of the first category obtained by the classification unit, input the document image into a resolution enhancer to obtain an image representation of the document image with enhanced resolution;

[0041] a reply unit configured to input the respective image representations and the text representation obtained by the enhancement unit into the LLM to obtain reply texts with respect to the plurality of document images and the question text.

[0042] In a third aspect, a computer-readable storage medium is provided, which stores a computer program, and when the computer program is executed in a computer, the computer is caused to execute the method of the first aspect.

[0043] In a fourth aspect, a computing device is provided, which comprises a memory and a processor, the memory storing executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0044] The method and device provided by the embodiments of the present specification first obtain, through a multi-modal encoding model, respective image representations corresponding to a plurality of document images and a text representation corresponding to a question text; then input the respective image representations and the text representation into a text supervised scorer to obtain respective relevance scores of the plurality of document images with respect to the question text; then select a plurality of document images belonging to a first category from the plurality of document images; the first category falls into a highest relevance score interval among a plurality of intervals divided according to the relevance scores; then, for any document image of the first category, input the document image into a resolution enhancer to obtain an image representation of the document image with enhanced resolution; finally, input the respective image representations and the text representation into an LLM to obtain a reply text with respect to the plurality of document images and the question text. As can be seen from the above, the embodiments of the present specification obtain respective relevance scores of a plurality of document images with respect to a question text through a text supervised scorer, then select a key image according to the respective relevance scores, and enhance the resolution of the key image through a resolution enhancer, thereby improving the model effect and selectively enhancing the resolution of the image, which can efficiently utilize the computing resources. The scheme can improve the accuracy of the generated answer and has high processing efficiency.

[0045] In addition, the text supervised scorer and the resolution enhancer are pluggable components and can be adapted to different LVLMs, without the need to modify the underlying model architecture of the LVLM to improve performance and efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0046] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0047] Figure 1 The schematic diagram of the implementation scenario of an embodiment disclosed in the present specification;

[0048] Figure 2 The flowchart of a visual question answering method based on a plurality of document images according to an embodiment is shown;

[0049] Figure 3 The schematic diagram of an encoding process based on a multi-modal encoding model according to an embodiment is shown;

[0050] Figure 4 The schematic diagram of a processing process based on a text supervised scorer according to an embodiment is shown;

[0051] Figure 5 The schematic diagram of a processing process based on a resolution enhancer according to an embodiment is shown;

[0052] Figure 6 Fig. 11 shows a schematic diagram of a processing procedure based on information compressor according to one embodiment;

[0053] Figure 7 Fig. 12 shows a schematic diagram of input and output of LLM according to one embodiment;

[0054] Figure 8 Fig. 13 shows a schematic block diagram of a visual question answering apparatus based on multiple document images according to one embodiment. DETAILED DESCRIPTION

[0055] The schemes provided in the present specification are described below with reference to the accompanying drawings.

[0056] Figure 1 Fig. 1 shows a schematic diagram of an implementation scenario of one embodiment disclosed in the present specification. The implementation scenario relates to visual question answering based on multiple document images. It can be understood that in the visual question answering task, the question text and the multiple document images are taken as input, and the answer text is taken as output. Both the question and the answer are in the form of text. On the one hand, the document images usually contain a large amount of text, symbols, charts, etc., and have extremely high information compression characteristics. Such dense information arrangement puts higher requirements on the understanding ability of the model, because low resolution may cause text blur and detail loss, thereby affecting the parsing accuracy. Therefore, high-resolution images are usually used to ensure clear text and complete structure. On the other hand, the document images are not only information-intensive, but also usually more challenging in quantity than natural images. Each page of the document images has independent visual text information, and there may be semantic association or cross-page dependency between these information, which needs to efficiently integrate multiple pages of information and perform targeted marking and processing. Single processing of each page of the document images may lead to loss of overall context, while large-scale processing may face the bottleneck of computing resources and efficiency.

[0057] REFERENCE Figure 1In the embodiments of the present specification, by determining the respective relevance scores of the respective document images to the question text, the resolution is dynamically adjusted, so as to adaptively enhance the resolution of the key images and efficiently utilize the computing resources. It can be understood that the key images are the document images with higher relevance scores, for example, the document image A, the document image B and the document image C, whose relevance scores are 0.14, 0.26 and 0.13 respectively, are higher than those of other document images, and the image size thereof can be increased to improve the resolution thereof. For other document images, a plurality of document images with the lowest relevance scores can be further screened out, and the image size thereof can be reduced to compress the unimportant images, for example, the document image E and the document image F, whose relevance scores are 0.10 and 0.01 respectively, and for the document image E, part of the image blocks thereof are discarded to reduce the resolution thereof, and for the document image F, all the image blocks thereof are discarded. For a plurality of document images with the relevance scores in the middle position, no processing is performed, and the standard resolution thereof is maintained, for example, the document image D, whose relevance score is 0.12, maintains the original size thereof.

[0058] It should be noted that Figure 1 For a typical implementation scenario, three processing conditions are included, for a plurality of document images, wherein a plurality of images have enhanced resolution, a plurality of images have reduced resolution, and a plurality of images maintain the standard resolution. In actual application, according to the relevance scores of the plurality of document images, only two of the three processing conditions can be included, for example, for a plurality of document images, wherein a plurality of images have enhanced resolution, and the other images have reduced resolution; or for a plurality of document images, wherein a plurality of images have enhanced resolution, and the other images maintain the standard resolution.

[0059] Figure 2 A flowchart of a visual question answering method based on a plurality of document images according to an embodiment is shown, which is executed by an LVLM, the LVLM includes a multi-modal encoding model and a large language model (LLM), and the method can be based on Figure 1 The implementation scenario shown in FIG. 1. As Figure 2As shown, the visual question answering method based on multiple document images in this embodiment includes the following steps: step 21, obtaining respective image representations corresponding to the multiple document images and a text representation corresponding to the question text through a multi-modal encoding model; step 22, inputting the respective image representations and the text representation into a text supervised scorer to obtain respective relevance scores of the multiple document images to the question text; step 23, selecting a plurality of document images belonging to a first category from the multiple document images; the first category falls into a highest relevance score interval among a plurality of intervals divided according to the relevance scores; step 24, for any document image of the first category, inputting the document image into a resolution enhancer to obtain an image representation of the document image with enhanced resolution; and step 25, inputting the respective image representations and the text representation into an LLM to obtain answer texts for the multiple document images and the question text. The specific execution modes of the above steps are described below.

[0060] Firstly, in step 21, respective image representations corresponding to the multiple document images and a text representation corresponding to the question text are obtained through a multi-modal encoding model. It can be understood that the multi-modal encoding model is a component of the LVLM.

[0061] The LVLM is a model combining natural language processing and computer vision technology, and can understand and generate text related to images.

[0062] In one example, the multi-modal encoding model includes a visual encoder and a text encoder, and the obtaining of the respective image representations corresponding to the multiple document images and the text representation corresponding to the question text through the multi-modal encoding model includes:

[0063] The multiple document images are respectively input into the visual encoder to obtain the respective image representations corresponding to the multiple document images;

[0064] The question text is input into the text encoder to obtain the text representation corresponding to the question text.

[0065] In this example, a vision transformer (VIT) can be reused to obtain the respective image representations corresponding to the respective document images, and the image representations can be in the form of vectors. The text encoder can specifically use a tokenizer to divide the text into a plurality of tokens, and then map each token to a corresponding vector space. In the field of natural language processing, a token is the smallest semantic unit in a text, which can be a word, a subword, a character, etc.

[0066] Figure 3 An encoding process schematic diagram based on a multi-modal encoding model according to one embodiment is shown. Referring to Figure 3The multi-modal encoding model includes a VIT and a text encoder. The multiple document images are document image A, document image B, document image C, document image D, document image E, and document image F in sequence. The document image A is input into the VIT to obtain an image representation of the document image A. The document image B is input into the VIT to obtain an image representation of the document image B. The document image C is input into the VIT to obtain an image representation of the document image C. The document image D is input into the VIT to obtain an image representation of the document image D. The document image E is input into the VIT to obtain an image representation of the document image E. The document image F is input into the VIT to obtain an image representation of the document image F. The question text is “What is the name of …?” and is input into the text encoder to obtain a text representation corresponding to the question text.

[0067] Then, in step 22, the image representation and the text representation are input into a text supervised scorer to obtain respective relevance scores of the multiple document images to the question text. It can be understood that the relevance score reflects the contribution degree of different document images to the question text, that is, the document image with a higher relevance score is more helpful to answer the question described in the question text.

[0068] In one example, the image representation includes a block feature vector corresponding to each image block of the document image, and the text representation includes a word feature vector corresponding to each word of the question text. The inputting of the image representation and the text representation into the text supervised scorer to obtain respective relevance scores of the multiple document images to the question text includes:

[0069] determining an attention score between any image block and any word according to the block feature vector and the word feature vector;

[0070] performing first pooling processing on the attention score to obtain a first pooling score between any image block and the question text;

[0071] performing second pooling processing on the first pooling score between each image block and the question text to obtain a second pooling score between any document image and the question text;

[0072] performing normalization processing on the second pooling score between each document image and the question text to obtain the respective relevance scores of the multiple document images to the question text.

[0073] In this example, the attention score between any image block and any word is first determined, and then the attention score is aggregated to the image level through the pooling processing, and the relevance score between 0 and 1 is obtained through the normalization processing, so as to facilitate the differentiation of the contribution degree of different document images to the question text.

[0074] Figure 4A processing process schematic diagram of a text supervision scorer according to one embodiment is shown. Refer to Figure 4 The input of the text supervision scorer is the image representation corresponding to the M document images respectively and the text representation of the question text, and the output is M relevance scores of the M document images to the question text. The image representation of the kth document image is denoted as V k , where 1≤k≤M, the text representation of the question text is denoted as T, the image representation includes the block feature vector corresponding to each image block of the document image, and the text representation includes the word feature vector corresponding to each word of the question text. The block feature vector and the word feature vector are respectively represented by a small square in the figure. The attention score between each image block and each word forms a matrix composed of small squares, after the first pooling processing, the dimension of the matrix is reduced in the dimension corresponding to the text, and after the second pooling processing, the dimension is reduced in the dimension corresponding to the image. Normalization processing is performed again so that the relevance score belongs to a decimal between 0 and 1. In the figure, s k represents the relevance score of the kth document image to the question text, and 1≤k≤M.

[0075] Further, the determination of the attention score between any image block and any word according to the block feature vector and the word feature vector comprises:

[0076] The block feature vector and the word feature vector are input into the first attention layer of the LLM to obtain the attention score between any image block and any word.

[0077] In this example, the structure of the LLM is reused, and there is no need to specially train the text supervision scorer.

[0078] Then in step 23, a plurality of document images belonging to a first category are selected from the plurality of document images; the first category falls into a highest relevance score interval among a plurality of intervals divided according to the relevance score. It can be understood that the first category represents that the contribution degree of the document image to the question belongs to the highest level.

[0079] According to the actual needs of the scene, the plurality of document images can be divided into a preset number of categories in the embodiments of the present specification, each category falls into an interval among a plurality of intervals divided according to the relevance score, and represents a corresponding level of the contribution degree of the document image to the question, for example, divided into four categories, including a very important category, an important category, an unimportant category, and a very unimportant category. It can be understood that the contribution degrees of these categories are different.

[0080] In step 24, for any document image of the first category, input the document image into the resolution enhancer to obtain the image representation of the document image with enhanced resolution. It can be understood that, for multiple document images, only the document images belonging to the first category are enhanced in resolution.

[0081] In one example, the resolution enhancer comprises a visual encoder. The inputting, for any document image of the first category, the document image into the resolution enhancer to obtain the image representation of the document image with enhanced resolution comprises:

[0082] Splitting the document image into multiple sub-images;

[0083] Inputting any sub-image into the visual encoder to obtain a sub-image representation corresponding to the sub-image;

[0084] Concatenating the sub-image representations corresponding to the sub-images and the image representation of the document image to obtain the image representation of the document image with enhanced resolution.

[0085] In this example, the resolution is enhanced by adding labels to the image.

[0086] Figure 5 A schematic diagram of a processing procedure based on a resolution enhancer according to one embodiment is shown. Referring to Figure 5 After obtaining the relevance scores of the multiple document images with respect to the query text, the document images corresponding to the top-ranked relevance scores can be selected. This can be implemented based on a topk(·) function. For example, the document images corresponding to the top two relevance scores, document image B and document image H, can be selected. Next, the selected document images are split into multiple sub-images. This can be implemented based on a Crop(·) function. For example, document image B is split into sub-image B1 and sub-image B2, and document image H is split into sub-image H1 and sub-image H2. Then, any sub-image is input into a visual encoder VIT to obtain a sub-image representation corresponding to the sub-image. For example, the sub-image representation of sub-image B1, the sub-image representation of sub-image B2, the sub-image representation of sub-image H1, and the sub-image representation of sub-image H2 are obtained. Finally, the sub-image representations corresponding to the sub-images and the image representation of the document image are concatenated to obtain the image representation of the document image with enhanced resolution. The concatenation can be implemented based on a Cat(·) function. For example, the sub-image representation of sub-image B1, the sub-image representation of sub-image B2, and the image representation of document image B are concatenated to obtain the image representation of document image B with enhanced resolution. In this example, the sub-image representation of sub-image B1 and the sub-image representation of sub-image B2 are enhanced labels, and the image representation of document image B is an original label.

[0087] Finally, in step 25, the image representations and the text representation are input into the LLM to obtain the reply text for the plurality of document images and the question text. It can be understood that the image representations include the image representation of the document image with the resolution improved.

[0088] In one example, before step 25, the method further includes:

[0089] The document image with the relevance score falling into the first interval range is determined as the second category of document image; the first interval range is the lowest interval of the relevance score among the plurality of intervals divided according to the relevance score.

[0090] For any second category of document image, the image representation of the document image with the image scale compressed is obtained by inputting the document image into the information compressor.

[0091] In this example, for the plurality of document images, the second category of document images with the lowest contribution to answering the question can be further screened out, and the image scale of the document images is compressed, which helps to improve the processing efficiency of the subsequent LLM.

[0092] Further, the image representation includes the block feature vector corresponding to each image block of the document image; and the image representation of the document image with the image scale compressed is obtained by inputting the second category of document image into the information compressor, including:

[0093] For any second category of document image, the information compressor is input to determine whether the relevance score falls into the first sub-interval or the second sub-interval; the first sub-interval and the second sub-interval constitute the first interval range, and any value in the first sub-interval is less than any value in the second sub-interval.

[0094] If the information compressor determines that the relevance score falls into the first sub-interval, all block feature vectors of the document image are discarded.

[0095] If the information compressor determines that the relevance score falls into the second sub-interval, part of the block feature vectors of the document image are discarded.

[0096] In this example, the image scale compression is further refined into two kinds, i.e., discarding all block feature vectors of the document image and discarding part of the block feature vectors of the document image, so as to retain relatively important marks through differential processing and improve the accuracy of answering the question.

[0097] Figure 6 A schematic diagram of the processing process based on the information compressor according to one embodiment is shown. Referring to Figure 6After obtaining the relevance scores of the question text with respect to each of the plurality of document image pairs, the document images whose relevance scores fall within a first interval range can be selected as the document images of the second category, the first interval range being the lowest interval in the plurality of intervals in which the relevance scores are ranked from high to low, for example, the first interval range is [0, βγ], the first interval range can be further divided into a first sub-interval and a second sub-interval, for example, the first sub-interval is [0, αγ], and the second sub-interval is (αγ, βγ], where α, β, and γ are preset parameters, and their values can be set according to the scene requirements. For any document image of the second category, input the information compressor, and determine whether the relevance score falls within the first sub-interval or the second sub-interval. If the information compressor determines that the relevance score falls within the first sub-interval, discard all block feature vectors of the document image, for example, the relevance scores of images A and F fall within the first sub-interval, and all block feature vectors of the document image are discarded, that is, the image is discarded. If the information compressor determines that the relevance score falls within the second sub-interval, discard part of the block feature vectors of the document image, for example, the relevance scores of images C and E fall within the second sub-interval, and part of the block feature vectors of the document image are discarded, that is, the image block is discarded. In the figure, the block feature vectors of the discarded image blocks are marked with × for intuitive representation.

[0098] Further, the method further comprises:

[0099] The plurality of document images other than the first category and the second category are determined as document images of a third category.

[0100] For any document image of the third category, the image representation of the document image is kept unchanged.

[0101] In this example, the document images of the third category have a contribution degree between the first category and the second category, and the image representation of the document image can be kept unchanged, that is, the resolution is not changed, thereby retaining relatively important information.

[0102] Figure 7 An input-output schematic diagram of an LLM according to an embodiment is shown. Referring to Figure 7 , the input of the LLM includes two parts, image representation and text representation, wherein the image representation has been dynamically adjusted in resolution, the resolution of some document images has been increased, the resolution of some document images has been reduced, and the resolution of some document images has been kept unchanged, wherein the image representations of two document images are distinguished by a blank interval, each small square represents a block image representation, a discarded block image representation is represented by ×, and a small square without × and with a dashed line frame represents an increased block image representation.

[0103] The embodiments of the present specification can adopt a pluggable component to adapt to various large vision language models (LVLMs), improve flexibility and scalability. The pluggable component includes a textual influence evaluator (TIE) for evaluating the contribution of different images to a question. Further, it can also include a key image enhancer (KIE) for enhancing the resolution of important images, and a dynamic visual dropper (DVD) for compressing or discarding unimportant images. These pluggable components do not require additional training.

[0104] The embodiments of the present specification bypass the optical character recognition (OCR) tool for document images and directly perform multimodal fusion understanding, which can efficiently integrate multiple pages of information, has the ability to process dynamic length input, and improves robustness and generalization capability. Its application scenarios are wide, for example, in the medical health scenario, it is used to improve the accuracy and efficiency of report interpretation and physical examination report analysis; in the financial and contract management scenario, it is used to quickly and accurately process complex documents and improve work efficiency; in the education and scientific research scenario, it supports reading comprehension of academic papers and research reports.

[0105] Through the method provided by the embodiments of the present specification, first, the multimodal encoding model is used to obtain image representations corresponding to a plurality of document images and a text representation corresponding to a question text; then, the image representations and the text representation are input into a textual influence evaluator to obtain a plurality of relevance scores of the plurality of document images to the question text; then, a plurality of document images belonging to a first category are selected from the plurality of document images; the first category falls into a highest relevance score interval among a plurality of intervals divided according to the relevance scores; for any document image of the first category, the document image is input into a resolution enhancer to obtain an image representation of the document image with enhanced resolution; finally, the image representations and the text representation are input into an LLM to obtain a reply text to the plurality of document images and the question text. As can be seen from the above, the embodiments of the present specification obtain a plurality of relevance scores of a plurality of document images to a question text through a textual influence evaluator, and then select key images according to the relevance scores, and enhance the resolution of the key images through a resolution enhancer, thereby improving the model effect, and selectively enhancing the image resolution, which can efficiently utilize the computing resources. This scheme can improve the accuracy of generated answers and has high processing efficiency.

[0106] According to another aspect, embodiments also provide a visual question answering apparatus based on a plurality of document images, executed by a large visual language model (LVLM) comprising a multi-modal encoding model and a large language model (LLM), the apparatus being configured to perform the method according to the embodiments of the present specification. Figure 8 A schematic block diagram of a visual question answering apparatus based on a plurality of document images according to an embodiment is shown. As shown in Figure 8 The apparatus 800 comprises:

[0107] An encoding unit 81 configured to obtain, by the multi-modal encoding model, respective image representations of the plurality of document images and a text representation of the question text.

[0108] A scoring unit 82 configured to input the respective image representations and the text representation obtained by the encoding unit 81 into a text supervised scorer to obtain respective relevance scores of the plurality of document images to the question text.

[0109] A classification unit 83 configured to select, from the plurality of document images, a number of document images belonging to a first category; the first category falling into a highest relevance score interval among a plurality of intervals divided according to the relevance scores obtained by the scoring unit 82.

[0110] An enhancement unit 84 configured to, for any document image of the first category obtained by the classification unit 83, input the document image into a resolution enhancer to obtain an image representation of the document image with enhanced resolution.

[0111] A reply unit 85 configured to input the respective image representations and the text representation obtained by the enhancement unit 84 into the LLM to obtain a reply text to the plurality of document images and the question text.

[0112] Optionally, as an embodiment, the multi-modal encoding model comprises a visual encoder and a text encoder, and the encoding unit 81 comprises:

[0113] An image encoding sub-unit configured to input the plurality of document images into the visual encoder respectively to obtain the respective image representations of the plurality of document images.

[0114] A text encoding sub-unit configured to input the question text into the text encoder to obtain the text representation of the question text.

[0115] Optionally, as an embodiment, the image representation comprises a block feature vector corresponding to each image block of the document image, and the text representation comprises a word feature vector corresponding to each wordpiece of the question text; and the scoring unit 82 comprises:

[0116] an attention subunit configured to determine an attention score between any image block and any token according to the respective block feature vector and the respective token feature vector;

[0117] a first pooling subunit configured to perform first pooling processing on the respective attention score obtained by the attention subunit to obtain a first pooling score between any image block and the question text;

[0118] a second pooling subunit configured to perform second pooling processing on the first pooling score between any image block and the question text obtained by the first pooling subunit to obtain a second pooling score between any document image and the question text;

[0119] a normalization subunit configured to perform normalization processing on the second pooling score between any document image and the question text obtained by the second pooling subunit to obtain respective relevance scores of the plurality of document images to the question text.

[0120] Further, the attention subunit is specifically configured to input the respective block feature vector and the respective token feature vector into a first attention layer of the LLM to obtain the attention score between any image block and any token.

[0121] Optionally, as one embodiment, the resolution enhancer comprises a visual encoder; the enhancement unit 84 comprises:

[0122] a splitting subunit configured to split any document image of the first category into a plurality of sub-images;

[0123] an encoding subunit configured to input any sub-image obtained by the splitting subunit into the visual encoder to obtain a sub-graph representation corresponding to the sub-image;

[0124] a splicing subunit configured to splice the sub-graph representation corresponding to each sub-image obtained by the encoding subunit and the image representation of the document image to obtain the image representation of the document image after resolution enhancement.

[0125] Optionally, as one embodiment, the classification unit 83 is further configured to determine a document image with a relevance score falling within a first interval range as a document image of the second category; the first interval range is a relevance score lowest interval among a plurality of interval ranges divided according to the relevance score;

[0126] The apparatus further comprises a compression unit configured to input any document image of the second category obtained by the classification unit 83 into an information compressor to obtain an image representation of the document image after compression of the image scale.

[0127] Further, the image representation comprises respective block feature vectors corresponding to respective image blocks of the document image; and the compression unit comprises:

[0128] a judging subunit configured to, for any second category of document image, input the document image into an information compressor, and determine whether the relevance score of the document image falls into a first subinterval or a second subinterval according to the information compressor; the first subinterval and the second subinterval constitute the first interval range, and any value in the first subinterval is smaller than any value in the second subinterval;

[0129] a first compression subunit configured to, if the information compressor determines that the relevance score of the document image falls into the first subinterval, discard all block feature vectors of the document image;

[0130] a second compression subunit configured to, if the information compressor determines that the relevance score of the document image falls into the second subinterval, discard part of the block feature vectors of the document image.

[0131] Further, the classification unit 83 is further configured to determine a plurality of document images other than the first category and the second category as document images of a third category.

[0132] The device further comprises a maintaining unit configured to maintain the image representation of the document image unchanged for any document image of the third category obtained by the classification unit 83.

[0133] By the device provided by the embodiments of the present specification, first, the encoding unit 81 obtains the respective image representations corresponding to the plurality of document images and the text representation corresponding to the question text through the multi-modal encoding model; then, the scoring unit 82 inputs the respective image representations and the text representation into the text supervised scorer to obtain respective relevance scores of the plurality of document images to the question text; next, the classification unit 83 selects a plurality of document images belonging to the first category from the plurality of document images; the first category falls into the highest relevance score interval among the plurality of intervals divided according to the relevance scores; the enhancing unit 84 further inputs any document image of the first category into the resolution enhancer to obtain the image representation of the document image with enhanced resolution; finally, the reply unit 85 inputs the respective image representations and the text representation into the LLM to obtain the reply text to the plurality of document images and the question text. As can be seen from the above, the embodiments of the present specification obtain respective relevance scores of the plurality of document images to the question text through the text supervised scorer, and then select the key image according to the respective relevance scores, and enhance the resolution of the key image through the resolution enhancer, thereby improving the model effect, and selectively enhancing the image resolution, which can efficiently utilize the computing resources. The scheme can improve the accuracy of the generated answer and has high processing efficiency.

[0134] According to another aspect, embodiments also provide a computer-readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method described Figure 2 above.

[0135] According to yet another aspect, an embodiment also provides a computing device comprising a memory having executable code stored therein and a processor that, when executing the executable code, implements the method described in connection with Figure 2 the above.

[0136] Those skilled in the art should be aware that, in the above one or more examples, the functions described in the specification of the present application can be implemented in hardware, software, firmware or any combination thereof. When implemented in software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or code on a computer readable medium.

[0137] The above detailed description merely describes the specific implementation of the present application, and is not intended for limiting the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific implementation of the present application, and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present application shall be included in the protection scope of the present application.

Claims

1. A visual question answering method based on a plurality of document images, performed by using a large visual language model (LVLM), the LVLM comprising a multi-modal encoding model and a large language model (LLM), the method comprising: obtaining, by the multi-modal encoding model, respective image representations of the plurality of document images and a text representation of a question text; inputting the respective image representations and the text representation into a text supervised scorer to obtain respective relevance scores of the plurality of document images to the question text; selecting, from the plurality of document images, a plurality of document images belonging to a first category, the first category falling into a highest relevance score interval among a plurality of relevance score intervals divided according to the relevance scores; for any document image of the first category, inputting the document image into a resolution enhancer to obtain an image representation of the document image with enhanced resolution, so as to update the image representation of the document image; and inputting the respective image representations and the text representation into the LLM to obtain answer texts for the plurality of document images and the question text. The multi-modal encoding model comprises a visual encoder and a text encoder, and the obtaining, by the multi-modal encoding model, the respective image representations of the plurality of document images and the text representation of the question text comprises: inputting the plurality of document images into the visual encoder to obtain the respective image representations of the plurality of document images; and inputting the question text into the text encoder to obtain the text representation of the question text. The image representation comprises a block feature vector corresponding to each image block of the document image, and the text representation comprises a word feature vector corresponding to each word of the question text; and the inputting the respective image representations and the text representation into the text supervised scorer to obtain the respective relevance scores of the plurality of document images to the question text comprises: determining, according to the respective block feature vectors and the respective word feature vectors, an attention score between any image block and any word; performing first pooling processing on the respective attention scores to obtain a first pooling score between any image block and the question text; performing second pooling processing on the first pooling scores between the respective image blocks and the question text to obtain a second pooling score between any document image and the question text; and performing normalization processing on the second pooling scores between the respective document images and the question text to obtain the respective relevance scores of the plurality of document images to the question text. The determining, according to the respective block feature vectors and the respective word feature vectors, the attention score between any image block and any word comprises: inputting the respective block feature vectors and the respective word feature vectors into a first attention layer of the LLM to obtain the attention score between any image block and any word. The resolution enhancer comprises the visual encoder; and the inputting, for any document image of the first category, the document image into the resolution enhancer to obtain the image representation of the document image with enhanced resolution comprises: splitting the document image of the first category into a plurality of sub-images; inputting any sub-image into the visual encoder to obtain a sub-image representation corresponding to the sub-image. ​ 2. The method of claim 1, wherein, ​ ​ ​ 3. The method of claim 1, wherein, ​ ​ ​ ​ ​ 4. The method of claim 3, wherein, ​ ​ 5. The method of claim 1, wherein, ​ ​ ​ ​ The sub-image representations corresponding to the sub-images and the image representation of the document image are spliced to obtain an image representation of the document image with enhanced resolution.

6. The method of claim 1, wherein, The method further comprises: The document image with the relevance score falling into the first interval range is determined as a second category of document image; the first interval range is the lowest interval of the relevance score among the multiple intervals divided according to the relevance score; For any second category of document image, the image representation of the document image with compressed image size is obtained by inputting the document image into the information compressor, so as to update the image representation of the document image.

7. The method of claim 6, wherein, The image representation comprises block feature vectors corresponding to each image block of the document image respectively; The image representation of the document image with compressed image size is obtained by inputting the second category of document image into the information compressor, comprising: For any second category of document image, the information compressor is used to judge whether the relevance score falls into the first sub-interval or the second sub-interval; The first sub-interval and the second sub-interval constitute the first interval range, and any value in the first sub-interval is smaller than any value in the second sub-interval; If the information compressor judges that the relevance score falls into the first sub-interval, all block feature vectors of the document image are discarded; If the information compressor judges that the relevance score falls into the second sub-interval, part of the block feature vectors of the document image are discarded.

8. The method of claim 6, wherein, The method further comprises: A plurality of document images other than the first category and the second category are determined as a third category of document image; For any third category of document image, the image representation of the document image is kept unchanged.

9. An apparatus for visual question answering based on a plurality of document images, executed by a large visual language model (LVLM), the LVLM comprising a multi-modal encoding model and a large language model (LLM), the apparatus comprising: An encoding unit configured to obtain, by the multi-modal encoding model, image representations corresponding to the plurality of document images respectively and a text representation corresponding to a question text; A scoring unit configured to input the image representations and the text representation obtained by the encoding unit into a text supervised scorer to obtain respective relevance scores of the plurality of document images to the question text; A classification unit configured to select a plurality of document images belonging to a first category from the plurality of document images, the first category falling into a highest interval of the relevance scores among a plurality of intervals divided according to the relevance scores obtained by the scoring unit; An enhancement unit configured to input any first category of document image obtained by the classification unit into a resolution enhancer to obtain an image representation of the document image with enhanced resolution, so as to update the image representation of the document image; A reply unit configured to input the image representations and the text representation corresponding to the plurality of document images into the LLM to obtain reply texts for the plurality of document images and the question text.

10. A computer-readable storage medium having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method of any one of claims 1-8.

11. A computing device comprising a memory having executable code stored therein and a processor that, when executing the executable code, implements the method of any one of claims 1-8.

Images