Visual question and answer method and device based on multiple document images

Abstract

The embodiment of the invention provides a visual question and answer method and device based on a plurality of document images, and the method and device are executed by using an LVLM, and the LVLM comprises a multi-modal coding model and an LLM. The method comprises the steps of obtaining image representations corresponding to a plurality of document images and text representations corresponding to question texts through a multi-modal coding model; inputting each image representation and the text representation into a text supervision scoring device to obtain each correlation score of the plurality of document images to the question text; selecting a plurality of document images belonging to the first category from the plurality of document images; the first category falls into an interval with the highest correlation score in a plurality of intervals divided according to the correlation scores; inputting the document image of any first category into a resolution intensifier to obtain an image representation of the document image after the resolution is improved; and inputting the image representations and the text representations into LLM to obtain reply texts aiming at the plurality of document images and the question texts.

Description

Technical Field

[0001] One or more embodiments of this specification relate to the field of computers, and more particularly, to a method and apparatus for visual question answering based on multiple document images. Background Art

[0002] Document images are images rich in textual content. Examples include medical examination reports, financial statements, contracts, and academic papers. It's understood that document images can contain only text or a combination of text and graphics. Visual question answering (VQA) involves generating answers based on images and related questions.

[0003] Visual question answering tasks based on multiple document images have their own unique characteristics. First, the information density of document images is much higher than that of natural images. Natural images typically contain simple scenes, objects, or actions, while document images contain a large amount of text, symbols, charts, and so on. Second, document images are often more numerous than natural images. For example, a medical examination report, financial statement, or contract document may contain 20 or more pages, each of which contains independent visual text information.

[0004] For the above reasons, in visual question answering tasks based on multiple document images, we often face the problem of inaccurate generated answers or low processing efficiency. Summary of the Invention

[0005] One or more embodiments of this specification describe a visual question answering method and apparatus based on multiple document images, which can improve the accuracy of answer generation and have high processing efficiency.

[0006] In a first aspect, a visual question answering method based on multiple document images is provided, which is performed using a large vision language model (LVLM). The LVLM includes a multimodal encoding model and a large language model (LLM). The method includes: Obtaining, through the multimodal encoding model, image representations corresponding to the plurality of document images and text representations corresponding to the question text; Input each image representation and text representation into the text supervision scorer to obtain the relevance scores of multiple document images to the question text; Selecting a plurality of document images belonging to a first category from a plurality of document images; the first category falls into a highest relevance score interval among a plurality of intervals divided by relevance scores; For any document image of the first category, input it into a resolution enhancer to obtain an image representation of the document image after resolution enhancement; The image representations and text representations are input into the LLM to obtain answer texts for the plurality of document images and question texts.

[0007] In one embodiment, the multimodal encoding model includes a visual encoder and a text encoder, and obtaining the image representations corresponding to the plurality of document images and the text representation corresponding to the question text through the multimodal encoding model includes: Inputting multiple document images into a visual encoder respectively to obtain image representations corresponding to the multiple document images; Input the question text into the text encoder to obtain the text representation corresponding to the question text.

[0008] In one embodiment, the image representation includes block feature vectors corresponding to respective image blocks of the document image, and the text representation includes word feature vectors corresponding to respective word segments of the question text; inputting the image representations and text representations into a text supervision scorer to obtain respective relevance scores of the multiple document images to the question text includes: Determine the attention score between any image block and any word based on each block feature vector and each word feature vector; Perform the first pooling process on each attention score to obtain the first pooling score between any image block and the question text; Performing a second pooling process on the first pooling scores between each image block and the question text to obtain a second pooling score between any document image and the question text; The second pooled scores between each document image and the question text are normalized to obtain respective relevance scores of the multiple document images to the question text.

[0009] Furthermore, determining the attention score between any image block and any word segmentation according to each block feature vector and each word feature vector includes: Each block feature vector and each word feature vector are input into the first attention layer of LLM to obtain the attention score between any image block and any word segmentation.

[0010] In one embodiment, the resolution enhancer includes a visual encoder; wherein for any document image of the first category, inputting it into the resolution enhancer to obtain an image representation of the document image with enhanced resolution includes: For any document image of the first category, split it into multiple sub-images; Input any sub-image into the visual encoder to obtain a sub-image representation corresponding to the sub-image; The sub-image representations corresponding to the sub-images and the image representation of the document image are spliced together to obtain the image representation of the document image with improved resolution.

[0011] In one embodiment, the method further comprises: Determine the document image whose relevance score falls within a first interval as a document image of the second category; the first interval is the interval with the lowest relevance score among the multiple intervals divided according to the relevance score; For any document image of the second category, it is input into the information compressor to obtain an image representation of the document image after the compressed image scale.

[0012] Furthermore, the image representation includes block feature vectors corresponding to respective image blocks of the document image; and for any document image of the second category, inputting it into the information compressor to obtain the image representation of the document image after compression of the image scale includes: For any document image of the second category, input it into the information compressor, and the information compressor determines whether its relevance score falls into the first sub-interval or the second sub-interval; the first sub-interval and the second sub-interval constitute the first interval range, and any value in the first sub-interval is smaller than any value in the second sub-interval; If the information compressor determines that the relevance score falls within the first subinterval, all block feature vectors of the document image are discarded; If the information compressor determines that the correlation score falls into the second sub-interval, some block feature vectors of the document image are discarded.

[0013] Furthermore, the method further comprises: determining a plurality of document images other than the first category and the second category as document images of a third category; For any document image of the third category, the image representation of the document image is kept unchanged.

[0014] In a second aspect, a visual question answering apparatus based on multiple document images is provided, which is performed using a large visual language model (LVLM). The LVLM includes a multimodal encoding model and a large language model (LLM). The apparatus includes: An encoding unit, configured to obtain, through the multimodal encoding model, image representations corresponding to the plurality of document images and a text representation corresponding to the question text; A scoring unit, configured to input the image representations and text representations obtained by the encoding unit into a text supervision scorer to obtain respective relevance scores of the plurality of document images to the question text; a classification unit, configured to select a plurality of document images belonging to a first category from the plurality of document images; the first category falling into an interval with the highest relevance score among the plurality of intervals divided according to the relevance scores obtained by the scoring unit; an enhancement unit, configured to input any document image of the first category obtained by the classification unit into a resolution enhancer to obtain an image representation of the document image with enhanced resolution; The answering unit is configured to input the image representations and text representations obtained by the enhancing unit into the LLM to obtain answer texts for the plurality of document images and question texts.

[0015] According to a third aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed in a computer, the computer is caused to execute the method according to the first aspect.

[0016] In a fourth aspect, a computing device is provided, comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method of the first aspect is implemented.

[0017] Through the method and apparatus provided in the embodiments of this specification, first, a multimodal encoding model is used to obtain image representations corresponding to multiple document images and text representations corresponding to question texts; then, each image representation and text representation is input into a text supervision scorer to obtain respective relevance scores of the multiple document images to the question text; then, several document images belonging to a first category are selected from the multiple document images; the first category falls into the interval with the highest relevance score among multiple intervals divided by relevance scores; then, for any document image in the first category, it is input into a resolution enhancer to obtain an image representation of the document image with enhanced resolution; finally, each image representation and text representation is input into an LLM to obtain a response text for the multiple document images and the question text. As can be seen from the above, in the embodiments of this specification, a text supervision scorer is used to obtain respective relevance scores of the multiple document images to the question text, and then a key image is selected based on each relevance score, and the resolution of the key image is enhanced by the resolution enhancer, thereby improving the model effect, and selectively enhancing the image resolution, which can efficiently utilize computing resources. This solution can improve the accuracy of answer generation and has high processing efficiency.

[0018] In addition, the text supervision scorer and resolution enhancer are pluggable components that can be adapted to different LVLMs, improving performance and efficiency without modifying the underlying model architecture of the LVLM. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0020] Figure 1 A schematic diagram of an implementation scenario of an embodiment disclosed in this specification; Figure 2 A flowchart of a visual question answering method based on multiple document images according to one embodiment is shown; Figure 3 A schematic diagram of an encoding process based on a multimodal encoding model according to an embodiment is shown; Figure 4 A schematic diagram of a processing process based on a text supervision scorer according to one embodiment is shown; Figure 5 A schematic diagram of a processing process based on a resolution enhancer according to one embodiment is shown; Figure 6 A schematic diagram of a processing process based on an information compressor according to one embodiment is shown; Figure 7 A schematic diagram illustrating input and output of an LLM according to one embodiment is shown; Figure 8 A schematic block diagram of a visual question answering apparatus based on multiple document images according to one embodiment is shown. DETAILED DESCRIPTION

[0021] The solution provided in this specification is described below in conjunction with the accompanying drawings.

[0022] Figure 1This is a schematic diagram of an implementation scenario of an embodiment disclosed in this specification. This implementation scenario involves visual question answering based on multiple document images. It can be understood that in the visual question answering task, the question text and multiple document images are used as input, and the answer text is used as output, and both the question and the answer are in text form. On the one hand, document images usually contain a large amount of text, symbols, charts, etc., and have extremely high information compression characteristics. This dense information arrangement places higher demands on the model's understanding ability, because low resolution may cause blurred text and loss of details, which in turn affects the accuracy of parsing. Therefore, high-resolution images are usually used to ensure that the text is clear and the structure is complete. On the other hand, document images are not only information-intensive, but also usually more challenging in terms of quantity than natural images. Each page of the document image has independent visual text information, and there may be semantic associations or cross-page dependencies between these information. It is necessary to efficiently integrate multi-page information and perform targeted marking and processing. Processing each page of the document image alone may lead to the loss of the overall context, and large-scale processing may face bottlenecks in computing resources and efficiency.

[0023] Reference Figure 1 In the embodiments of this specification, by determining the relevance scores of each document image to the question text and dynamically adjusting the resolution, the resolution of key images is adaptively enhanced, effectively utilizing computing resources. It is understood that key images are document images with relatively high relevance scores. For example, document images A, B, and C have relevance scores of 0.14, 0.26, and 0.13, respectively. Compared to other document images, their relevance scores are relatively high, and their image scales can be increased to improve their resolution. For other document images, several document images with the lowest relevance scores can be further screened out, and their image scales can be reduced to compress unimportant images. For example, document images E and F have relevance scores of 0.10 and 0.01, respectively. For document image E, some image blocks are discarded to reduce its resolution, while for document image F, all image blocks are discarded. Document images with intermediate relevance scores can be left unprocessed and maintained at their standard resolution. For example, document image D, with a relevance score of 0.12, maintains its original scale.

[0024] It should be noted that Figure 1This is a typical implementation scenario, which includes three processing scenarios: for multiple document images, the resolution of some images is enhanced, the resolution of some images is reduced, and the resolution of others is maintained at standard resolution. In actual applications, depending on the relevance scores of the multiple document images, only two of these processing scenarios may be included. For example, for multiple document images, the resolution of some images is enhanced while the resolution of the others is reduced; or for multiple document images, the resolution of some images is enhanced while the resolution of the others is maintained at standard resolution.

[0025] Figure 2 A flowchart of a visual question answering method based on multiple document images according to an embodiment is shown, which is performed using LVLM. LVLM includes a multimodal encoding model and a large language model (LLM). The method can be based on Figure 1 The implementation scenario shown in Figure 2 As shown, the visual question answering method based on multiple document images in this embodiment includes the following steps: Step 21, obtaining image representations corresponding to the multiple document images and text representations corresponding to the question text through a multimodal encoding model; Step 22, inputting the image representations and text representations into a text supervision scorer to obtain the relevance scores of the multiple document images to the question text; Step 23, selecting a number of document images belonging to a first category from the multiple document images; the first category falls into the interval with the highest relevance score among multiple intervals divided by relevance scores; Step 24, for any document image of the first category, inputting it into a resolution enhancer to obtain an image representation of the document image with enhanced resolution; Step 25, inputting the image representations and text representations into an LLM to obtain a reply text for the multiple document images and the question text. The specific execution method of each of the above steps is described below.

[0026] First, in step 21, the multimodal coding model is used to obtain image representations corresponding to the plurality of document images and text representations corresponding to the question text. It can be understood that the multimodal coding model is a component of the LVLM.

[0027] Among them, LVLM is a model that combines natural language processing and computer vision technology, which can understand and generate text related to images.

[0028] In one example, the multimodal encoding model includes a visual encoder and a text encoder, and obtaining image representations corresponding to the plurality of document images and a text representation corresponding to the question text through the multimodal encoding model includes: Inputting multiple document images into a visual encoder respectively to obtain image representations corresponding to the multiple document images; Input the question text into the text encoder to obtain the text representation corresponding to the question text.

[0029] In this example, the vision transformer (VIT) can be reused to obtain image representations corresponding to each document image. The image representations can be in the form of vectors. The text encoder can specifically use a tokenizer to segment the text into tokens and then map each token to a corresponding vector space. In natural language processing, a token is the smallest semantic unit in a text and can be a word, subword, or character.

[0030] Figure 3 FIG. 1 shows a schematic diagram of an encoding process based on a multimodal encoding model according to an embodiment. Figure 3 The multimodal encoding model includes a VIT and a text encoder. The multiple document images are document image A, document image B, document image C, document image D, document image E, and document image F. Document image A is input into the VIT to obtain an image representation of document image A, document image B is input into the VIT to obtain an image representation of document image B, document image C is input into the VIT to obtain an image representation of document image C, document image D is input into the VIT to obtain an image representation of document image D, document image E is input into the VIT to obtain an image representation of document image E, and document image F is input into the VIT to obtain an image representation of document image F. The question text is "What is the name of...?", which is input into the text encoder to obtain a text representation corresponding to the question text.

[0031] Then, in step 22, each image representation and text representation is input into a text supervision scorer to obtain a relevance score for each of the multiple document images to the question text. It will be appreciated that the relevance scores reflect the contribution of different document images to the question text. In other words, document images with higher relevance scores are more helpful in answering the question text.

[0032] In one example, the image representation includes block feature vectors corresponding to respective image blocks of the document image, and the text representation includes word feature vectors corresponding to respective word segments of the question text; inputting the image representations and text representations into a text supervision scorer to obtain respective relevance scores of the multiple document images to the question text includes: Determine the attention score between any image block and any word based on each block feature vector and each word feature vector; Perform the first pooling process on each attention score to obtain the first pooling score between any image block and the question text; Performing a second pooling process on the first pooling scores between each image block and the question text to obtain a second pooling score between any document image and the question text; The second pooled scores between each document image and the question text are normalized to obtain respective relevance scores of the multiple document images to the question text.

[0033] In this example, the attention score between any image block and any word segment is first determined, then aggregated to the image level through pooling, and normalized to obtain a correlation score between 0 and 1, making it easier to distinguish the contribution of different document images to the question text.

[0034] Figure 4 FIG. 1 shows a schematic diagram of a processing process based on a text supervision scorer according to an embodiment. Figure 4 The input of the text supervision scorer is the image representation corresponding to M document images and the text representation of the question text, and its output is M relevance scores of M document images to the question text. The image representation of the kth document image is denoted as V k , where 1≤k≤M, the text representation of the question text is denoted as T, the image representation includes the block feature vectors corresponding to each image block of the document image, and the text representation includes the word feature vectors corresponding to each word segment of the question text. The block feature vectors and word feature vectors are represented by a small square in the figure. The attention scores between each image block and each word segment form a matrix composed of small squares. After the first pooling process, the dimension of the matrix is reduced in the dimension corresponding to the text, and after the second pooling process, it is reduced in the dimension corresponding to the image, and then normalized so that the relevance score is a decimal between 0 and 1. In the figure, s k Represents the relevance score of the k-th document image to the question text, 1≤k≤M.

[0035] Furthermore, determining the attention score between any image block and any word segmentation according to each block feature vector and each word feature vector includes: Each block feature vector and each word feature vector are input into the first attention layer of LLM to obtain the attention score between any image block and any word segmentation.

[0036] In this example, the LLM structure is reused, and there is no need to specifically train a text supervision scorer.

[0037] Next, at step 23, a plurality of document images belonging to a first category are selected from the plurality of document images; the first category falls within the highest relevance score interval among the plurality of intervals divided by relevance score. It will be appreciated that the first category represents the document image with the highest level of contribution to the question.

[0038] In the embodiment of this specification, multiple document images can be divided into preset categories according to the needs of the actual scenario. Each category falls into one of the multiple intervals divided by the relevance score, representing the corresponding level of contribution of the document image to the problem. For example, it is divided into 4 categories, including a very important category, an important category, an unimportant category, and a very unimportant category. It can be understood that the contribution levels of these categories are different.

[0039] Then, in step 24, any document image of the first category is input into a resolution enhancer to obtain an image representation of the document image after resolution enhancement. It is understood that, among the multiple document images, only the document images belonging to the first category are subjected to resolution enhancement.

[0040] In one example, the resolution enhancer includes a visual encoder; for any document image of the first category, inputting it into the resolution enhancer to obtain an image representation of the document image with enhanced resolution, including: For any document image of the first category, split it into multiple sub-images; Input any sub-image into the visual encoder to obtain a sub-image representation corresponding to the sub-image; The sub-image representations corresponding to the sub-images and the image representation of the document image are spliced together to obtain the image representation of the document image with improved resolution.

[0041] In this example, the image resolution is improved by adding markers.

[0042] Figure 5 FIG. 1 is a schematic diagram showing a processing process based on a resolution enhancer according to an embodiment. Figure 5After obtaining the relevance scores of multiple document images to the question text, the document images corresponding to the top several relevance scores can be selected, which can be implemented based on the topk(·) function. For example, the document images with the top two relevance scores, document image B and document image H, are selected. Next, the selected document images are split into multiple sub-images, which can be implemented based on the crop(·) function. For example, document image B is split into sub-image B1 and sub-image B2, and document image H is split into sub-image H1 and sub-image H2. Any sub-image is then input into the visual encoder VIT to obtain the sub-image representation corresponding to the sub-image, for example , respectively obtaining the sub-image representation of sub-image B1, the sub-image representation of sub-image B2, the sub-image representation of sub-image H1, and the sub-image representation of sub-image H2; finally, splicing the sub-image representations corresponding to each sub-image and the image representation of the document image to obtain the image representation of the document image with enhanced resolution. The splicing can be implemented based on the Cat(·) function. For example, the sub-image representation of sub-image B1, the sub-image representation of sub-image B2, and the image representation of image B are spliced to obtain the image representation of the document image B with enhanced resolution, wherein the sub-image representation of sub-image B1 and the sub-image representation of sub-image B2 are enhanced marks, and the image representation of the document image B is the original mark.

[0043] Finally, in step 25, each image representation and text representation is input into the LLM to obtain a response text for the plurality of document images and question texts. It can be understood that the above-mentioned each image representation includes the image representation of the document image after resolution enhancement.

[0044] In one example, before step 25, the method further includes: Determine the document image whose relevance score falls within a first interval as a document image of the second category; the first interval is the interval with the lowest relevance score among the multiple intervals divided according to the relevance score; For any document image of the second category, it is input into the information compressor to obtain an image representation of the document image after the compressed image scale.

[0045] In this example, for multiple document images, document images of the second category can also be screened out. These document images have the lowest contribution to answering the question, and compressing their image scale helps improve the processing efficiency of subsequent LLM.

[0046] Furthermore, the image representation includes block feature vectors corresponding to respective image blocks of the document image; and for any document image of the second category, inputting it into the information compressor to obtain the image representation of the document image after compression of the image scale includes: For any document image of the second category, input it into the information compressor, and the information compressor determines whether its relevance score falls into the first sub-interval or the second sub-interval; the first sub-interval and the second sub-interval constitute the first interval range, and any value in the first sub-interval is smaller than any value in the second sub-interval; If the information compressor determines that the relevance score falls within the first subinterval, all block feature vectors of the document image are discarded; If the information compressor determines that the correlation score falls into the second sub-interval, some block feature vectors of the document image are discarded.

[0047] In this example, the compressed image scale is further refined into two types: discarding all block feature vectors of the document image, and discarding part of the block feature vectors of the document image. This allows for differential processing to retain relatively important markers and improve the accuracy of answering questions.

[0048] Figure 6 FIG. 1 is a schematic diagram showing a processing process based on an information compressor according to an embodiment. Figure 6 After obtaining the relevance scores of the multiple document images to the question text, the document images whose relevance scores fall within the first interval range can be selected and determined as the document images of the second category. The first interval range is the lowest interval among the multiple intervals sorted from high to low in relevance scores. For example, the first interval range is [0, βγ]. The first interval range can be further divided into a first sub-interval and a second sub-interval. For example, the first sub-interval is [0, αγ]. , the second subinterval is (αγ, βγ], where α, β, and γ are preset parameters, and their values can be set according to scenario requirements. For any document image of the second category, it is input into the information compressor, and the information compressor determines whether its correlation score falls into the first subinterval or the second subinterval. If the information compressor determines that its correlation score falls into the first subinterval, all block feature vectors of the document image are discarded. For example, if the correlation scores of image A and image F fall into the first subinterval, all block feature vectors of the document image are discarded, that is, the image is discarded; if the information compressor determines that its correlation score falls into the second subinterval, some block feature vectors of the document image are discarded. For example, if the correlation scores of image C and image E fall into the second subinterval, some block feature vectors of the document image are discarded, that is, the image block is discarded. For intuitive representation in the figure, the × mark is used to represent the block feature vector of the discarded image block.

[0049] Furthermore, the method further comprises: determining a plurality of document images other than the first category and the second category as document images of a third category; For any document image of the third category, the image representation of the document image is kept unchanged.

[0050] In this example, the contribution of the document image of the third category is between that of the first category and the second category, and the image representation of the document image can be kept unchanged, that is, the resolution is not changed, thereby retaining relatively important information.

[0051] Figure 7 FIG. 1 shows an input and output diagram of an LLM according to an embodiment. Figure 7 The input of LLM consists of two parts: image representation and text representation. The image representation has been dynamically adjusted in resolution. The resolution of some document images has been improved, the resolution of some document images has been reduced, and the resolution of some document images has remained unchanged. The image representations of two document images are distinguished by blank intervals. Each small square represents a block image representation. × represents the discarded block image representation. The small squares without × and framed by dotted lines represent the added block image representation.

[0052] The embodiments of this specification may use pluggable components to adapt to a variety of large-scale visual language models (LVLM) to improve flexibility and scalability. The pluggable components include a textual influence evaluator (TIE) for evaluating the contribution of different images to the problem. It may further include a key image enhancer (KIE) for enhancing the resolution of important images; and a dynamic visual dropper (DVD) for compressing or discarding unimportant images. These pluggable components do not require additional training.

[0053] The embodiments of this specification bypass optical character recognition (OCR) tools for document images, directly performing multimodal fusion understanding. This allows for efficient integration of multi-page information, the ability to handle dynamic-length input, and improved robustness and generalization. This approach has a wide range of applications, including improving the accuracy and efficiency of report interpretation and physical examination report analysis in healthcare; rapidly and accurately processing complex documents and improving work efficiency in finance and contract management; and supporting the reading comprehension of academic papers and research reports in education and scientific research.

[0054] According to the method provided in the embodiment of this specification, first, a multimodal encoding model is used to obtain the image representations corresponding to the multiple document images and the text representations corresponding to the question text; then, the image representations and text representations are input into a text supervision scorer to obtain the relevance scores of the multiple document images to the question text; then, several document images belonging to the first category are selected from the multiple document images; the first category falls into the interval with the highest relevance score among the multiple intervals divided by the relevance score; then, for any document image of the first category, it is input into a resolution enhancer to obtain the image representation of the document image after the resolution is improved; finally, the image representations and text representations are input into the LLM to obtain the answer text for the multiple document images and the question text. As can be seen from the above, in the embodiment of this specification, the text supervision scorer is used to obtain the relevance scores of the multiple document images to the question text, and then the key images are selected according to the relevance scores, and the resolution of the key images is enhanced by the resolution enhancer, thereby improving the model effect, and selectively enhancing the image resolution, which can efficiently utilize computing resources. This solution can improve the accuracy of answer generation and has high processing efficiency.

[0055] According to another embodiment, a visual question answering device based on multiple document images is also provided, which is executed using a large visual language model LVLM, wherein the LVLM includes a multimodal encoding model and a large language model LLM. The device is used to execute the method provided in the embodiment of this specification. Figure 8 FIG. 1 is a schematic block diagram of a visual question answering apparatus based on multiple document images according to an embodiment. Figure 8 As shown, the apparatus 800 includes: An encoding unit 81 is configured to obtain image representations corresponding to the plurality of document images and a text representation corresponding to the question text using the multimodal encoding model; A scoring unit 82 is configured to input the image representations and text representations obtained by the encoding unit 81 into a text supervision scorer to obtain relevance scores of the plurality of document images to the question text; a classification unit 83 configured to select a plurality of document images belonging to a first category from the plurality of document images; the first category falling into the interval with the highest relevance score among the plurality of intervals divided according to the relevance scores obtained by the scoring unit 82; an enhancement unit 84 configured to input any document image of the first category obtained by the classification unit 83 into a resolution enhancer to obtain an image representation of the document image with enhanced resolution; The answering unit 85 is configured to input the image representations and text representations obtained by the enhancing unit 84 into the LLM to obtain answer texts for the multiple document images and question texts.

[0056] Optionally, as an embodiment, the multimodal encoding model includes a visual encoder and a text encoder, and the encoding unit 81 includes: An image encoding subunit, configured to input the plurality of document images into a visual encoder respectively to obtain image representations corresponding to the plurality of document images; The text encoding subunit is used to input the question text into the text encoder to obtain the text representation corresponding to the question text.

[0057] Optionally, as an embodiment, the image representation includes block feature vectors corresponding to respective image blocks of the document image, and the text representation includes word feature vectors corresponding to respective word segments of the question text; the scoring unit 82 includes: The attention subunit is used to determine the attention score between any image block and any word based on the feature vectors of each block and each word; A first pooling subunit is used to perform a first pooling process on each attention score obtained by the attention subunit to obtain a first pooling score between any image block and the question text; A second pooling subunit is configured to perform a second pooling process on the first pooling scores between each image block and the question text obtained by the first pooling subunit to obtain a second pooling score between any document image and the question text; The normalization subunit is used to normalize the second pooling scores between each document image and the question text obtained by the second pooling subunit to obtain each relevance score of the multiple document images to the question text.

[0058] Furthermore, the attention subunit is specifically used to input each block feature vector and each word feature vector into the first attention layer of LLM to obtain an attention score between any image block and any word segmentation.

[0059] Optionally, as an embodiment, the resolution enhancer includes a visual encoder; and the enhancement unit 84 includes: a splitting sub-unit, configured to split any document image of the first category into a plurality of sub-images; an encoding subunit, configured to input any sub-image obtained by splitting the sub-units into the visual encoder to obtain a sub-image representation corresponding to the sub-image; The splicing subunit is used to splice the sub-image representations corresponding to the sub-images obtained by the encoding subunit and the image representation of the document image to obtain the image representation of the document image with improved resolution.

[0060] Optionally, as an embodiment, the classification unit 83 is further configured to determine a document image whose relevance score falls within a first interval as a document image of the second category; the first interval is an interval with the lowest relevance score among the multiple intervals divided by the relevance score; The apparatus further comprises a compression unit configured to input any second category document image obtained by the classification unit 83 into an information compressor to obtain an image representation of the document image after compression of the image scale.

[0061] Furthermore, the image representation includes block feature vectors corresponding to respective image blocks of the document image; and the compression unit includes: a determination subunit, configured to input any document image of the second category into the information compressor, and have the information compressor determine whether its relevance score falls within the first subinterval or the second subinterval; the first subinterval and the second subinterval constitute the first interval range, and any value within the first subinterval is smaller than any value within the second subinterval; a first compression subunit, configured to discard all block feature vectors of the document image if the information compressor determines that the relevance score falls within a first subinterval; The second compression subunit is configured to discard part of the block feature vectors of the document image if the information compressor determines that the correlation score falls into the second subinterval.

[0062] Furthermore, the classification unit 83 is further configured to determine a plurality of document images other than the first category and the second category as document images of the third category; The apparatus further includes a maintaining unit configured to maintain the image representation of the document image unchanged for any document image of the third category obtained by the classification unit 83 .

[0063] Through the device provided by the embodiment of this specification, first, the encoding unit 81 obtains the image representations corresponding to the multiple document images and the text representations corresponding to the question text through the multimodal encoding model; then the scoring unit 82 inputs the image representations and text representations into the text supervision scorer to obtain the relevance scores of the multiple document images to the question text; then the classification unit 83 selects a number of document images belonging to the first category from the multiple document images; the first category falls into the interval with the highest relevance score among the multiple intervals divided by the relevance score; the enhancement unit 84 then inputs any document image of the first category into the resolution enhancer to obtain the image representation of the document image with enhanced resolution; finally, the reply unit 85 inputs the image representations and text representations into the LLM to obtain the reply text for the multiple document images and the question text. As can be seen from the above, the embodiment of this specification obtains the relevance scores of the multiple document images to the question text through the text supervision scorer, then selects the key images based on the relevance scores, and enhances the resolution of the key images through the resolution enhancer, thereby improving the model effect, and selectively enhances the image resolution, which can efficiently utilize computing resources. This solution can improve the accuracy of answer generation and has high processing efficiency.

[0064] According to another embodiment, there is also provided a computer readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 2 The method described.

[0065] According to another embodiment, a computing device is provided, including a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the system realizes the combination of Figure 2 The method described.

[0066] Those skilled in the art will appreciate that, in one or more of the above examples, the functions described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium.

[0067] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made on the basis of the technical solution of the present invention should be included in the scope of protection of the present invention.

Claims

1. A visual question answering method based on multiple document images, performed using a large visual language model (LVLM), wherein the LVLM includes a multimodal encoding model and a large language model (LLM), the method comprising: Obtaining, through the multimodal encoding model, image representations corresponding to the plurality of document images and text representations corresponding to the question text; Input each image representation and text representation into the text supervision scorer to obtain the relevance scores of multiple document images to the question text; Selecting a plurality of document images belonging to a first category from a plurality of document images; the first category falls into a highest relevance score interval among a plurality of intervals divided by relevance scores; For any document image of the first category, input it into a resolution enhancer to obtain an image representation of the document image after resolution enhancement; The image representations and text representations are input into the LLM to obtain answer texts for the plurality of document images and question texts.

2. The method according to claim 1, wherein The multimodal encoding model includes a visual encoder and a text encoder, and obtaining image representations corresponding to the plurality of document images and a text representation corresponding to the question text through the multimodal encoding model includes: Inputting multiple document images into a visual encoder respectively to obtain image representations corresponding to the multiple document images; Input the question text into the text encoder to obtain the text representation corresponding to the question text.

3. The method according to claim 1, wherein The image representation includes block feature vectors corresponding to each image block of the document image, and the text representation includes word feature vectors corresponding to each word segment of the question text; inputting each image representation and text representation into a text supervision scorer to obtain each relevance score of the multiple document images to the question text includes: Determine the attention score between any image block and any word based on each block feature vector and each word feature vector; Perform the first pooling process on each attention score to obtain the first pooling score between any image block and the question text; Performing a second pooling process on the first pooling scores between each image block and the question text to obtain a second pooling score between any document image and the question text; The second pooled scores between each document image and the question text are normalized to obtain respective relevance scores of the multiple document images to the question text.

4. The method according to claim 3, wherein: Determining the attention score between any image block and any word segmentation according to each block feature vector and each word feature vector includes: Each block feature vector and each word feature vector are input into the first attention layer of LLM to obtain the attention score between any image block and any word segmentation.

5. The method according to claim 1, wherein The resolution enhancer includes a visual encoder; The step of inputting any document image of the first category into a resolution enhancer to obtain an image representation of the document image after resolution enhancement includes: For any document image of the first category, split it into multiple sub-images; Input any sub-image into the visual encoder to obtain a sub-image representation corresponding to the sub-image; The sub-image representations corresponding to the sub-images and the image representation of the document image are spliced together to obtain the image representation of the document image with improved resolution.

6. The method of claim 1, wherein: The method further comprises: Determine the document image whose relevance score falls within a first interval as a document image of the second category; the first interval is the interval with the lowest relevance score among the multiple intervals divided according to the relevance score; For any document image of the second category, it is input into the information compressor to obtain an image representation of the document image after the compressed image scale.

7. The method according to claim 6, wherein: The image representation includes block feature vectors corresponding to respective image blocks of the document image; The step of inputting any document image of the second category into the information compressor to obtain an image representation of the document image after compression of the image scale comprises: For any document image of the second category, input it into the information compressor, and the information compressor determines whether its relevance score falls into the first sub-interval or the second sub-interval; The first sub-interval and the second sub-interval constitute the first interval range, and any value in the first sub-interval is smaller than any value in the second sub-interval; If the information compressor determines that the relevance score falls within the first subinterval, all block feature vectors of the document image are discarded; If the information compressor determines that the correlation score falls into the second sub-interval, some block feature vectors of the document image are discarded.

8. The method of claim 6, wherein: The method further comprises: determining a plurality of document images other than the first category and the second category as document images of a third category; For any document image of the third category, the image representation of the document image is kept unchanged.

9. A visual question answering device based on multiple document images, using a large visual language model (LVLM), wherein the LVLM includes a multimodal encoding model and a large language model (LLM), the device comprising: An encoding unit, configured to obtain, through the multimodal encoding model, image representations corresponding to the plurality of document images and a text representation corresponding to the question text; A scoring unit, configured to input the image representations and text representations obtained by the encoding unit into a text supervision scorer to obtain respective relevance scores of the plurality of document images to the question text; a classification unit, configured to select a plurality of document images belonging to a first category from the plurality of document images; the first category falling into an interval with the highest relevance score among the plurality of intervals divided according to the relevance scores obtained by the scoring unit; an enhancement unit, configured to input any document image of the first category obtained by the classification unit into a resolution enhancer to obtain an image representation of the document image with enhanced resolution; The answering unit is configured to input the image representations and text representations obtained by the enhancing unit into the LLM to obtain answer texts for the plurality of document images and question texts.

10. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 8.

11. A computing device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 8 is implemented.

Images