Image-based question and answer method and device, equipment and storage medium

Abstract

The invention provides an image-based question and answer method and device, equipment and a storage medium, and relates to the technical field of image-text processing. The method comprises the following steps: acquiring a to-be-processed image and a corresponding problem text, and inputting the to-be-processed image and the problem text into an image-text large model comprising a visual expert module, a large language sub-model and a plurality of small models of different task types; wherein the input of each small model is a to-be-processed image and a question text, and the output of each small model is a first sub-image feature and a first sub-answer text. Next, the visual expert module determines a target task type according to the question text, and endows the output of the small model corresponding to the target task type with weight confidence higher than that of other small models; and finally, the large language sub-model generates an answer text according to the to-be-processed image, the question text and the output endowed with the weight confidence of each small model. According to the method provided by the invention, the image-text understanding capability of the image-text large model is improved, so that accurate answers can be output.

Description

Technical Field

[0001] The present application relates to the field of image and text processing technology, and in particular to an image-based question and answer method, device, equipment and storage medium. Background Art

[0002] In recent years, with the development of various artificial intelligence models, multimodal large language models (MLLM) have become increasingly popular in the research field. Based on the large language model (LLM), MLLM incorporates other non-textual modal information, showing more general capabilities and supporting a wider range of tasks.

[0003] The existing technology uses MLLM based on LLM and combines it with a visual encoder to implement the image-text question answering process. First, the image to be processed and the corresponding question text are input into the MLLM, and then the visual encoder is used to extract features of the image to be processed. Then, the image features extracted by the visual encoder and the question text corresponding to the image to be processed are mapped to the LLM, and the LLM outputs the corresponding answer.

[0004] However, the existing technology has a technical problem that the output answer is not accurate enough due to insufficient image and text comprehension ability. Summary of the invention

[0005] The present application provides an image-based question-answering method, apparatus, device and storage medium, which are used to solve the technical problem in the prior art that the output answers are not accurate enough due to insufficient image and text comprehension ability.

[0006] In a first aspect, the present application provides an image-based question answering method, comprising:

[0007] Obtaining an image to be processed and a question text corresponding to the image to be processed;

[0008] Input the image to be processed and the question text into a large image-text model, and obtain the answer text output by the large image-text model;

[0009] The large image-text model includes a visual expert module, a large language sub-model, and multiple small models of different task types;

[0010] The input of each small model is the image to be processed and the question text, and the output is the first sub-image feature and the first sub-answer text; the visual expert module is used to determine the target task type according to the question text, and assign a weighted confidence to the output of the small model corresponding to the target task type that is higher than that of other small models; the large language sub-model is used to generate the answer text according to the image to be processed, the question text and the output of each small model with the weighted confidence assigned.

[0011] Optionally, the large image-text model also includes an image feature extraction module, which is used to extract the second sub-image features of the image to be processed, and the large language sub-model is used to generate the answer text based on the image to be processed, the question text, the second sub-image features and the output of the weighted confidence of each small model.

[0012] Optionally, the large image-text model further includes a cross attention mechanism, and the cross attention mechanism is used to concatenate the second sub-image feature and the first sub-image feature of each small model with weighted confidence, to generate a third sub-image feature;

[0013] The cross attention mechanism is also used to concatenate the question text and the first sub-answer text of each small model with weighted reset confidence to generate a target text;

[0014] The cross-attention mechanism is also used to concatenate the third sub-image features and the target text and input them into the large language sub-model.

[0015] Optionally, the task types include at least two of the following: detection, segmentation, classification, and optical character recognition.

[0016] Optionally, the large image-text model further includes an adapter, and the adapter is used to align the third sub-image feature to the feature space of the large language sub-model to generate a new third sub-image feature.

[0017] Optionally, before obtaining the image to be processed and the question text corresponding to the image to be processed, the method further includes:

[0018] Obtain multiple sample images and a sample question text and a sample answer text corresponding to each sample image;

[0019] Training is performed based on each sample image and the corresponding sample question text and sample answer text to obtain a large image-text model.

[0020] In a second aspect, the present application provides an image-based question-answering device, comprising:

[0021] An acquisition module, used for acquiring an image to be processed and a question text corresponding to the image to be processed;

[0022] An input module, used for inputting the image to be processed and the question text into the image-text model;

[0023] The acquisition module is further used to acquire the answer text output by the large picture and text model;

[0024] The large image-text model includes a visual expert module, a large language sub-model, and multiple small models of different task types;

[0025] The input of each small model is the image to be processed and the question text, and the output is the first sub-image feature and the first sub-answer text; the visual expert module is used to determine the target task type according to the question text, and assign a weighted confidence to the output of the small model corresponding to the target task type that is higher than that of other small models; the large language sub-model is used to generate the answer text according to the image to be processed, the question text and the output of each small model with the weighted confidence assigned.

[0026] Optionally, the large image-text model also includes an image feature extraction module, which is used to extract the second sub-image features of the image to be processed, and the large language sub-model is used to generate the answer text based on the image to be processed, the question text, the second sub-image features and the output of the weighted confidence of each small model.

[0027] Optionally, the large image-text model further includes a cross attention mechanism, and the cross attention mechanism is used to concatenate the second sub-image feature and the first sub-image feature of each small model with weighted confidence, to generate a third sub-image feature;

[0028] The cross attention mechanism is also used to concatenate the question text and the first sub-answer text of each small model with weighted reset confidence to generate a target text;

[0029] The cross-attention mechanism is also used to concatenate the third sub-image features and the target text and input them into the large language sub-model.

[0030] Optionally, the task types include at least two of the following: detection, segmentation, classification, and optical character recognition.

[0031] Optionally, the large image-text model further includes an adapter, and the adapter is used to align the third sub-image feature to the feature space of the large language sub-model to generate a new third sub-image feature.

[0032] Optionally, the image-based question-answering device further includes: a training module,

[0033] The acquisition module is further used to acquire multiple sample images and sample question text and sample answer text corresponding to each sample image;

[0034] The training module is used to perform training based on each sample image and the corresponding sample question text and sample answer text to obtain a large image-text model.

[0035] In a third aspect, the present application provides an electronic device, including:

[0036] Processor and memory;

[0037] Wherein, the memory stores computer-executable instructions;

[0038] The processor executes the computer-executable instructions stored in the memory to implement the method as described in the first aspect and various possible implementation methods.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect and various possible implementation methods.

[0040] In a fifth aspect, the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it is used to implement the method described in the first aspect and various possible implementation methods.

[0041] The image-based question-answering method, apparatus, device and storage medium provided in the present application obtain the image to be processed and the corresponding question text, and input the image to be processed and the question text into a large picture-text model including a visual expert module, a large language sub-model, and a plurality of small models of different task types. Among them, the input of each small model is the image to be processed and the question text, and the output is the first sub-image feature and the first sub-answer text. Integrating the capabilities of the small model into the large picture-text model allows the large picture-text model to pay attention to more details of the image to be processed. While ensuring the generalization ability of the large picture-text model, the unique specialization ability of the small model is utilized to further improve the picture-text understanding ability of the large picture-text model. Next, the visual expert module determines the target task type according to the question text, and assigns a weighted confidence higher than that of other small models to the output of the small model corresponding to the target task type, so as to have stronger processing capabilities in dealing with different visual tasks, so that the large picture-text model has stronger encoding and detail perception capabilities for the image to be processed. Finally, the large language sub-model can understand the image to be processed and the corresponding question text more comprehensively based on the image to be processed, the question text, and the output of the weighted confidence of each small model, and then output the accurate answer text. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] Figure 1 It is an overall architecture diagram of the prior art picture-text question answering method;

[0044] Figure 2 A flowchart of an image-based question-answering method provided in an embodiment of the present application;

[0045] Figure 3 The overall architecture diagram of the image-based question-answering method provided in the embodiment of the present application;

[0046] Figure 4 An application scenario diagram of the image-based question-answering method provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of the structure of the image-based question-answering device provided in this application;

[0048] Figure 6 This is a hardware structure diagram of the electronic device provided for this application.

[0049] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0050] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0051] The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein, for example.

[0052] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0053] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0054] Terminology explanation:

[0055] LLM: It is a type of artificial intelligence model based on deep learning. By training on large-scale text data, it can understand and generate text similar to human language, and perform a variety of natural language processing tasks including text generation, translation, and sentiment analysis.

[0056] MLLM: Based on the natural language processing capabilities of LLM, it is further expanded and can process and generate data in multiple modalities, which can include text, images, audio, video, etc. This multimodal capability allows MLLM to integrate information from different modalities to achieve more comprehensive and accurate understanding and generation capabilities.

[0057] In recent years, artificial intelligence technology has developed rapidly, and various artificial intelligence models have emerged. With the increase in computing power and data resources, researchers are able to develop more complex and powerful models, which has laid the foundation for the emergence and popularity of MLLM.

[0058] The core of MLLM is LLM, which expands and integrates information from other modalities on the basis of LLM. LLM has demonstrated powerful capabilities in natural language processing tasks, so MLLM can not only utilize LLM's language understanding and generation capabilities, but also process non-text data at the same time. For example, many models such as GPT4-V and Gemini have demonstrated amazing capabilities in visual perception and understanding.

[0059] Since MLLMs can integrate information from multiple modalities, they exhibit more general capabilities. This means that MLLMs can be applied to a wider range of tasks, not just text-only tasks, but also complex tasks that require combining multiple data types, such as picture-text question answering, image caption generation, audio-video analysis, etc.

[0060] When processing image-text question answering tasks through MLLM, LLM is used as the basis and combined with the visual encoder to complete the processing of images and texts. Figure 1 The overall architecture diagram of the prior art picture-text question answering method is as follows: Figure 1 As shown in the figure, after the image to be processed and the corresponding question text are input into MLLM, the visual encoder is used to extract features of the image to be processed. The image features extracted by the visual encoder are then mapped to LLM in the form of text, and the question text is input into LLM at the same time. LLM outputs the corresponding answer based on the mapped image features and question text.

[0061] When processing image-text question answering tasks through MLLM, LLM is used as the basis and combined with the visual encoder to complete the processing of images and texts. Figure 1 The overall architecture diagram of the prior art picture-text question answering method is as follows: Figure 1 As shown in the figure, after the image to be processed and the corresponding question text are input into the MLLM, the visual encoder is used to extract features of the image to be processed. The image features extracted by the visual encoder are then mapped to the LLM in the form of text, and the question text is input into the LLM at the same time. Among them, the LLM includes multiple sub-models that handle different tasks. For example, one sub-model may focus on language understanding, and another sub-model may focus on language generation. The mapped image features and question text are processed by multiple sub-models in the LLM, and finally the LLM outputs the corresponding answer.

[0062] However, there is a huge gap between the visual encoder and LLM in terms of parameter scale and feature representation capability. When the image features extracted by the visual encoder are directly mapped to the LLM in text form, the collaboration between the visual encoder and LLM may not be close enough, resulting in information loss or misunderstanding.

[0063] In addition, it takes a certain amount of computing resources to extract features from the image to be processed through the visual encoder. The higher the resolution of the image to be processed, the more computing resources are required. When computing resources are limited, it is necessary to reduce the resolution of the image to reduce the computational complexity so that the visual encoder can still run effectively under limited resources. Although this method can reduce the demand for computing resources, it may not be able to capture complex image features, resulting in loss of detailed information on the image, which in turn makes LLM unable to obtain accurate and complete information.

[0064] It can be seen that although MLLM has stronger generalization ability and can integrate information from different modalities, the collaboration between the visual encoder and LLM in MLLM is not close enough, and the visual encoder needs certain computing resources to process images, which leads to the inability of the visual encoder to pass accurate and complete information to LLM, further resulting in insufficient image and text comprehension ability of LLM and inaccurate output answers.

[0065] In response to the above technical problems, the inventors found in the process of studying the processing of image-text question-answering tasks through MLLM that the image features obtained by only extracting features of the image to be processed through the visual encoder are not complete and accurate enough. Based on this, the inventors added small models for processing different tasks to the existing MLLM, and used these small models to process the image to be processed again, outputting image features and sub-answer texts corresponding to the image features. Afterwards, the image features extracted by the visual encoder, the image features output by the small model, and the sub-answer text are input into the LLM together to ensure that the LLM can obtain complete and accurate image information, thereby improving the image and text processing capabilities of the LLM and enabling it to output accurate answers.

[0066] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0067] Figure 2 The following is a flow chart of the image-based question-answering method provided in the embodiment of the present application. Figure 2 As shown, an image-based question-answering method provided in an embodiment of the present application includes:

[0068] S201, obtaining an image to be processed and a question text corresponding to the image to be processed.

[0069] The question text is a text input related to the image to be processed, usually a question or query requiring an answer to be provided based on the content of the image to be processed.

[0070] There is some correlation between the question text and the image to be processed, which may require the identification, description or analysis of certain specific elements in the image to be processed. For example, given an image of an animal, the question text may be "What animal is this?"; or given an image of a store sign, the question text may be "Describe the content of the store sign in the picture."

[0071] S202: Input the image to be processed and the question text into the large picture-text model, and obtain the answer text output by the large picture-text model.

[0072] First of all, it should be noted that the large picture and text model mentioned in this application is MLLM, which will not be repeated later.

[0073] In one possible implementation, Figure 3 The overall architecture diagram of the image-based question-answering method provided in the embodiment of the present application. Figure 3 As shown, the large image-text model includes a visual expert module, multiple small models of different task types, a large language sub-model, an image feature extraction module, and an adapter.

[0074] Next, the function of each component of the graphic model is explained.

[0075] Among them, the image feature extraction module can be regarded as a visual encoder, which is mainly used to encode (compress) the image to be processed, extract the second sub-image features of the image to be processed, and put the second sub-image features into the image feature space of the large image-text model.

[0076] Multiple small models of different task types are planned by the visual expert module, which analyzes the intent of the image and question text to be processed, determines the current scene requirements, determines the most core target task type, and gives the output of the small model corresponding to the target task type a higher weighted confidence than other small models. It is worth noting that the task types include at least two of the following: detection, segmentation, classification, and optical character recognition.

[0077] For example, when a user submits a graphic understanding requirement for analyzing text in an image, the visual expert module will first determine that the requirement is an optical character recognition task (target task type) and give a small model feature reward mechanism corresponding to the optical character recognition task to increase the weight ratio (weighted confidence) of the small model.

[0078] Explanation: The input of each small model is the image to be processed and the question text, and the output includes two parts, namely the first sub-image feature and the first sub-answer text. Among them, the first sub-answer text is mainly used to construct the prompt word; the first sub-image feature can be output by the last layer (fully connected layer) of the small model, or by the previous layer (neural network layer), and the first sub-image feature also needs to be placed in the image feature space of the large image-text model.

[0079] It is understandable that the fully connected layer outputs detailed image features. In some cases where particularly detailed image features are not required, the image features of the neural network layer can be output. The specific layer of image features to be output depends on actual needs and is not limited here.

[0080] The adapter is mainly to align the first sub-image features and the second sub-image features in the image feature space to the language feature space of the large language sub-model. Specifically, the adapter is a component that contains a randomly initialized cross-attention mechanism, which consists of a set of trainable embedding vectors. The adapter takes the first sub-image features and the second sub-image features as key inputs, and uses the cross-attention mechanism to concatenate the first sub-image features and the second sub-image features to generate the third sub-image features.

[0081] Furthermore, the adapter aligns the third sub-image features to the language feature space of the large language sub-model to generate new third sub-image features.

[0082] In addition, the adapter uses a cross-attention mechanism to concatenate the question text and the first sub-answer text to generate the target text and put it into the language feature space of the large language sub-model. After that, the third sub-image features in the language feature space are concatenated with the target text and input into the large language sub-model.

[0083] The large language sub-model is the most critical basic component of the large text and image model, which is mainly used to understand user instructions and output corresponding results. For example, Qwen7B / Yi-34B can be used as large language sub-models with different parameter magnitudes.

[0084] That is to say, the large language sub-model generates the final answer text based on the image to be processed, the question text, the second sub-image features extracted by the image feature extraction module, and the output of each small model with weighted confidence (the first sub-image features and the first sub-answer text).

[0085] It can be seen that the large image and text model can process image and text information at the same time. In order to enable the model to learn richer features and models and improve its accuracy and robustness in different scenarios and tasks, large-scale training can be carried out to further adapt to diverse data and improve generalization capabilities.

[0086] Specifically, training can be performed in the following manner: obtain multiple sample images and sample question text and sample answer text corresponding to each sample image, perform training based on each sample image and the corresponding sample question text and sample answer text, and obtain a large image-text model.

[0087] The image-based question-answering method provided by the present application obtains the image to be processed and the corresponding question text, and inputs the image to be processed and the question text into the large picture-text model. The image to be processed and the question text are processed by multiple small models of different task types in the large picture-text model to obtain the first sub-image feature and the first sub-answer text; and the image to be processed is processed by the image feature extraction module in the large picture-text model to obtain the second sub-image feature. Integrating the capabilities of the small model into the large picture-text model allows the large picture-text model to pay attention to more details of the image to be processed. While ensuring the generalization ability of the large picture-text model, the unique specialization ability of the small model is used to further improve the picture-text understanding ability of the large picture-text model. In the process of processing the image to be processed and the question text by the small model, the target task type is determined according to the question text based on the visual expert module in the large picture-text model, and the output of the small model corresponding to the target task type is given a weighted confidence higher than that of other small models, so as to have stronger processing capabilities in dealing with different visual tasks, so that the large picture-text model has stronger encoding and detail perception capabilities for the processed images. Next, the first sub-image features and the second sub-image features, the first sub-answer text and the question text are concatenated respectively to obtain the corresponding third sub-image features and the target text. At the same time, the third sub-image features and the target text are aligned through the adapter in the image-text large model, that is, the third sub-image features and the target text are mapped to the language feature space of the large language sub-model in the image-text large model, so as to ensure that the large language sub-model can understand the image to be processed and the corresponding question text more comprehensively, and then output an accurate answer text.

[0088] Next, a specific embodiment is used to Figure 2 The image-based question answering method shown is further explained.

[0089] Figure 4 The application scenario diagram of the image-based question-answering method provided in the embodiment of the present application is as follows: Figure 4 As shown in the figure, the input of the image-text model is the image to be processed containing the shop sign and the corresponding question text "Introduce the content of the shop sign in the picture".

[0090] First, the image to be processed is subjected to feature extraction through the image feature extraction module. Among them, the image feature extraction module can use ViT-S0400M-14-SigLIP-384 as a visual encoder, which can well extract the second sub-image features of the image to be processed and conform to the target mainstream picture-text question-answering framework.

[0091] Next, a convolutional neural network is used to extract features from the image to be processed from another perspective. The convolutional neural network takes the visual expert module as the core and integrates multiple small models of different task types. Among them, the small models used in this embodiment include a classification task small model, an optical character recognition task small model, and a detection task small model.

[0092] Specifically, the visual expert module first performs intent analysis on the image to be processed and the corresponding question text, determines the user's input intent, further determines the most core target task type, and improves the weighted confidence of the output of the small model corresponding to the target task type. For example, if the visual expert module determines that the target task type is an optical character recognition task, it will improve the weighted confidence of the output of the small model corresponding to the optical character recognition task, making the output of this small model more important in the subsequent forward feedback.

[0093] After that, the small model outputs the extracted first sub-image features and the corresponding first sub-answer text, and concatenates the first sub-answer text with the question text to form a new text input (target text) to further enhance the capabilities of the large image-text model from the perspective of prompt words. For example, "What does the store sign in the image contain? <ocr> Lao Zhang's small-grain stinky tofu is a unique dish in Changsha and a century-old delicacy. Franchise hotline: 400-660-XXXX< / ocr> <cls> house< / cls> <det> None< / det> ”. Further, the target text is serialized to be converted into a text embedding vector that can be accepted by the large language sub-model.

[0094] It should be understood that the first sub-image features and the second sub-image features extracted by the small model and the image feature extraction module are still in the image feature space and cannot be directly understood by the large language sub-model. Therefore, the adapter is used as the intermediate connection layer between the image feature space and the language feature space to align the image feature space with the language feature space.

[0095] In one possible implementation, the first sub-image feature and the second sub-image feature are concatenated to obtain the third sub-image feature. The adapter uses a cross-attention mechanism to calculate the attention weights of the third sub-image feature and the language feature space, capturing the correlation between the two, thereby mapping the third sub-image feature to the language feature space (alignment), and obtaining an image embedding vector in the language feature space.

[0096] Finally, the image embedding vector obtained by mapping the third sub-image features and the text embedding vector obtained by mapping the serialized target text are concatenated together and input into the large language sub-model to obtain the answer text output by the large language sub-model.

[0097] It can be seen from this that integrating the capabilities of the small model into the large image and text model can ensure the generalization ability of the large image and text model while utilizing the unique specialization of the small model, so that the large image and text model can pay attention to more details of the image to be processed, further improving the image and text understanding ability of the large image and text model.

[0098] Figure 5 This is a schematic diagram of the structure of the image-based question-answering device provided in this application. Figure 5 As shown, the image-based question-answering device 500 includes: an acquisition module 501 and an input module 502;

[0099] The acquisition module 501 is used to acquire the image to be processed and the question text corresponding to the image to be processed;

[0100] An input module 502 is used to input the image to be processed and the question text into the image-text model;

[0101] The acquisition module 501 is also used to obtain the answer text output by the large picture and text model;

[0102] The large image-text model includes a visual expert module, a large language sub-model, and multiple small models of different task types.

[0103] The input of each small model is the image to be processed and the question text, and the output is the first sub-image feature and the first sub-answer text; the visual expert module is used to determine the target task type based on the question text, and assign a weighted confidence higher than that of other small models to the output of the small model corresponding to the target task type; the large language sub-model is used to generate the answer text based on the image to be processed, the question text and the output of each small model with weighted confidence.

[0104] Optionally, the large image-text model also includes an image feature extraction module, which is used to extract the second sub-image features of the image to be processed. The large language sub-model is used to generate the answer text based on the image to be processed, the question text, the second sub-image features and the output of the weighted confidence of each small model.

[0105] Optionally, the large image-text model further includes a cross attention mechanism, which is used to concatenate the second sub-image feature and the first sub-image feature of each small model with weighted confidence, to generate a third sub-image feature;

[0106] The cross-attention mechanism is also used to concatenate the question text and the first sub-answer text of each small model with weighted reset confidence to generate the target text;

[0107] The cross-attention mechanism is also used to concatenate the third sub-image features and the target text and input them into the large language sub-model.

[0108] Optionally, the task type includes at least two of the following: detection, segmentation, classification, and optical character recognition.

[0109] Optionally, the large image-text model also includes an adapter, which is used to align the third sub-image features to the feature space of the large language sub-model to generate new third sub-image features.

[0110] Optionally, the image-based question-answering device 500 further includes: a training module 503,

[0111] The acquisition module 501 is further used to acquire multiple sample images and sample question text and sample answer text corresponding to each sample image;

[0112] The training module 503 is used to perform training based on each sample image and the corresponding sample question text and sample answer text to obtain a large image-text model.

[0113] It should be understood that the acquisition module 501 can be implemented as Figure 3 The visual expert module or image feature extraction module shown.

[0114] The image-based question-answering device provided in the embodiments of the present application can be used to execute the image-based question-answering method in any of the above embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0115] It should be noted that it should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. And these modules can all be implemented in the form of software calling through processing elements; they can also be all implemented in the form of hardware; some modules can also be implemented in the form of software calling through processing elements, and some modules can be implemented in the form of hardware. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in the processor element or instructions in the form of software.

[0116] Figure 6 The hardware structure diagram of the electronic device provided in this application. Figure 6 As shown, the electronic device 600 includes:

[0117] Processor 601 and memory 602;

[0118] Memory stores computer-executable instructions;

[0119] The processor executes the computer-executable instructions stored in the memory 602 , so that the electronic device executes the above-mentioned method.

[0120] It should be understood that the processor 601 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), or application-specific integrated circuits (ASIC). A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor. The memory 602 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory, and may also be a USB flash drive, a mobile hard disk, a read-only memory, a disk, or an optical disk.

[0121] The electronic device provided in the embodiment of the present application can be used to execute the method provided in any of the above method embodiments. Its implementation principles and technical effects are similar and will not be repeated here.

[0122] The embodiment of the present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the above method.

[0123] The embodiment of the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it is used to implement the method as described above.

[0124] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0125] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0126] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0127] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0128] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0129] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a memory, including a number of instructions to enable a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0130] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0131] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0132] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. An image-based question answering method, characterized in that: include: Obtaining an image to be processed and a question text corresponding to the image to be processed; Input the image to be processed and the question text into a large image-text model, and obtain the answer text output by the large image-text model; The large image-text model includes a visual expert module, a large language sub-model, and multiple small models of different task types; The input of each small model is the image to be processed and the question text, and the output is the first sub-image feature and the first sub-answer text; the visual expert module is used to determine the target task type according to the question text, and assign a weighted confidence to the output of the small model corresponding to the target task type that is higher than that of other small models; the large language sub-model is used to generate the answer text according to the image to be processed, the question text and the output of each small model with the weighted confidence assigned.

2. The method according to claim 1, characterized in that: The large image-text model also includes an image feature extraction module, which is used to extract the second sub-image features of the image to be processed. The large language sub-model is used to generate the answer text based on the image to be processed, the question text, the second sub-image features and the output of the weighted confidence of each small model.

3. The method according to claim 2, characterized in that The large image-text model further includes a cross-attention mechanism, which is used to concatenate the second sub-image feature and the first sub-image feature of each small model with a weighted reset confidence to generate a third sub-image feature; The cross attention mechanism is also used to concatenate the question text and the first sub-answer text of each small model with weighted reset confidence to generate a target text; The cross-attention mechanism is also used to concatenate the third sub-image features and the target text and input them into the large language sub-model.

4. The method according to any one of claims 1 to 3, characterized in that: The task types include at least two of the following: detection, segmentation, classification, and optical character recognition.

5. The method according to any one of claim 3, characterized in that: The large image-text model also includes an adapter, which is used to align the third sub-image features to the feature space of the large language sub-model to generate new third sub-image features.

6. The method according to any one of claims 1 to 3, characterized in that: Before obtaining the image to be processed and the question text corresponding to the image to be processed, the method further includes: Obtain multiple sample images and a sample question text and a sample answer text corresponding to each sample image; Training is performed based on each sample image and the corresponding sample question text and sample answer text to obtain a large image-text model.

7. An image-based question-answering device, characterized in that: include: An acquisition module, used for acquiring an image to be processed and a question text corresponding to the image to be processed; An input module, used for inputting the image to be processed and the question text into the image-text model; The acquisition module is further used to acquire the answer text output by the large picture and text model; The large image-text model includes a visual expert module, a large language sub-model, and multiple small models of different task types; The input of each small model is the image to be processed and the question text, and the output is the first sub-image feature and the first sub-answer text; the visual expert module is used to determine the target task type according to the question text, and assign a weighted confidence to the output of the small model corresponding to the target task type that is higher than that of other small models; the large language sub-model is used to generate the answer text according to the image to be processed, the question text and the output of each small model with the weighted confidence assigned.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which is used to implement the method according to any one of claims 1 to 6 when being executed by a processor.

Images