Visual question answering method, device, electronic device, storage medium and computer program product

Abstract

The present disclosure relates to a visual question answering method, apparatus, electronic device, storage medium, and computer program product. The method comprises the following steps: obtaining a target image and a target question for the target image; generating an initial program based on the target question; adding a target code for recording the program execution process to the initial program; inputting the target image into an extended program to obtain a predicted answer for the target question, execution process information of the extended program, and a screenshot image; and generating a multimodal explanation for the predicted answer based on the execution process information and the screenshot image. In this way, a decision-making basis image and a semantic association analysis can be generated simultaneously while outputting the predicted answer, that is, an explanation of the reasoning process of the predicted answer can be added to the output, so that the user can intuitively understand the correspondence between image features and semantic reasoning, thereby improving the transparency of reasoning and the credibility of decision-making, thereby facilitating the promotion and application of visual question answering technology.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more particularly, to a visual question answering method, apparatus, electronic device, storage medium, and computer program product. Background Art

[0002] Visual Question Answering (VQA) combines computer vision and natural language processing technologies to provide users with a more intelligent image understanding and interactive experience. Specifically, it enables computers to "see" images and answer questions about them. Given an image and a question about it, the visual question answering software can output the answer to the question based on the input image. This requires the visual question answering software to possess multimodal understanding capabilities, and currently, most mainstream methods rely on multimodal pre-training to achieve this. VQA technology can be applied in a variety of fields, including but not limited to medicine, education, surveillance, and entertainment.

[0003] However, the visual question answering system in related technologies can usually only provide the final answer, but lacks an explanation of the internal reasoning process, that is, the model transparency is low, which will lead to reduced user trust and is not conducive to the promotion and application of visual question answering technology. Summary of the Invention

[0004] The present disclosure provides a visual question answering method, apparatus, electronic device, storage medium, and computer program product to at least address the problem in the above-mentioned related technologies that visual question answering systems are usually only able to provide the final answer but lack an explanation of the internal reasoning process, resulting in low model transparency.

[0005] According to a first aspect of an embodiment of the present disclosure, a visual question answering method is provided, comprising: acquiring a target image and a target question for the target image; generating an initial program based on the target question, wherein the initial program is used to perform reasoning to obtain an answer to the target question; adding a target code for recording the program execution process to the initial program to obtain an extended program; inputting the target image into the extended program to obtain a predicted answer to the target question, execution process information of the extended program, and a screenshot image, wherein the screenshot image is a screenshot image associated with a target object pointed to by the target question and captured from the target image; generating a multimodal explanation for the predicted answer based on the execution process information and the screenshot image, wherein the multimodal form at least integrates a text form and an image form.

[0006] Optionally, generating an initial program based on the target problem includes: obtaining a preset program prompt, wherein the preset program prompt includes a structural thinking chain construction method and a program prompt example, the structural thinking chain construction method is used to indicate how to construct a corresponding structural thinking chain based on the problem, and the program prompt example is used to indicate how to generate a corresponding program based on the structural thinking chain; based on the structural thinking chain construction method, constructing a target structural thinking chain corresponding to the target problem; based on the program prompt example, generating the initial program corresponding to the target structural thinking chain.

[0007] Optionally, the preset program prompt further includes an application programming interface description, wherein the application programming interface description is used to indicate how to use the program to call the open world tool; inputting the target image into the extension program to obtain a predicted answer to the target question, execution process information of the extension program, and a screenshot image includes: executing the extension program, calling the open world tool based on the application programming interface description to process the target image, and obtaining the predicted answer, the execution process information, and the screenshot image.

[0008] Optionally, the open world tool includes the following items: an image target detector, wherein the image target detector is used to detect objects of a specified type in an image; an image cropping tool, wherein the image cropping tool is used to crop part or all of the area contained in the image; a customized Python function, wherein the customized Python function is used to perform at least one of a specific type of computing task, format conversion task, and logical judgment task to obtain the predicted answer and the execution process information.

[0009] Optionally, generating a multimodal explanation for the predicted answer based on the execution process information and the screenshot image includes: obtaining a preset explanation prompt example, wherein the preset explanation prompt example is used to indicate how to generate a corresponding natural language explanation based on the program execution process; generating a target text explanation in natural language for the predicted answer based on the preset explanation prompt example and the execution process information; and generating a multimodal explanation for the predicted answer based on the target text explanation and the screenshot image.

[0010] Optionally, generating a target text explanation in natural language for the predicted answer based on the preset explanation prompt example and the execution process information includes: inputting the preset explanation prompt example and the execution process information into a large language model to obtain a target text explanation in natural language for the predicted answer.

[0011] According to a second aspect of an embodiment of the present disclosure, a visual question-answering device is provided, comprising: an image and question acquisition module, configured to acquire a target image and a target question for the target image; an initial program generation module, configured to generate an initial program based on the target question, wherein the initial program is used to perform reasoning to obtain an answer to the target question; a program extension module, configured to add a target code for recording the program execution process to the initial program to obtain an extended program; an answer prediction module, configured to input the target image into the extended program to obtain a predicted answer to the target question, execution process information of the extended program, and a screenshot image, wherein the screenshot image is a screenshot image associated with a target object pointed to by the target question and captured from the target image; a multimodal explanation generation module, configured to generate a multimodal explanation for the predicted answer based on the execution process information and the screenshot image, wherein the multimodal form at least integrates a text form and an image form.

[0012] Optionally, the initial program generation module is configured to: obtain preset program prompts, wherein the preset program prompts include a structural thinking chain construction method and a program prompt example, the structural thinking chain construction method is used to indicate how to construct a corresponding structural thinking chain based on a problem, and the program prompt example is used to indicate how to generate a corresponding program based on the structural thinking chain; based on the structural thinking chain construction method, construct the target structural thinking chain corresponding to the target problem; based on the program prompt example, generate the initial program corresponding to the target structural thinking chain.

[0013] Optionally, the preset program prompt also includes an application programming interface description, wherein the application programming interface description is used to indicate how to use the program to call the open world tool; the answer prediction module is configured to: execute the extension program, call the open world tool based on the application programming interface description to process the target image, and obtain the predicted answer, the execution process information and the screenshot image.

[0014] Optionally, the open world tool includes the following items: an image target detector, wherein the image target detector is used to detect objects of a specified type in an image; an image cropping tool, wherein the image cropping tool is used to crop part or all of the area contained in the image; a customized Python function, wherein the customized Python function is used to perform at least one of a specific type of computing task, format conversion task, and logical judgment task to obtain the predicted answer and the execution process information.

[0015] Optionally, the multimodal explanation generation module is configured to: obtain a preset explanation prompt example, wherein the preset explanation prompt example is used to indicate how to generate a corresponding natural language explanation based on the program execution process; based on the preset explanation prompt example and the execution process information, generate a natural language target text explanation for the predicted answer; based on the target text explanation and the screenshot image, generate a multimodal explanation for the predicted answer.

[0016] Optionally, the multimodal explanation generation module is configured to: input the preset explanation prompt example and the execution process information into a large language model to obtain a natural language target text explanation for the predicted answer.

[0017] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to execute the instructions to implement the visual question answering method according to the present disclosure.

[0018] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the visual question answering method according to the present disclosure.

[0019] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which implements the visual question answering method according to the present disclosure when executed by a processor.

[0020] The technical solutions provided by the embodiments of the present disclosure bring at least the following beneficial effects:

[0021] In this disclosure, while outputting the predicted answer, a decision-making image and semantic association analysis can be generated simultaneously. This means that an explanation of the reasoning process for the predicted answer can be added to the output, allowing users to intuitively understand the correspondence between image features and semantic reasoning. This in turn improves reasoning transparency and decision credibility. This overcomes the shortcomings of low model transparency and user trust caused by the black box limitations of traditional visual question-answering models, thereby facilitating the promotion and application of visual question-answering technology. Furthermore, by outputting multimodal explanations for the predicted answer, the explanation is presented more clearly and intuitively, reducing the difficulty of understanding and making it easier for users to quickly and accurately understand the entire reasoning process.

[0022] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0024] Figure 1 is a flowchart illustrating a visual question answering method according to an exemplary embodiment of the present disclosure;

[0025] Figure 2 is an example diagram illustrating a target image and questions asked with respect to the target image according to an exemplary embodiment of the present disclosure;

[0026] Figure 3 is an example diagram illustrating a preset program prompt according to an exemplary embodiment of the present disclosure;

[0027] Figure 4 is an exemplary diagram illustrating an example of a preset explanation prompt according to an exemplary embodiment of the present disclosure;

[0028] Figure 5 is an example diagram illustrating a specific implementation process of a visual question answering method according to an exemplary embodiment of the present disclosure;

[0029] Figure 6 is a block diagram illustrating a visual question answering apparatus according to an exemplary embodiment of the present disclosure;

[0030] Figure 7 is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0032] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0033] It should be noted that the phrase "at least one of the several items" in this disclosure includes three types of parallel situations: "any one of the several items", "a combination of any multiple of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" means the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing both step 1 and step 2.

[0034] As mentioned above, existing visual question answering systems typically only provide a final answer, lacking explanations of the underlying reasoning process. This translates to low model transparency, which can lead to reduced user trust. Furthermore, current visual question answering systems also have shortcomings in example reliance, code generation flexibility, and external tool integration. In particular, when relying solely on a single example (one-shot), code generation is prone to logical redundancy and unstable execution.

[0035] In order to solve the above-mentioned problems existing in the related art, the visual question answering method, device, electronic device, storage medium and computer program product provided by the present disclosure can simultaneously generate a decision-making basis image and semantic association analysis while outputting the predicted answer, that is, it can increase the output of the explanation of the reasoning process of the predicted answer, so that the user can intuitively understand the correspondence between image features and semantic reasoning, thereby improving the transparency of reasoning and the credibility of decision-making, that is, it can overcome the defects of low model transparency and low user trust caused by the black box limitations of traditional visual question answering models, thereby facilitating the promotion and application of visual question answering technology. In addition, by outputting a multimodal explanation of the predicted answer, the display of the explanation is clearer and more intuitive, reducing the difficulty of understanding, and making it easier for users to quickly and accurately understand the entire reasoning process.

[0036] Figure 1 is a flowchart illustrating a visual question answering method according to an exemplary embodiment of the present disclosure.

[0037] Reference Figure 1 In step 101, a target image and a target question for the target image may be obtained. The target image and the target question may constitute multimodal question data, the content and format of which may be flexibly changed according to the actual application scenario.

[0038] Figure 2 : is an example diagram showing a target image and questions asked with respect to the target image according to an exemplary embodiment of the present disclosure. Figure 2, the target image can be a wild picture containing 4 puppies. And the question data related to the target image can be any question about the image content. Figure 2 In the example, the question data may be “What color is the dog on the far right?”. Furthermore, the question data may be automatically generated by the system or manually input. This disclosure does not impose any specific restrictions on the content, format, or generation method of the question data.

[0039] It should be noted that in the present disclosure, the visual question answering method according to the present disclosure can be executed by simply installing the visual question answering software on the terminal. For example, the terminal in the present disclosure can be, but is not limited to: a mobile phone, a tablet computer, a laptop computer, a wearable device, etc.

[0040] In step 102 , an initial program may be generated based on a target question, wherein the initial program is used to perform reasoning to obtain an answer to the target question.

[0041] According to an exemplary embodiment of the present disclosure, a preset program prompt can be obtained, wherein the preset program prompt can include a method for constructing a structural thinking chain and a program prompt example. The "structural thinking chain construction method" can be used to indicate how to construct a corresponding structural thinking chain based on a problem; the "program prompt example" can be used to indicate how to generate a corresponding program based on the structural thinking chain, that is, the program prompt example can demonstrate the corresponding relationship between the problem and the program. In addition, the preset program prompt in the present disclosure can be a single-sample program prompt.

[0042] Then, based on the structural thinking chain construction method, a target structural thinking chain corresponding to the target problem can be constructed. Next, based on the program prompt example, an initial program corresponding to the target structural thinking chain can be generated, that is, a structured initial logic program can be generated.

[0043] By setting up pre-set program prompts, the model used to generate the initial program can quickly learn the format of the program to be generated, thereby ensuring that the generated initial program meets format and logic requirements, while also improving program generation efficiency. Furthermore, by adopting a learning method based on single-sample program prompts, excessive reliance on large amounts of training data can be avoided, thereby improving the generalization ability of the model used to generate the initial program.

[0044] Furthermore, in the present disclosure, preset program prompts and the target question can also be input into the large language model, and then the large language model can generate and output the initial program. Exemplarily, the large language model can be, but not limited to: GPT-4o. The initial program may be composed of one or more program modules, each of which can correspond to a part of the logic in processing the target question. Specifically, the large language model can decompose the target question into several program steps, which can then be executed together with the target image to generate an answer to the question.

[0045] Figure 3 1 is an example diagram showing a preset program prompt according to an exemplary embodiment of the present disclosure. Figure 3 , the preset program prompts provide the writing format of question "Question", structural thinking chain "SCoT" and program "Program", and the large language model can generate the initial program based on this. Figure 2 As an example, let Figure 2 After the target question in the sentence "What color is the dog on the far right?" and the preset program prompts are input into the large language model, the large language model can first generate a structural thinking chain as follows:

[0046] “Structured Chain-of-Thought:

[0047] Input: image

[0048] Output: result: str

[0049] 1: Initialize ImageBox from the image: image_box

[0050] 2: Locate dog that exists definitely in image_box: dog_boxes

[0051] 3: If no dog is found in image_box:

[0052] 4:return "No dog found"

[0053] 5: Select the right dog found in image_box: right_dog

[0054] 6: Ask a VQA model what the color of the dog in right_dog: result

[0055] 7: Return result".

[0056] Next, the large language model can also generate the following initial program based on the above structural thinking chain:

[0057] “def inference(image):

[0058] image_crop = GetCrop(image)

[0059] dog_crops = image_crop.loc("dog")

[0060] if not dog_crops:

[0061] return "No dog found"

[0062] dog_crops.sort(key=lambda x: x.horizontal_center, reverse=True)

[0063] right_dog = dog_crops[0]

[0064] color = right_dog.simple_vqa("What color?")

[0065] return color".

[0066] In step 103, object code for recording the program execution process can be added to the initial program. That is, the initial program can be interpretably expanded to obtain an extended program. Specifically, the content of the initial program can be analyzed to understand the function of the program statements. Then, the code function of the initial program can be expanded to generate a data structure to store basic execution information during the program execution phase.

[0067] For example, Figure 2For example, the target codes added to the initial program for recording the program execution process may be: "image-crop.save()", "dog-crops.save()" and "right-dog.save()". Specifically, "image-crop.save()" is used to instruct to read and save the entire target image, that is, it is used to instruct to read and save the target image containing the four dogs; "dog-crops.save()" is used to instruct to save the partial image of each of the four dogs detected from the target image, refer to Figure 2 , Figure 2 The right side of shows a partial image of each of the four dogs; "right-dog.save()" is used to instruct to save the partial image of the dog located on the far right.

[0068] In step 104, the target image can be input into the extension program to obtain a predicted answer to the target question, information about the execution process of the extension program, and a screenshot image. The screenshot image can be a screenshot image captured from the target image and associated with the target object pointed to by the target question.

[0069] For example, Figure 2 For example, Figure 2 The target question shown in is: "What color is the dog on the far right?". At this time, the screenshot image can be a screenshot image that is associated with the target object pointed to by the target question "What color is the dog on the far right?": "the dog on the far right" and is captured from the target image. The execution process information can be expressed in the form of a dictionary, which can include the intermediate variable values and visual frame variable values of the extension program during the execution process. "Intermediate variable values" can include but are not limited to: how many times the extension program has executed the loop, from what position of the extension program the return is made, etc.; "visual frame variable values" can be the coordinate values of the area where the object is located detected from the target image. For example, still using Figure 2 For example, the “visual frame variable value” can be the coordinate value of the area where the dog is located detected from the target image.

[0070] According to an exemplary embodiment of the present disclosure, the preset program prompt may further include an application programming interface description, wherein the application programming interface description may be used to indicate how to use the program to call the open world tool, that is, the application programming interface description may show how to call various types of specific functions.

[0071] By executing the extension program, the open world tool can be called based on the application programming interface description to process the target image, and then the predicted answer, execution process information and screenshot image can be obtained. Specifically, in the process of executing the extension program, the open world multimodal tool can be used to convert the target problem into an execution process composed of predefined program modules. For example, the key visual targets in the target image can be detected and marked to ensure that the captured execution process data can fully reflect the reasoning process, and the intermediate variable values and the visual frame variable values can be recorded synchronously. For example, still with Figure 2 For example, after executing the extended program, the intermediate process data including all dog images and the individually extracted rightmost dog image can be obtained.

[0072] It's important to note that this open-world multimodal tool can typically be called as a program module. For example, the open-world object detection model GroundingDINO can be used for object detection. GroundingDINO is an open-vocabulary object detection (OVOD) model that combines DINO (a DETR-based detection architecture) with the Grounding mechanism to achieve text-based object detection.

[0073] According to an exemplary embodiment of the present disclosure, the above-mentioned open-world tool may include the following items: an image target detector, wherein the image target detector can be used to detect objects of a specified type in an image, that is, the image target detector can be used to identify and locate specific types of targets in an image; an image cropping tool, wherein the image cropping tool can be used to crop a specific area in an image, that is, the image cropping tool can be used to crop part of the area or all of the area contained in the image; a customized Python function, wherein the customized Python function can be used to execute at least one of a specific type of computing task, format conversion task, and logical judgment task to obtain a predicted answer and execution process information, and can automatically adjust the open-world tool parameters according to the detection results.

[0074] It should be noted that, in addition to the above-mentioned tools, open world tools may also include other types of tools, such as visual language models, etc. The above-mentioned implementation is merely an exemplary description.

[0075] In step 105 , a multimodal explanation for the predicted answer may be generated based on the execution process information and the screenshot image, wherein the multimodal form may be integrated with at least a text form and an image form.

[0076] According to an exemplary embodiment of the present disclosure, a preset explanation prompt example can be obtained, wherein the preset explanation prompt example can be used to indicate how to generate a corresponding natural language explanation based on the program execution process. Then, based on the preset explanation prompt example and the execution process information, a target text explanation for the predicted answer in natural language can be generated. Next, based on the target text explanation and the screenshot image, a multimodal explanation for the predicted answer can be generated.

[0077] Figure 4 1 is an example diagram showing an example of a preset explanation prompt according to an exemplary embodiment of the present disclosure. Figure 4 The preset explanation prompt example gives the writing format of the program execution process, which can be used to generate text explanation data.

[0078] According to an exemplary embodiment of the present disclosure, preset explanation prompt examples and execution process information can also be input into a large language model to obtain a natural language target text explanation for the predicted answer. Exemplarily, the large language model can be, but is not limited to, a GPT-40 model. Specifically:

[0079] The execution process information of the extension program and the single-sample explanation prompt example can also be input into the large language model, and then the large language model converts the execution process information into a coherent target text explanation according to the guidance of the single-sample explanation prompt example. The format of the target text explanation can be the same as the format of the single-sample explanation prompt example. In addition, the target text explanation can be composed of a specific data structure, for example, it can include but not be limited to: execution process parameter values, multimodal data names, and positioning box values. In addition, the original values of the visual box variables may not be directly presented in the target text explanation, but their positioning information may be reflected in the form of tags. Specifically, the target text explanation can be composed of natural language, and the positioning information can be embedded in the corresponding visual target name in the target text explanation in the form of tags.

[0080] For example, still Figure 2 For example, for the two variables of the visual target "rightmost dog (right_dog)" and "image crop area (image_crop)", the target text explanation can be embedded with the tag "[crop]" or "[box]", which can form the following example:

[0081] “Black. Because the dog [box] closest to the right in the image [box] is black.

[0082] Boxes:

[0083] BOX[0]=[10, 258, 206, 380]

[0084] BOX[1]=[0, 0, 250, 400]".

[0085] Among them, [10, 258, 206, 380] and [0, 0, 250, 400] can represent the coordinate positions of the corresponding image cropping areas. For example, a coordinate system can be established with the lower left corner of the entire target image as the origin. In this case, for [10, 258, 206, 380], "10" and "206" can represent the horizontal coordinate values of the lower left corner vertex and the lower right corner vertex of the image cropping area, respectively; "258" and "380" can represent the vertical coordinate values of the lower left corner vertex and the upper left corner vertex, respectively.

[0086] In this way, by adopting a single-sample explanation prompt example, not only can the format of the generated target text explanation be standardized, but also excessive reliance on a large amount of training data can be avoided, thereby improving the generalization ability of the model used to generate natural language text explanations for predicting answers.

[0087] Figure 5 1 is an example diagram showing a specific implementation process of the visual question answering method according to an exemplary embodiment of the present disclosure. Figure 5 The target question for the target image and the single-sample program prompt can be fed into the large language model. The large language model can then generate an initial program corresponding to the target question, guided by the single-sample program prompt. Several lines of code for recording the program execution process can then be added to the initial program to obtain an extended program.

[0088] Next, the target image can be input into the extension program, and then the extension program can call the open world tool to process the target image based on the application programming interface instructions contained in the above single sample program prompt to obtain the predicted answer, execution process information and screenshot image.

[0089] The predicted answer, execution process information, and pre-set single-sample explanation prompts can then be fed into the large language model to obtain a natural language explanation of the target text for the predicted answer. Next, the visual object can be embedded in the corresponding position of the target text explanation, resulting in an intuitive and easy-to-understand multimodal explanation.

[0090] This disclosure provides a visual question-answering method based on a single-example, code-driven large model. By introducing preset program prompts and explanation prompts, the model can not only provide predicted answers but also output multimodal explanations of the reasoning process without retraining the model. This improves the interpretability and credibility of the predicted answers, thereby enhancing model transparency and user trust.

[0091] Figure 6 is a block diagram illustrating a visual question answering apparatus 600 according to an exemplary embodiment of the present disclosure.

[0092] Reference Figure 6 The visual question answering device 600 may include an image and question acquisition module 601, an initial program generation module 602, a program expansion module 603, an answer prediction module 604 and a multimodal explanation generation module 605.

[0093] The image and question acquisition module 601 can acquire a target image and a target question for the target image. The target image and the target question can constitute multimodal question data, the content and format of which can be flexibly changed according to the actual application scenario.

[0094] It should be noted that in the present disclosure, the visual question answering method according to the present disclosure can be executed by simply installing the visual question answering software on the terminal. For example, the terminal in the present disclosure can be, but is not limited to: a mobile phone, a tablet computer, a laptop computer, a wearable device, etc.

[0095] The initial program generation module 602 may generate an initial program based on the target question, wherein the initial program is used to perform reasoning to obtain an answer to the target question.

[0096] According to an exemplary embodiment of the present disclosure, the initial program generation module 602 can obtain preset program prompts, wherein the preset program prompts can include a structured thinking chain construction method and a program prompt example. The "structured thinking chain construction method" can be used to indicate how to construct a corresponding structured thinking chain based on a problem; the "program prompt example" can be used to indicate how to generate a corresponding program based on the structured thinking chain, that is, the program prompt example can demonstrate the corresponding relationship between the problem and the program. In addition, the preset program prompt in the present disclosure can be a single-sample program prompt.

[0097] Then, the initial program generation module 602 can construct a target structural thinking chain corresponding to the target problem based on the structural thinking chain construction method. Next, the initial program generation module 602 can generate an initial program corresponding to the target structural thinking chain based on the program prompt example, that is, it can generate a structured initial logic program.

[0098] By setting up pre-set program prompts, the model used to generate the initial program can quickly learn the format of the program to be generated, thereby ensuring that the generated initial program meets format and logic requirements, while also improving program generation efficiency. Furthermore, by adopting a learning method based on single-sample program prompts, excessive reliance on large amounts of training data can be avoided, thereby improving the generalization ability of the model used to generate the initial program.

[0099] The program extension module 603 can add object code for recording the program execution process to the initial program, that is, the initial program can be interpretably expanded to obtain an extended program. Specifically, the content of the initial program can be analyzed to understand the program statement functions. Then, the code functions of the initial program can be expanded to generate a data structure to store basic execution information during the program execution phase.

[0100] The answer prediction module 604 may input the target image into the extension program to obtain a predicted answer to the target question, information about the execution process of the extension program, and a screenshot image. The screenshot image may be a screenshot image associated with the target object pointed to by the target question and obtained by intercepting the target image.

[0101] According to an exemplary embodiment of the present disclosure, the preset program prompt may further include an application programming interface description, wherein the application programming interface description may be used to indicate how to use the program to call the open world tool, that is, the application programming interface description may show how to call various types of specific functions.

[0102] By executing the extended program, the answer prediction module 604 can invoke open-world tools based on the API instructions to process the target image, thereby obtaining a predicted answer, execution process information, and a screenshot image. Specifically, during the execution of the extended program, the open-world multimodal tools can be used to convert the target question into an execution process composed of predefined program modules. For example, key visual objects in the target image can be detected and marked to ensure that the captured execution process data fully reflects the reasoning process, and the values of intermediate variables and visual frame variables can be simultaneously recorded.

[0103] It should be noted that this open-world multimodal tool can typically be called as a program module. For example, the open-world object detection model GroundingDINO can be used for object detection. GroundingDINO is an open-vocabulary object detection model that combines DINO with the Grounding mechanism to achieve text-based object detection.

[0104] According to an exemplary embodiment of the present disclosure, the above-mentioned open-world tool may include the following items: an image target detector, wherein the image target detector can be used to detect objects of a specified type in an image, that is, the image target detector can be used to identify and locate specific types of targets in an image; an image cropping tool, wherein the image cropping tool can be used to crop a specific area in an image, that is, the image cropping tool can be used to crop part of the area or all of the area contained in the image; a customized Python function, wherein the customized Python function can be used to execute at least one of a specific type of computing task, format conversion task, and logical judgment task to obtain a predicted answer and execution process information, and can automatically adjust the open-world tool parameters according to the detection results.

[0105] It should be noted that, in addition to the above-mentioned tools, open world tools may also include other types of tools, such as visual language models, etc. The above-mentioned implementation is merely an exemplary description.

[0106] The multimodal explanation generating module 605 may generate a multimodal explanation for the predicted answer based on the execution process information and the screenshot image, wherein the multimodal form may at least integrate a text form and an image form.

[0107] According to an exemplary embodiment of the present disclosure, the multimodal explanation generation module 605 can obtain a preset explanation prompt example, wherein the preset explanation prompt example can be used to indicate how to generate a corresponding natural language explanation based on the program execution process. Then, the multimodal explanation generation module 605 can generate a natural language target text explanation for the predicted answer based on the preset explanation prompt example and the execution process information. Next, the multimodal explanation generation module 605 can generate a multimodal explanation for the predicted answer based on the target text explanation and the screenshot image.

[0108] According to an exemplary embodiment of the present disclosure, the multimodal explanation generation module 605 may also input preset explanation prompt examples and execution process information into a large language model to obtain a natural language target text explanation for the predicted answer. Exemplarily, the large language model may be, but is not limited to, a GPT-40 model. Specifically:

[0109] The multimodal interpretation generation module 605 can also input the execution process information of the extension program and the single sample interpretation prompt example into the large language model, and then the large language model converts the execution process information into a coherent target text interpretation according to the guidance of the single sample interpretation prompt example. The format of the target text interpretation can be the same as the format of the single sample interpretation prompt example. In addition, the target text interpretation can be composed of a specific data structure, for example, it can include but is not limited to: execution process parameter values, multimodal data names, and positioning box values. In addition, the original values of the visual box variables can be not directly presented in the target text interpretation, but their positioning information can be reflected in the form of tags. Specifically, the target text interpretation can be composed of natural language, and the positioning information can be embedded in the corresponding visual target name in the target text interpretation in the form of tags.

[0110] In this way, by adopting a single-sample explanation prompt example, not only can the format of the generated target text explanation be standardized, but also excessive reliance on a large amount of training data can be avoided, thereby improving the generalization ability of the model used to generate natural language text explanations for predicting answers.

[0111] Figure 7 is a block diagram illustrating an electronic device 700 according to an exemplary embodiment of the present disclosure.

[0112] Reference Figure 7 The electronic device 700 includes at least one memory 701 and at least one processor 702. The at least one memory 701 stores instructions. When the instructions are executed by the at least one processor 702, the visual question answering method according to the exemplary embodiment of the present disclosure is executed.

[0113] As an example, electronic device 700 may be a PC, tablet device, personal digital assistant, smartphone, or other device capable of executing the aforementioned instructions. Here, electronic device 700 is not necessarily a single electronic device, but may also be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 700 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device that interfaces with local or remote devices (e.g., via wireless transmission).

[0114] In electronic device 700, processor 702 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, the processor may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, and the like.

[0115] The processor 702 can execute instructions or codes stored in the memory 701, wherein the memory 701 can also store data. Instructions and data can also be sent and received over the network via the network interface device, wherein the network interface device can use any known transmission protocol.

[0116] The memory 701 may be integrated with the processor 702, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, the memory 701 may comprise a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The memory 701 and the processor 702 may be operatively coupled or may communicate with each other, for example, via an I / O port, a network connection, or the like, such that the processor 702 can access files stored in the memory.

[0117] In addition, the electronic device 700 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 700 may be connected to each other via a bus and / or a network.

[0118] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium may also be provided, which, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, enables the electronic device to perform the above-mentioned visual question answering method. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as a multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device configured to store a computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0119] According to an exemplary embodiment of the present disclosure, a computer program product may further be provided, including a computer program, wherein when the computer program is executed by a processor, the visual question answering method according to the present disclosure is implemented.

[0120] According to the visual question answering method, device, electronic device, storage medium and computer program product disclosed in the present invention, a decision-making basis image and semantic association analysis can be generated simultaneously while outputting the predicted answer. That is, the explanation of the reasoning process of the predicted answer can be increased, so that the user can intuitively understand the correspondence between image features and semantic reasoning, thereby improving the transparency of reasoning and the credibility of decision-making. That is, the defects of low model transparency and low user trust caused by the black box limitations of traditional visual question answering models can be overcome, thereby facilitating the promotion and application of visual question answering technology. In addition, by outputting a multimodal explanation of the predicted answer, the display of the explanation is made clearer and more intuitive, reducing the difficulty of understanding and making it easier for users to quickly and accurately understand the entire reasoning process.

[0121] According to exemplary embodiments of the present disclosure, by setting preset program prompts, the model used to generate the initial program can quickly learn the format of the program to be generated, thereby ensuring that the generated initial program meets format and logic requirements, while also improving program generation efficiency. Furthermore, by adopting a learning method based on single-sample program prompts, excessive reliance on large amounts of training data can be avoided, thereby improving the generalization ability of the model used to generate the initial program.

[0122] According to an exemplary embodiment of the present disclosure, by adopting a single-sample explanation prompt example, not only can the format of the generated target text explanation be standardized, but also excessive reliance on a large amount of training data can be avoided, thereby improving the generalization ability of the model for generating natural language text explanations for predicting answers.

[0123] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0124] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A visual question answering method, characterized in that: include: Obtaining a target image and a target question for the target image; generating an initial program based on the target problem, wherein the initial program is used to perform reasoning to obtain an answer to the target problem; Adding object code for recording the program execution process to the initial program to obtain an extended program; Inputting the target image into the extended program to obtain a predicted answer to the target question, execution process information of the extended program, and a screenshot image, wherein the screenshot image is a screenshot image captured from the target image and is associated with a target object pointed to by the target question; generating a multimodal explanation for the predicted answer based on the execution process information and the screenshot image, wherein the multimodal explanation comprises at least a text form and an image form; The step of generating an initial program based on the target problem includes: Obtaining a preset program prompt, wherein the preset program prompt includes a structured thinking chain construction method and a program prompt example, wherein the structured thinking chain construction method is used to indicate how to construct a corresponding structured thinking chain based on a problem, and the program prompt example is used to indicate how to generate a corresponding program based on the structured thinking chain, and the preset program prompt is a single-sample program prompt; Based on the structural thinking chain construction method, construct a target structural thinking chain corresponding to the target problem; Based on the program prompt example, the initial program corresponding to the target structure thinking chain is generated.

2. The visual question answering method according to claim 1, wherein: The preset program prompt further includes an application programming interface description, wherein the application programming interface description is used to indicate how to use the program to call the open world tool; Inputting the target image into the extension program to obtain a predicted answer to the target question, execution process information of the extension program, and a screenshot image includes: By executing the extension program, the open world tool is called based on the application programming interface description to process the target image, thereby obtaining the predicted answer, the execution process information and the screenshot image.

3. The visual question answering method according to claim 2, wherein: The open world tools include the following: An image object detector, wherein the image object detector is used to detect objects of a specified type in an image; An image cropping tool, wherein the image cropping tool is used to crop a portion or the entire area contained in an image; A customized Python function, wherein the customized Python function is used to perform at least one of a specific type of computing task, format conversion task, and logical judgment task to obtain the predicted answer and the execution process information.

4. The visual question answering method according to claim 1, wherein: Generating a multimodal explanation for the predicted answer based on the execution process information and the screenshot image includes: Obtaining a preset explanation prompt example, wherein the preset explanation prompt example is used to indicate how to generate a corresponding natural language explanation based on a program execution process; Based on the preset explanation prompt example and the execution process information, generating a target text explanation in natural language for the predicted answer; Based on the target text explanation and the screenshot image, a multimodal explanation for the predicted answer is generated.

5. The visual question answering method according to claim 4, wherein: The generating of a target text explanation in natural language for the predicted answer based on the preset explanation prompt example and the execution process information includes: The preset explanation prompt example and the execution process information are input into a large language model to obtain a target text explanation in natural language for the predicted answer.

6. A visual question-answering device, characterized in that: include: An image and question acquisition module, configured to acquire a target image and a target question for the target image; an initial program generation module, configured to generate an initial program based on the target problem, wherein the initial program is used to perform reasoning to obtain an answer to the target problem; A program extension module is configured to add object code for recording the program execution process to the initial program to obtain an extended program; an answer prediction module configured to input the target image into the extension program, obtain a predicted answer to the target question, execution process information of the extension program, and a screenshot image, wherein the screenshot image is a screenshot image associated with a target object pointed to by the target question and obtained by intercepting the target image; a multimodal explanation generating module configured to generate a multimodal explanation for the predicted answer based on the execution process information and the screenshot image, wherein the multimodal form at least integrates a text form and an image form; Wherein, the initial program generation module is specifically configured as follows: Obtaining a preset program prompt, wherein the preset program prompt includes a structured thinking chain construction method and a program prompt example, wherein the structured thinking chain construction method is used to indicate how to construct a corresponding structured thinking chain based on a problem, and the program prompt example is used to indicate how to generate a corresponding program based on the structured thinking chain, and the preset program prompt is a single-sample program prompt; Based on the structural thinking chain construction method, construct a target structural thinking chain corresponding to the target problem; Based on the program prompt example, the initial program corresponding to the target structure thinking chain is generated.

7. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the visual question answering method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform the visual question answering method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the visual question answering method according to any one of claims 1 to 5 is implemented.

Images