Visual question and answer data enhancement method and device, equipment and storage medium

Abstract

The invention provides a visual question and answer data enhancement method and device, equipment and a storage medium, and belongs to the technical field of artificial intelligence. Extracting a target noun used for describing an object in the original image from the original question, inputting the target noun into a large language model, generating a plurality of new question and answer pairs related to the target noun through the large language model, and combining the new question and answer pairs and the original question and answer pairs to obtain an enhanced text prompt; performing target detection on the original image according to the text prompt after data enhancement to obtain a bounding box of an object corresponding to the text prompt, and performing segmentation extraction on the image based on the bounding box to obtain a segmentation mask image; and redrawing the original image according to the segmented mask image and the text prompt after data enhancement to obtain an enhanced image. According to the method, visual question and answer data with higher quality can be obtained, so that the accuracy of a visual question and answer model is improved, and the language deviation problem in VQA is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a visual question and answer data enhancement method, device, equipment and storage medium. Background Art

[0002] Visual Question Answering (VQA) is a complex task that requires visual processing and reasoning. Given an image and a question related to the image, the visual question answering model predicts the answer. Visual perception and understanding tasks in open-world scenarios are critical to the advancement of applications such as autonomous driving, robot navigation, and intelligent security monitoring.

[0003] Data augmentation (DA) is a technique that improves the performance of machine learning models by artificially increasing the diversity and quantity of training data. It is widely used in the training process of visual question-answering models. Data augmentation can increase the diversity of data, improve the generalization ability of visual question-answering models, and enhance the robustness of models. Currently, a mainstream data augmentation method includes using generative adversarial networks to generate counterfactual and adversarial images; editing some words in the original question, reverse translation, or random combination of image-text pair samples. These methods result in poor quality of the enhanced data set due to factors such as unrealistic synthetic data, data imbalance, and randomness, thus affecting the accuracy of the visual question-answering model. Summary of the invention

[0004] In order to solve the problem of low accuracy of visual question answering models caused by poor quality of visual question answering data in the prior art, the present invention provides a visual question answering data enhancement method, device, equipment and storage medium.

[0005] In order to achieve the above object, the present invention provides the following technical solutions: A visual question answering data augmentation method, comprising: Obtain the original dataset of the visual question answering task, which includes the original image and original question-answer pairs; Extract the target noun used to describe the object in the original image from the original question, input the target noun into the large language model, generate multiple new question-answer pairs related to the target noun through the large language model, and merge the new question-answer pairs with the original question-answer pairs to obtain enhanced text prompts; According to the data-enhanced text prompt, target detection is performed on the original image to obtain the bounding box of the object corresponding to the text prompt, and the image is segmented and extracted based on the bounding box to obtain a segmentation mask image; The original image is redrawn according to the segmentation mask image and the text prompt after data enhancement to obtain the enhanced image.

[0006] Optionally, according to the data-enhanced text prompt, target detection is performed on the original image to obtain a bounding box of an object corresponding to the text prompt, and the image is segmented and extracted based on the bounding box to obtain a segmentation mask image, including: The original image and the enhanced text prompt are input into the Groungding DINO model to obtain the object bounding box, object label and label confidence corresponding to the text prompt; Annotate the original image based on object bounding boxes, object labels, and label confidence; The annotated image is input into the SAM model for segmentation, and the segmented object mask and background mask are drawn on the annotated image.

[0007] Optionally, the image is redrawn according to the segmentation mask image and the data-enhanced text prompt to obtain an enhanced image, including: The original image, segmentation mask image, and enhanced text prompt are input into the trained Stable Diffusion model to generate the redrawn image through a forward pass.

[0008] Optionally, it also includes: Based on the enhanced text prompts and enhanced images, a visual question answering model is built, including: Inputting the text query in the enhanced text prompt into the large language model, and dynamically building the initial visual question answering model based on the text query through the large language model; Execute the program code of the initial visual question answering model to recognize the enhanced image and output the query result; Evaluate the query results based on the target answer to generate feedback, and optimize the program code of the initial visual question answering model based on the feedback; The process of iterative feedback and optimization is continued until the preset stopping condition is reached to obtain the target visual question answering model.

[0009] Optionally, program code for dynamically building an initial visual question answering model based on text queries through a large language model includes: Get the predefined API specification and input the API specification and text query into the big language model; Generate Python functions based on API specifications and text queries through a large language model.

[0010] The present invention also provides a visual question answering data enhancement device, comprising: An acquisition module is used to obtain the original data set of the visual question answering task, where the original data set includes the original image and the original question answer pair; The text enhancement module is used to extract the target noun used to describe the object in the original image from the original question, input the target noun into the large language model, generate multiple new question-answer pairs related to the target noun through the large language model, and merge the new question-answer pairs with the original question-answer pairs to obtain enhanced text prompts; The image detection and segmentation module is used to detect the target in the original image according to the text prompt after data enhancement, obtain the bounding box of the object corresponding to the text prompt, and segment and extract the image based on the bounding box to obtain the segmentation mask image; The image enhancement module is used to redraw the original image according to the segmentation mask image and the text prompt after data enhancement to obtain an enhanced image.

[0011] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned visual question answering data enhancement method when executing the program.

[0012] The present invention also provides a computer-readable storage medium, wherein the storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned visual question answering data enhancement method is implemented.

[0013] The visual question answering data enhancement method provided by the present invention has the following beneficial effects: The present invention extracts the target noun used to describe the object in the original image in the original question, inputs the target noun into the large language model, generates multiple new question-answer pairs related to the target noun through the large language model, merges the new question-answer pairs with the original question-answer pairs to obtain enhanced text prompts; redraws the image based on the data-enhanced question-answer pairs as text prompts; because the large language model has a strong contextual understanding ability and can understand complex text descriptions and semantic information, when used to guide image generation, this understanding ability enables the model to more accurately capture the details and context in the text description, thereby generating a more realistic and description-compliant image. Therefore, the present invention can obtain higher quality data by performing data enhancement based on the large language model in visual question answering, thereby improving the accuracy of the visual question answering model and reducing the language bias problem in VQA. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiment of the present invention and its design scheme, the following briefly introduces the drawings required for this embodiment. The drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 A schematic flow chart of a visual question answering data enhancement method provided by an embodiment of the present invention; Figure 2 A structural schematic diagram of a Groungding DINO model provided in an embodiment of the present invention; Figure 3 A schematic diagram of the structure of a SAM model provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a visual question answering data enhancement device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0016] In order to enable those skilled in the art to better understand the technical solution of the present invention and implement it, the present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the scope of protection of the present invention.

[0017] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the technical solutions of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0018] In addition, the terms "first", "second", etc. are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless otherwise clearly specified or limited, the terms "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances. In the description of the present invention, unless otherwise specified, "plurality" means two or more, which will not be described in detail here.

[0019] Based on the problem that the quality of visual question answering datasets in the prior art is poor, which leads to low accuracy of visual question answering models, this paper proposes a visual question answering data enhancement method, such as Figure 1 As shown, the method comprises the following steps: S1: Get the original dataset for the vision task.

[0020] Among them, the original data set includes the original image, the question corresponding to the original image, and the answer corresponding to the original question. The question-answer pair consisting of the original question and the original answer belongs to the text prompt of the image.

[0021] S2, extracts the target noun used to describe the object in the original image in the original question, inputs the target noun into the large language model, generates multiple new question-answer pairs related to the target noun through the large language model, and merges the new question-answer pairs with the original question-answer pairs to obtain enhanced text prompts.

[0022] It should be understood that in the present invention, the process of using a large language model to expand the original data set based on the text prompts in the original data set to obtain data-enhanced text prompts is a data enhancement process, that is, generating additional training samples outside the original data set.

[0023] For example, taking the VQA dataset as the original dataset, the original dataset includes about 265,016 images, the number of questions involved in each image is greater than five on average, each question contains 10 benchmark truths, and each question contains 3 reasonable answers.

[0024] For each image, questions are processed. Examples of question samples are as follows: {"image_id":131093,"question":"Are there rocks in the picture?","question_id": 131093000}, {"image_id": 131093, "question": "Has the sheep recently been shaved?", "question_id": 131093001}, {"image_id": 131093, "question": "How many sheepsare this?", "question_id": 131093002}, {"image_id": 131093, "question": "Arethere three sheep?", "question_id": 131093003}.

[0025] Then, a large language model (such as ChatGPT, etc.) is used to extract meaningful target nouns used to describe objects in the image based on the original question, ignoring "picture" or nouns that have no meaning, and obtaining the target nouns in the image. The large language model generates new question-answer pair samples based on the target nouns and reasonable text prompts (such as "describe the objects in the image"). The new question-answer pairs will closely revolve around the target nouns in the image to ensure the accuracy and relevance of the content. Optionally, one target noun generates three new samples as counterfactual samples.

[0026] In the above embodiment, the excellent text understanding ability of the large language model is utilized to avoid the process of extracting meaningful nouns and unifying their singular and plural forms using traditional methods such as spaCy POS tagger. At the same time, there is no need to traverse all questions and images in the original training set to compose all possible reasonable VQAs.

[0027] Optionally, the large language model may select GPT-3.5-turbo API, GPT4 or Claude3, which is not specifically limited in the embodiment of the present invention.

[0028] S3: According to the data-enhanced text prompt, target detection is performed on the original image to obtain the bounding box of the object corresponding to the text prompt, and the image is segmented and extracted based on the bounding box to obtain a segmentation mask image.

[0029] In the embodiment of the present invention, target detection is performed by the Groungding DINO model, and image segmentation is performed by the Segment Anything Model (SAM) model. The implementation process of S2 is as follows: A1: Create a Groungding DINO model instance and a SAM model instance.

[0030] Specifically, create a Groungding DINO model instance, download the configuration file of the model weights, and use the SLConfig method of GroundingDINO to read the configuration file. Use PyTorch's load function to load the model file from disk to memory, and then load the model state dictionary obtained from the model weights into the model. Before that, clean the weights, that is, keep the user data and do not change the original weights, and set the model to eval, that is, turn off some features used by the model during training, such as Dropout, BatchNorm, etc.

[0031] Then, create a SAM model instance and load the sam_vit_h model weights. The model weights include vit_h, vit_l, and vit_b, which represent the number of image encoder parameters. Generally, the above performance increases with the increase of the number of parameters, but smaller image encoders perform better under certain experimental settings. Those skilled in the art can choose different vit model weights based on actual experimental results. Use the SAM native SamPredictor class and its methods to create a model instance.

[0032] A2: Input the original image and the enhanced text prompt into the Groungding DINO model, run the GroungdingDINO model instance, perform object detection on the image according to the data-enhanced text prompt, and obtain the object bounding box, object label, and label confidence corresponding to the text prompt.

[0033] Figure 2 A structural schematic diagram of a Groungding DINO model provided in an embodiment of the present invention.

[0034] For example, Figure 2 As shown in the figure, the text backbone encodes the input text prompt and extracts the text features. The image backbone encodes the input image and extracts the image features. The feature enhancer fuses the extracted text features and image features. In the feature enhancer, the association between cross-modal features can be enhanced through self-attention mechanisms, image-to-text cross-attention, and text-to-image cross-attention mechanisms to improve the accuracy of target detection. Through the designed language-guided query selection mechanism, image features that are more relevant to the text description are selected as the query of the decoder. The selected query is decoded using the cross-modal decoder, and the target detection result is output to obtain the object bounding box. , object label And the label confidence value .

[0035] Furthermore, based on the object bounding box , label confidence , object label , for the original image Annotate and get the image after adding annotations The above process can be simply expressed as follows: ; ; Among them, the predict function and annotate function are implemented internal methods.

[0036] A3: Input the annotated image into the SAM model instance for segmentation, and draw the segmented object mask and background mask on the annotated image.

[0037] Figure 3 A schematic diagram of the structure of a SAM model provided in an embodiment of the present invention.

[0038] For example, Figure 3 As shown in the figure, the annotated image is input, the annotated image is converted from BGR to RGB, the image is converted into an image embedding through the image encoder, the prompt encoder converts the prompt (point prompt, box prompt and text prompt) into a prompt embedding, and the mask decoder combines the image embedding and the prompt embedding. Through a series of operations (such as self-attention, cross attention, etc.), the mask decoder calculates the probability of each pixel in the image belonging to the foreground or background, and generates the final segmentation mask image based on these probabilities. In the mask image, the object area is usually marked as the foreground, and the background area is marked as the background.

[0039] It should be understood that before the bounding box is input into the SAM model, the format of the bounding box is converted from (center_x, center_y, width, height) to (xmin, ymin, xmax, ymax) and a Tensor operation is performed to adapt to the input of the SAM model.

[0040] Furthermore, after obtaining the mask image, the original image and the mask image are converted into RGBA format to handle transparency, and the mask image is superimposed on the original image. Because there are two ways to redraw the specified object target and the fixed object to redraw the external environment, the reverse image of the mask is then obtained.

[0041] S4, redrawing the original image according to the segmentation mask image and the text prompt after data enhancement to obtain an enhanced image.

[0042] In the present invention, a pre-trained Stable Diffusion model is used to perform image redrawing.

[0043] Specifically, first load the pre-trained Stable Diffusion model, build the pipeline, input parameters (including image, mask image, text prompt after data enhancement), etc., resize the image to fit the model input, set the random seed, and the model generates a Vincent image (the image after redrawing) through forward pass.

[0044] In the above embodiment, based on the data enhancement of the large language model, the generated images are more natural and close to reality, and the image-text pairs are also expanded and enriched through the large language model, which solves the current problems of difficulty in obtaining high-quality data, high labor costs, inaccurate data, and language bias problems in VQA tasks.

[0045] After the above data enhancement, model training is performed based on the data enhancement dataset to build a visual question answering model that can be used for visual question answering tasks.

[0046] In the present invention, a large language model is used to dynamically generate the program code of the visual question answering model to solve the VQA task. The implementation process is as follows: B1: The program code for inputting the text query in the enhanced text prompt into the big language model and dynamically building the initial visual question answering model based on the text query through the big language model.

[0047] B2: Execute the program code of the initial visual question answering model to recognize the enhanced image and output the query result.

[0048] For example, given a visual input (Image) and text queries , while the prompt consists of an application programming interface (API), which is provided to the large language model as part of the input context. The final input of the large language model is a series of code texts, which contains the API specification, followed by the text query. That is, through the program generation module Search by text Generate Python program code from API specifications .

[0049] Specifically, the API specification covers different perception and knowledge modules, such as object detection, depth estimation, or language model query. The API defines two global classes, ImagePatch and VideoSegment, representing image parts and video segments, respectively. Each module is implemented as a class method that internally calls a pre-trained model to compute the result. For example, the compute_depth method of ImagePatch returns an estimate of the median depth of the pixels in the image patch. The API specifies the input and output types for each method it defines, as well as documentation strings that explain the purpose of these functions in natural language. Like most APIs, it also provides examples to show how to use these classes and their functions, similar to contextual learning.

[0050] Then the application execution module Based on visual input implement , and get the output (text or image), where Represents the combination of Python interpreter and pre-trained model, Python interpreter supports logical operations, while pre-trained model supports perception operations. The program runs with Python interpreter, so its execution is a simple Python call and can use all built-in Python functions such as sorting, date and time, control flow if / else; while pre-trained model can implement perception modules in previous API such as object detection.

[0051] B3: Program code for evaluating query results based on the target answer to generate feedback and optimizing the initial visual question answering model based on the feedback.

[0052] B4: Iterate the feedback and optimization process until the preset stopping condition is reached to obtain the target visual question answering model.

[0053] For example, for the above visual question answering problem, given the visual input (image or video), prompt (API specification plus text query) and large language model to generate initial code program , and the initial query results , the same Represents the combination of Python interpreter and pre-trained model. The formula is as follows: ; .

[0054] Since there is no guarantee that the initial code and results It is always accurate, so the execution results of each step in the code and the final return are used to generate feedback, and the original initial code is modified according to the error . Feedback types include visual feedback , Text Feedback , and error feedback For visual feedback, for example, to detect whether there is an apple in an image, the image block containing the apple in the intermediate process will be input into the visual language model to verify whether it contains an apple and provide visual feedback ; For text feedback, use the language model to verify whether the intermediate variables are consistent with the final results There are logical errors; for error feedback, such as initial code There are semantic or syntactic errors that cause compilation failure, provide error feedback .

[0055] Using the same large language model Input visual feedback , Text Feedback , and error feedback And the initial code program and initial prompt (API specification plus text query), output modified code program , and obtain new results , the formula for this step is as follows: ; .

[0056] Iterate the feedback and optimization process until the preset stop condition is reached, setting the number of iterations to , the formula is as follows: ; .

[0057] It should be understood that the stopping condition in practice depends on the specific task and experimental results.

[0058] In the above embodiment, a large language model is used to dynamically construct program code according to the task text query, and an image or video is used as a parameter to return the query result of the image or video, thereby achieving excellent performance on zero samples. By iterative feedback and optimization of the large language model itself, the accuracy of the code generated by the large language model can be improved, and a more accurate visual question answering model can be obtained.

[0059] Based on the same inventive concept, the embodiment of the present invention also provides a visual question answering data enhancement device, such as Figure 4 As shown, the device comprises: The acquisition module 41 is used to acquire the original data set of the visual question answering task, where the original data set includes the original image and the original question-answer pair.

[0060] The text enhancement module 42 is used to extract the target noun in the original question for describing the object in the original image, input the target noun into the large language model, generate multiple new question-answer pairs related to the target noun through the large language model, and merge the new question-answer pairs with the original question-answer pairs to obtain enhanced text prompts.

[0061] The image detection and segmentation module 43 is used to perform target detection on the original image according to the text prompt after data enhancement, obtain the bounding box of the object corresponding to the text prompt, and segment and extract the image based on the bounding box to obtain a segmentation mask image.

[0062] The image enhancement module 44 is used to redraw the original image according to the segmentation mask image and the text prompt after data enhancement to obtain an enhanced image.

[0063] Each module in the above-mentioned visual question answering data enhancement device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0064] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the visual question answering data enhancement method embodiment. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0065] Furthermore, the present invention also provides a non-temporary computer-readable storage medium containing instructions, and a computer program is stored on the storage medium. For example, a memory containing instructions, the above instructions can be executed by a processor of a computer device to complete the above method. For example, a non-temporary computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a tape, a floppy disk, and an optical data storage device. When the computer program is executed by the processor, the steps in the embodiment of the visual question answering data enhancement method can be implemented. The specific implementation method can be found in the method embodiment, which will not be repeated here.

[0066] It should be understood by those skilled in the art that embodiments of the present invention may provide methods, systems or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] The present invention is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as a combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0068] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0069] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0070] It should be pointed out that the specific implementation methods described above can enable those skilled in the art to understand the invention more comprehensively, but do not limit the invention in any way. Therefore, although the invention has been described in detail in this specification and embodiments, those skilled in the art should understand that the invention can still be modified or replaced by equivalents; and all technical solutions and improvements that do not deviate from the spirit and scope of the invention are included in the protection scope of the invention. Any figure mark in the claims should not be regarded as limiting the claims involved. Any simple change or equivalent replacement of the technical solution that can be obviously obtained by any technician familiar with the field within the technical scope disclosed in the present invention belongs to the protection scope of the present invention.

Claims

1. A visual question answering data augmentation method, It is characterized in that include: Obtain the original dataset of the visual question answering task, which includes the original images and the original question-answer pairs; Extract the target noun used to describe the object in the original image from the original question, input the target noun into the large language model, generate multiple new question-answer pairs related to the target noun through the large language model, and merge the new question-answer pairs with the original question-answer pairs to obtain enhanced text prompts; According to the data-enhanced text prompt, target detection is performed on the original image to obtain the bounding box of the object corresponding to the text prompt, and the image is segmented and extracted based on the bounding box to obtain a segmentation mask image; The original image is redrawn according to the segmentation mask image and the text prompt after data enhancement to obtain the enhanced image.

2. The method for enhancing visual question answering data according to claim 1, It is characterized in that According to the data-enhanced text prompt, the original image is detected to obtain the bounding box of the object corresponding to the text prompt. The image is segmented and extracted based on the bounding box to obtain a segmentation mask image, including: The original image and the enhanced text prompt are input into the Groungding DINO model to obtain the object bounding box, object label and label confidence corresponding to the text prompt; Annotate the original image based on object bounding boxes, object labels, and label confidence; The annotated image is input into the SAM model for segmentation, and the segmented object mask and background mask are drawn on the annotated image.

3. The visual question answering data enhancement method according to claim 1 or 2, It is characterized in that Redraw the image according to the segmentation mask image and the data-enhanced text prompt to obtain an enhanced image, including: The original image, segmentation mask image, and enhanced text prompt are input into the trained Stable Diffusion model to generate the redrawn image through a forward pass.

4. The visual question answering data enhancement method according to claim 1 or 2, It is characterized in that Also includes: Based on the enhanced text prompts and enhanced images, a visual question answering model is built, including: Inputting the text query in the enhanced text prompt into the large language model, and dynamically building the initial visual question answering model based on the text query through the large language model; Execute the program code of the initial visual question answering model to recognize the enhanced image and output the query result; Evaluate the query results based on the target answer to generate feedback, and optimize the program code of the initial visual question answering model based on the feedback; The process of iterative feedback and optimization is continued until the preset stopping condition is reached to obtain the target visual question answering model.

5. The method for enhancing visual question answering data according to claim 4, It is characterized in that The program code for dynamically building an initial visual question answering model based on text queries through a large language model includes: Get the predefined API specification and input the API specification and text query into the big language model; Generate Python functions based on API specifications and text queries through a large language model.

6. A visual question answering data enhancement device, It is characterized in that include: An acquisition module is used to obtain the original data set of the visual question answering task, where the original data set includes the original image and the original question answer pair; The text enhancement module is used to extract the target noun used to describe the object in the original image from the original question, input the target noun into the large language model, generate multiple new question-answer pairs related to the target noun through the large language model, and merge the new question-answer pairs with the original question-answer pairs to obtain enhanced text prompts; The image detection and segmentation module is used to detect the target in the original image according to the text prompt after data enhancement, obtain the bounding box of the object corresponding to the text prompt, and segment and extract the image based on the bounding box to obtain the segmentation mask image; The image enhancement module is used to redraw the original image according to the segmentation mask image and the text prompt after data enhancement to obtain an enhanced image.

7. A computer device, It is characterized in that The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method for enhancing visual question answering data according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium, It is characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the visual question answering data enhancement method according to any one of claims 1 to 5 is implemented.

Images