Visual question and answer model training and application method, device and equipment

Abstract

The invention provides a visual question and answer model training and application method, device and equipment, and the visual question and answer model training method comprises the steps: determining a first output image through a diffusion model based on a first input image and a first visual prompt instruction of the first input image; determining an instruction learning target of a diffusion model based on the first input image and the first output image; and training a diffusion model based on the first input image, the first output image, the first visual prompt instruction and the instruction learning target to obtain a pre-trained diffusion model. According to the method, the instruction learning target is used as the learning guidance of the diffusion model, so that the diffusion model can understand the first visual prompt instruction through training, the pre-training model can generate the output image conforming to the problem description according to the first visual prompt instruction, analysis of the problem text in the diffusion model training is avoided, and the training efficiency is improved. And the training efficiency of the diffusion model is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and in particular to a visual question answering model training, application method, device and equipment. Background Art

[0002] With the rapid development of information technology, the image generation task, as a typical cross-modal generation task, has gradually been applied to multiple fields. The image generation task model can generate a result image that can be output and displayed based on the input image and the question raised to answer the question raised.

[0003] In related technologies, questions are usually asked in the form of text prompts, and then the text prompts of the questions are analyzed by a large multimodal model based on the input image and the questions asked, and the result image is generated by the large multimodal model based on the input image. However, because the text prompts are not intuitive and vivid enough, some scenes cannot be described in words, which leads to a large amount of training required for the model to understand the text prompts, and the training efficiency is low. Summary of the invention

[0004] The present disclosure provides a visual question answering model training, application method, device and equipment to at least solve the problem in the related art that the model is not easy to understand text prompts, resulting in low model training efficiency.

[0005] The first aspect of the present disclosure proposes a visual question-answering model training method, including: determining a first output image based on a first input image and a first visual prompt instruction of the first input image using a diffusion model; determining an instruction learning target of the diffusion model based on the first input image and the first output image; and training the diffusion model based on the first input image, the first output image, the first visual prompt instruction and the instruction learning target to obtain a pre-trained diffusion model.

[0006] The second aspect of the present disclosure proposes a method for applying a visual question-answering model, including: obtaining an image to be processed and a second question text corresponding to the image to be processed; determining a second visual prompt instruction corresponding to the second question text based on the image to be processed and the second question text; and generating an answer image based on the image to be processed and the second visual prompt instruction using a pre-trained diffusion model.

[0007] The third aspect embodiment of the present disclosure proposes a visual question-answering model training device, including: a first determination unit, used to determine a first output image based on a first input image and a first visual prompt instruction of the first input image, using a diffusion model; a second determination unit, used to determine an instruction learning target of the diffusion model based on the first input image and the first output image; a training unit, used to train the diffusion model based on the first input image, the first output image, the first visual prompt instruction and the instruction learning target to obtain a pre-trained diffusion model.

[0008] The fourth aspect embodiment of the present disclosure proposes a visual question-answering device, including: an acquisition unit, which acquires an image to be processed and a second question text corresponding to the image to be processed; a processing unit, which is used to determine a second visual prompt instruction corresponding to the second question text based on the image to be processed and the second question text; and a generation unit, which is used to generate an answer image based on the image to be processed and the second visual prompt instruction using a pre-trained diffusion model.

[0009] The fifth aspect embodiment of the present disclosure proposes an electronic device, comprising: a processor and a memory for storing a computer program that can be run on the processor, wherein when the processor is used to run the computer program, it executes the method described in the first aspect embodiment of the present disclosure, or executes the method described in the second aspect embodiment of the present disclosure.

[0010] The sixth aspect embodiment of the present disclosure proposes a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the method described in the first aspect embodiment of the present disclosure, or to execute the method described in the second aspect embodiment of the present disclosure.

[0011] The seventh aspect embodiment of the present disclosure proposes a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect embodiment of the present disclosure, or implements the method described in the second aspect embodiment of the present disclosure.

[0012] The present invention utilizes visual prompt instructions to train the diffusion model, and can more accurately convey key information in the input image and question to the diffusion model. Compared with the use of text prompts in the prior art, there is no need to perform text splitting, semantic analysis and other operations on the input content. The features of the question text and the input image can be directly extracted from the matrix parameters of the visual prompt instructions, making the diffusion model more intuitive in acquiring the features of the question text and the input image, thereby reducing the requirements of the diffusion model for understanding complex texts and making the model training process more stable and efficient.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure. Figure 1 A flowchart of a visual question answering model training method provided by an embodiment of the present disclosure; Figure 2 A flowchart of another visual question answering model training method provided by an embodiment of the present disclosure; Figure 3 A flowchart of another visual question answering model training method provided by an embodiment of the present disclosure; Figure 4 A flowchart of another visual question answering model training method provided by an embodiment of the present disclosure; Figure 5 A flowchart of a visual question answering model application method provided by an embodiment of the present disclosure; Figure 6 A flowchart of a visual question answering model training method provided in an embodiment of the present disclosure; Figure 7 A schematic diagram of the structure of a visual question answering model training device provided by an embodiment of the present disclosure; Figure 8 A schematic diagram of the structure of a visual question answering model application device provided by an embodiment of the present disclosure; Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0016] 1.1 First, the relevant technologies involved in this application are described: Prompt Learning, as an emerging natural language processing technology, shows great potential by designing specific prompts to guide pre-trained language models to complete downstream tasks. Its core lies in converting downstream tasks into a cloze problem. By adding a specific prompt to the input text, the model is guided to generate output that meets the task requirements. This method cleverly utilizes the rich knowledge contained in the pre-trained model and can achieve excellent performance on various natural language processing (NLP) tasks without large-scale fine-tuning of the model.

[0017] In the field of Vision-Language Models (VLMs), benefiting from the progress of deep learning technology and the wide availability of large-scale datasets, they have made significant progress in recent years. By learning the relationship between images and texts, they have demonstrated excellent performance on a variety of vision and language tasks, such as image classification, image generation, image retrieval, and natural language description of images. Representative models include the Contrastive Language-Image Pre-Training (CLIP) model, the A Large-scale Image and Noisy-text embedding (ALIGN) model, and the DALL·E text generation image model. Despite the excellent performance of these models, they still face some challenges when adapting to downstream tasks, such as the huge computing resource requirements during fine-tuning, some models can only be used through the Application Programming Interface (API) interface, and low interpretability.

[0018] Visual cues based on large models of visual language are an emerging research field, that is, guiding the model by adding specific visual information to the input. This approach can not only reduce the computational cost, but also improve the generalization ability of the model. Visual cues have a wide range of applications in image generation, object detection, image segmentation and other fields.

[0019] The question-generating task is a typical cross-modal generation task. Based on the input image and the corresponding question, the model can generate an image to answer the question. For example, given an image of a hand pulling a light cord in the light-on state, and the question "What will happen if the hand pulls down?", the model can generate an image of the light after turning off the light based on the input image and question, and based on common sense or prior knowledge, and the generated image is consistent with the input image.

[0020] 1.2 Introduction to models in related technologies: The latent guided diffusion (LGD) model predicts the latent space transformation through the cross-attention mechanism of the large language model-encoder (LLM-encoder) for the input image and question, and generates the result image based on the original image through the diffusion model.

[0021] Stable Diffusion model: The original image is encoded into the latent space through the variational autoencoder (VAE), and under conditions such as text, the latent space is gradually denoised to generate a high-quality image that highly matches the text description.

[0022] InstructPix2Pix model: By inputting the input image and text instructions into a generative adversarial network, the image can be accurately modified. The core innovation of this model is to combine natural language understanding with image generation, making image editing more flexible and intuitive. Users only need to provide simple text instructions to perform various image editing operations.

[0023] However, the models in related technologies usually use text prompts. Compared with visual prompts, text prompts are not intuitive, vivid, and specific enough. Some scenes cannot be described in words at all. In addition, the LLM-encoder's ability to understand the problem directly affects the generation results. At the same time, LGD is composed of several large models, which consumes huge computing resources.

[0024] In order to solve the technical problems existing in the related art, an embodiment of the present disclosure provides a visual question answering model training method.

[0025] The embodiments of the present disclosure will be described in detail below.

[0026] like Figure 1 As shown, an embodiment of the present disclosure provides a visual question answering model training method, comprising the following steps: Step 101 : determining a first output image based on a first input image and a first visual cue instruction of the first input image by using a diffusion model.

[0027] In some embodiments, the first visual prompt instruction is used to assist the diffusion model in generating a first output image that answers the question; specifically, the first visual prompt instruction can indicate visual features related to the question in the first input image, so that the diffusion model generates the first output image based on the visual features, thereby avoiding the diffusion model's analysis of the question text.

[0028] In some embodiments, the first input image and the first visual cue instruction are input into the diffusion model so that the diffusion model generates the first output image according to the first input image and the first visual cue instruction.

[0029] In some embodiments, the first input image is an image used to train the diffusion model, which can be any type of image, such as a landscape image, a person image, etc.

[0030] Step 102: determining an instruction learning target of a diffusion model based on the first input image and the first output image.

[0031] In some embodiments, the difference between the first input image and the first output image may be determined as an instruction learning target, wherein the instruction learning target may be used to indicate the degree to which the diffusion model learns the first visual cue instruction.

[0032] Step 103 : training a diffusion model based on the first input image, the first output image, the first visual cue instruction and the instruction learning target to obtain a pre-trained diffusion model.

[0033] In some embodiments, the instruction learning target can be used as a learning guide for the diffusion model, and the diffusion model can be iteratively trained using the first input image, the first output image, the first visual prompt instruction and the instruction learning target, and the diffusion model completed by the iterative training is determined as a pre-trained diffusion model, so that the pre-trained diffusion model can generate an output image that is more consistent with the question description based on the input image and the question text.

[0034] In summary, the visual question answering model training method proposed in the present disclosure includes: determining the first output image based on the first input image and the first visual prompt instruction of the first input image using the diffusion model; determining the instruction learning target of the diffusion model based on the first input image and the first output image; training the diffusion model based on the first input image, the first output image, the first visual prompt instruction and the instruction learning target to obtain a pre-trained diffusion model. The method disclosed in the present disclosure uses the instruction learning target as the learning guide of the diffusion model, so that the diffusion model can understand the first visual prompt instruction through training, so that the pre-trained model can generate an output image that meets the question description according to the first visual prompt instruction, avoiding the analysis of the question text in the diffusion model training, and improving the training efficiency of the diffusion model.

[0035] like Figure 2 As shown, the embodiment of the present disclosure provides a visual question answering model training method. Figure 1 Further description based on the illustrated embodiment includes the following steps: Step 201 : determining a first output image based on a first input image and a first visual cue instruction of the first input image by using a diffusion model.

[0036] Step 202: Perform a first encoding on the first input image to obtain a first input image code, and perform a second encoding on the first output image to obtain a first output image code.

[0037] In some embodiments, the second encoding may be encoded in the same manner as the first encoding, so that the diffusion model determines the difference between the first input image and the first output image.

[0038] In some embodiments, the first encoding and the second encoding may be performed through a CLIP model, but the present disclosure is not limited thereto. The specific encoding method of the first encoding and the second encoding is not limited thereto.

[0039] Step 203: determining an instruction learning target of the diffusion model based on the parameter difference between the first input image encoding and the first output image encoding.

[0040] For example, the CLIP model is used to perform the first encoding and the second encoding. The instruction learning objective can be expressed by the following formula:

[0041] in, Indicates the instruction learning goal, represents the first input image, represents the first output image, It indicates that the first input image is encoded in the first way by using the CLIP model, and the first output image is encoded in the second way by using the CLIP model.

[0042] Step 204: Sample the preset noise to obtain a first sampled noise.

[0043] In some embodiments, the preset noise may be a Gaussian distribution noise with a mean of 0 and a variance of 1, but is not limited thereto. The present disclosure does not limit the type of the preset noise.

[0044] Step 205: Determine a first noise image code based on the first output image code and the first sampled noise.

[0045] In some embodiments, the first noisy image may be generated by adding the first sampling noise to the first output image code.

[0046] It should be understood that, in the process of iterative training of the diffusion model, sampling noise may be added to the output image code in each iteration, or sampling noise may be added to part of the output image code in the iteration process according to a preset number of iteration time steps, and the present disclosure is not limited to this.

[0047] For example, the number of iteration time steps is set to T and the preset noise is ~N(0,1) as an example to illustrate that the time variable , and preset noise ~N(0,1) is sampled, and at the sampling time of the time variable t, a sampling noise of a preset noise is added to the output image encoding of the diffusion model.

[0048] In some embodiments, the first noise image code may be determined by the following formula:

[0049] in, represents the first noisy image encoding, represents the first output image encoding, represents the first sample noise.

[0050] Step 206 : Determine the prediction of adding noise based on the first noise image code, the first visual cue instruction and the first input image code.

[0051] In some embodiments, the prediction of adding noise can be determined based on the first noise image encoding, the first visual cue instruction and the first input image encoding using a pre-trained preset denoising model.

[0052] In some embodiments, the predicted added noise may be determined by the following formula:

[0053] in, It means adding noise to the prediction. Indicates the preset denoising model that has been pre-trained. represents the first noisy image encoding, Indicates the first visual prompt instruction, represents the first input image encoding, where The denoising model may be based on a U-shaped neural network convolution (UNet) structure or a transformer architecture, which is not limited in the present disclosure.

[0054] Step 207: multiply the norm of the vector difference between the predicted added noise and the first sampled noise by a preset first loss coefficient to determine a noise loss parameter.

[0055] In some embodiments, the noise loss parameter may be determined by the following formula:

[0056] in, represents the noise loss parameter, represents the first loss coefficient, represents the first sampling noise, Indicates that noise is added to the prediction.

[0057] Step 208: multiply the cosine similarity between the first visual prompt instruction and the instruction learning target by a preset second loss coefficient to determine an instruction learning loss parameter.

[0058] In some embodiments, the learning loss parameter may be determined by the following formula:

[0059] in, represents the learning loss parameter, represents the second loss coefficient, represents the cosine similarity function, and the remaining parameters can refer to the description in the above embodiment and will not be repeated here.

[0060] Step 209 , weighted summing of the noise loss parameter and the instruction learning loss parameter is performed to determine the fusion loss parameter.

[0061] In some embodiments, the noise loss parameter and the instruction learning loss parameter may be weightedly summed to determine the fusion loss parameter, or the noise loss parameter and the instruction learning loss parameter may be directly summed to determine the fusion loss parameter.

[0062] In some embodiments, the fusion loss parameter may be determined by the following formula:

[0063] in, Represents the fusion loss parameter.

[0064] Step 210: Update the first visual prompt instruction based on the first visual prompt instruction and the fusion loss parameter to determine an updated visual prompt instruction.

[0065] In some instances, a gradient calculation may be performed on the fusion loss parameter to obtain a model gradient parameter of the diffusion model; and then the difference between the first visual cue instruction and the model gradient parameter is used to determine the update visual cue instruction.

[0066] In some embodiments, the update visual prompt instruction may be determined by the following formula:

[0067] in, Indicates the update visual prompt instruction. represents the preset update weight, Represents a gradient operation.

[0068] Step 211 , based on the updated visual prompt instruction, iteratively train the diffusion model until the iteration reaches a preset maximum number of iteration steps to obtain a pre-trained diffusion model.

[0069] In some embodiments, an update visual prompt instruction may be used to replace the first visual prompt instruction, and the above steps 201 to 210 may be repeatedly performed to implement an iterative training of the diffusion model.

[0070] In summary, the visual question answering model training method proposed in the present disclosure includes: determining a first output image based on a first input image and a first visual cue instruction of the first input image by using a diffusion model; performing a first encoding on the first input image to obtain a first input image encoding, and performing a second encoding on the first output image to obtain a first output image encoding; determining an instruction learning target of the diffusion model based on a parameter difference between the first input image encoding and the first output image encoding; sampling a preset noise to obtain a first sampled noise; determining a first noise image encoding based on the first output image encoding and the first sampled noise; and determining a first noise image encoding based on the first noise image encoding, the first visual cue instruction, and the first input image encoding. code, determine the predicted added noise; multiply the norm of the vector difference between the predicted added noise and the first sampled noise by a preset first loss coefficient to determine the noise loss parameter; multiply the cosine similarity between the first visual cue instruction and the instruction learning target by a preset second loss coefficient to determine the instruction learning loss parameter; perform weighted summation on the noise loss parameter and the instruction learning loss parameter to determine the fusion loss parameter; based on the first visual cue instruction and the fusion loss parameter, update the first visual cue instruction to determine the updated visual cue instruction; based on the updated visual cue instruction, iteratively train the diffusion model until the iteration reaches a preset maximum number of iterations to obtain the pre-trained diffusion model. The method disclosed in the present invention, by taking the instruction learning target as the training guide of the diffusion model, can more directly guide the diffusion model to learn the visual features related to the question text, so that the diffusion model pays more attention to the features that can distinguish between the input image and the output image, so that the pre-trained diffusion model can generate images that better meet the requirements, thereby improving the training efficiency of the diffusion model; and the method adds noise during the training process, so that the diffusion model combines the visual prompt instructions with the scheduling process of the preset noise during the training process, thereby dynamically adjusting the intensity and distribution of the noise in the image, further improving the accuracy of the pre-trained model when outputting the image.

[0071] like Figure 3 As shown, the embodiment of the present disclosure provides a visual question answering model training method, which can be Figure 1 or Figure 2 The steps of the illustrated embodiment are performed before, including step 301: Step 301: Determine a first visual prompt instruction based on a first input image and a first question text of the first input image.

[0072] In some embodiments, visual features related to the first question text in the first input image may be determined based on the first input image and the first question text, so as to generate a first visual prompt instruction based on the visual features.

[0073] In some embodiments, the specific method for determining the first visual prompt instruction can refer to Figure 4 The steps shown are as follows: Figure 4 As shown, the steps include: Step 401: divide the first input image into equal parts to obtain a first input sub-image.

[0074] In some embodiments, the first input image may be divided into equal parts using the following formula:

[0075] in, represents the first input sub-image, represents the equal division function, represents the first input image. Since the first input sub-image is a plurality of images, the first sub-image can be represented as .

[0076] Step 402: Perform a third encoding on the first input image to obtain a second input image code, and perform a fourth encoding on the first input sub-image to obtain a first input sub-image code.

[0077] In some embodiments, the third encoding and the fourth encoding are encoded in the same manner.

[0078] In some embodiments, performing the third encoding on the first input image may be expressed as:

[0079] in, represents the second input image encoding, Indicates the encoding method of the third encoding.

[0080] In some embodiments, performing the fourth encoding on the first input sub-image may be expressed as:

[0081] in, Represents the first input sub-image encoding of the i-th first input sub-image.

[0082] Step 403: Perform fifth encoding on the first question text to obtain the first question text encoding.

[0083] In some embodiments, the first question text encoding may be determined by the following formula:

[0084] in, represents the first question text encoding, Indicates the encoding method of the fifth encoding, Represents the question text.

[0085] Step 404: Use a splicing function to splice the second input image code and the first input sub-image code to determine a fused input image code.

[0086] In some embodiments, the fused input image encoding may be determined by the following formula:

[0087] Where E represents the fused input image encoding, Represents the concatenation function.

[0088] Step 405 , using the preset query information, extracting the visual features of the first question text from the fused input image code and the first question text code.

[0089] In some embodiments, visual features related to the first question text may be extracted from the fused input image encoding and the first question text encoding by using preset query information (ie, a set of learnable preset queries).

[0090] In some embodiments, visual features may be extracted by the following formula:

[0091] in, Represents visual features, express Transformer, Indicates preset query information.

[0092] Step 406 : Map the visual features based on the diffusion model to obtain a first visual prompt instruction.

[0093] In some embodiments, since the extracted visual features are different from the dimensions of the diffusion model, it is necessary to map the visual features according to the model dimensions of the diffusion model, and determine the mapped visual features as the first visual prompt instruction.

[0094] In summary, the visual question answering model training method proposed in the present disclosure includes: dividing the first input image into equal parts to obtain a first input sub-image; performing a third encoding on the first input image to obtain a second input image encoding, and performing a fourth encoding on the first input sub-image to obtain a first input sub-image encoding; performing a fifth encoding on the first question text to obtain a first question text encoding; using a splicing function to splice the second input image encoding and the first input sub-image encoding to determine a fused input image encoding; using preset query information to extract the visual features of the first question text from the fused input image encoding and the first question text encoding; based on a diffusion model, mapping the visual features to obtain a first visual prompt instruction. The method disclosed in the present disclosure, by encoding the first input image and the first question text, thereby obtaining visual features related to the first question text, and then using the visual features to generate a first visual prompt instruction, realizes the generation of the first visual prompt instruction.

[0095] like Figure 5 As shown, an embodiment of the present disclosure provides a visual question answering model application method, comprising the following steps: Step 501: Obtain an image to be processed and a second question text corresponding to the image to be processed.

[0096] Step 502: Determine a second visual prompt instruction corresponding to the second question text based on the image to be processed and the second question text.

[0097] In some embodiments, the method for determining the second visual prompt instruction is similar to Figure 4 The method of determining the first visual prompt instruction in the embodiment shown is similar, and can be referred to Figure 4 The relevant description in the illustrated embodiment will not be repeated here.

[0098] Step 503: Generate a response image based on the image to be processed and the second visual prompt instruction using a pre-trained diffusion model.

[0099] In some embodiments, the editing area and editing direction of the image to be processed can be determined based on the second visual prompt instruction; and then based on the editing area and editing direction, the image to be processed is edited to generate a response image.

[0100] In summary, the application method of the visual question answering model proposed in the present disclosure includes: obtaining an image to be processed and a second question text corresponding to the image to be processed; determining a second visual prompt instruction corresponding to the second question text based on the image to be processed and the second question text; and generating an answer image using a pre-trained diffusion model based on the image to be processed and the second visual prompt instruction. The method disclosed in the present disclosure avoids direct analysis of the question text by the pre-trained diffusion model by using the second visual prompt instruction to generate the answer image, reduces the requirements of the pre-trained model for understanding complex text when generating the answer image, improves the efficiency of generating the answer image, and reduces the resource consumption of the pre-trained diffusion model when generating the answer image.

[0101] The following is an exemplary description of the present disclosure: 1. Visual Instruction Generator.

[0102] Visual cue instruction generator. That is, according to the input image And the corresponding problems , using lightweight transformers such as QueryingTransformer (Q-Former) to get visual cues relevant to the question These can be used as prior information to make the subsequent optimization more targeted, thereby reducing the difficulty of training the original graph and improving the convergence speed of the model. Figure 6 As shown, the specific steps are as follows: 1. Encode the input image. For the image generation task, the original input image and the generated image have strong similarities, but the local details have different results due to common sense reasoning and causal relationships. Therefore, in order to capture these details, the original input image (i.e., the first input image mentioned above) is divided into equal parts without changing the encoding of the original input image to obtain the equal-divided images (i.e., the first input sub-image mentioned above), and the original input image and the equal-divided images are encoded at the same time, and these codes are integrated. The details are as follows: 1.1 Original input image encoding.

[0103] The original input image is encoded using the existing image encoder image_encoder (i.e., the third encoding mentioned above) to obtain the original input image encoding (i.e., the second input image encoding mentioned above), that is, , where the original input image is encoded It can be composed of multiple coding blocks m1, m2...mn, where image_encoder can use CLIP's img_encoder or others.

[0104] 1.2 Divide the original input image into equal parts and encode it.

[0105] First, the original input image is divided into equal parts. , is the input image, Image division function, is the result of dividing the original input image into equal parts, where ; Secondly, the equally divided image is encoded using the same encoding method as the original input image (i.e. the fourth encoding mentioned above) to obtain the equally divided image encoding (i.e. the first input sub-image encoding mentioned above), that is, ,in is the i-th equally divided image of the original input image The encoding result is, where the image is equally divided The encoding result It can be composed of multiple coding blocks ni1, ni2...nin.

[0106] 1.3 Integrate the original input image encoding and the equally divided image post-encoding.

[0107] ,in is the encoding result of the original input image, is the encoding result of the i-th equally divided image, It is a splicing operation, that is, the original input image code and the equally divided image code are spliced ​​to obtain an integrated image code E (that is, the above-mentioned fused input image code), wherein the integrated image code E can be composed of multiple coding blocks p1, p2...pn.

[0108] 2. Coding problem.

[0109] Encode the question text (i.e. the fifth coding mentioned above), namely , is the feature encoding of the input question quesiton (i.e. the first question encoding mentioned above), It is a commonly used text encoder, such as the text encoder of CLIP.

[0110] 3. Obtain visual features related to the question. That is, through a lightweight transformer structure, a set of learnable queries (i.e., the preset query information) and questions are used to extract relevant visual features (i.e., the visual features of the first text) from the frozen image encoder. Here, the following formula is used: Q-Former, .

[0111] Among them, LQ is the learnable query, is the question encoding, E is the visual encoding, wherein the above set of learnable queries can be composed of multiple preset query information q1, q2...qn, and the above visual features It can be composed of multiple feature information groups r1, r2...rn.

[0112] 4. Use the fully connected layer to Mapped to the pre-trained denoising model as The required dimensions. ,Visual prompt instructions (i.e. the first visual prompt instruction mentioned above) is The result after processing by the fully connected layer is as follows: It can be composed of multiple instruction information V1, V2...Vn.

[0113] 2. Visual Cue Instruction Learning The visual cue instruction generator can interact the learnable query and question with visual information through the attention mechanism in the lightweight transformer. Although this method can focus on the area related to the question and extract the corresponding visual information, this information is not enough as a visual cue to guide the model to generate a suitable image based on the input image and question. Therefore, this information can be used as prior information to guide the diffusion model to learn more matching visual cues. The following takes the difference between the input image and the output image as the goal, combines the original input image, question, and learnable query, and learns visual cues based on the diffusion model. The specific steps are as follows: 1. Input image Encoding (i.e. the first encoding mentioned above) to obtain the input image encoding (i.e. the first input image encoding mentioned above), and the output image Encode (i.e. the second encoding mentioned above) to obtain the output image encoding (i.e. the first output image encoding), that is, using The same image encoding method has the same effect on the output image. Encode, that is .

[0114] 2. Calculate the input image and output image The difference between them is used as the target of visual cue instruction learning (i.e. the above instruction learning target). The CLIP model is used here to calculate the difference between them. .

[0115] 3. Learn visual cues Since the text-based prompts cannot better capture the exact editing direction, Based on the latent space, we integrate learnable visual cues and output graphs , learning and optimizing visual cues through the denoising process , let the pre-trained denoising model be , the number of optimization iterations is N, the number of time steps is T, and the specific steps are as follows: 3.1. Execute steps 3 and 4 of the visual prompt instruction generator to obtain visual prompt instructions related to the problem and the learnable parameters ; 3.2. From the uniformly distributed time and Gaussian noise (i.e. the above-mentioned preset noise) is sampled, i.e. the time variable Any parameter in [0, T] can be selected, and the probability of selecting each parameter is equal. ~N(0,1) is a Gaussian distributed random variable with mean parameter 0 and variance 1.

[0116] 3.3. At time step t, the noise Add to output image Corresponding code In (That is, the preset noise image encoding mentioned above).

[0117] 3.4. Predict the added noise through the pre-trained denoising model, that is, , that is, the above prediction adds noise, where It is a basic denoising model, which can be based on UNet structure or transformer architecture.

[0118] 3.5. Calculate the difference between predicted noise and real noise (that is, the noise loss parameter mentioned above), MSE loss is used here, that is, ,in is the loss coefficient.

[0119] 3.6. Calculate the difference between input and output images With visual cues The loss between (i.e. the above learning loss parameter), that is ,in is the loss coefficient.

[0120] 3.7. Calculate the total loss (i.e. the above fusion loss parameter) and update the gradient, i.e. .

[0121] 3.8 Update ,Right now .

[0122] In order to implement the visual question answering model training method provided by the embodiment of the present disclosure, the embodiment of the present disclosure also provides a visual question answering model training device, such as Figure 7 As shown, the visual question answering model training device 700 includes: A first determining unit 701 is configured to determine a first output image based on a first input image and a first visual cue instruction of the first input image by using a diffusion model; A second determining unit 702 is used to determine an instruction learning target of a diffusion model based on the first input image and the first output image; The training unit 703 is used to train the diffusion model based on the first input image, the first output image, the first visual cue instruction and the instruction learning target to obtain a pre-trained diffusion model.

[0123] In some embodiments, the second determination unit 702 is further used to perform a first encoding on the first input image to obtain the first input image encoding, and to perform a second encoding on the first output image to obtain the first output image encoding; based on the parameter difference between the first input image encoding and the first output image encoding, determine the instruction learning target of the diffusion model.

[0124] In some embodiments, the training unit 703 is also used to sample the preset noise to obtain a first sampled noise; determine the first noise image code based on the first output image code and the first sampled noise; determine the predicted added noise based on the first noise image code, the first visual cue instruction and the first input image code; iterate the diffusion model based on the predicted added noise, the first sampled noise, the first visual cue instruction and the instruction learning target to obtain a pre-trained diffusion model.

[0125] In some embodiments, the training unit 703 is also used to multiply the norm of the vector difference between the predicted added noise and the first sampled noise by a preset first loss coefficient to determine a noise loss parameter; multiply the cosine similarity between the first visual cue instruction and the instruction learning target by a preset second loss coefficient to determine an instruction learning loss parameter; perform weighted summation of the noise loss parameter and the instruction learning loss parameter to determine a fusion loss parameter; based on the first visual cue instruction and the fusion loss parameter, update the first visual cue instruction to determine an updated visual cue instruction; based on the updated visual cue instruction, iteratively train the diffusion model until the iteration reaches a preset maximum number of iterations to obtain a pre-trained diffusion model.

[0126] In some embodiments, the training unit 703 is further used to perform gradient calculation on the fusion loss parameter to obtain the model gradient parameter of the diffusion model; and to determine the update visual prompt instruction based on the difference between the first visual prompt instruction and the model gradient parameter.

[0127] In some embodiments, the first determining unit 701 is further configured to determine a first visual prompt instruction based on the first input image and the first question text of the first input image.

[0128] In some embodiments, the first determination unit 701 is also used to divide the first input image into equal parts to obtain a first input sub-image; perform a third encoding on the first input image to obtain a second input image encoding, and perform a fourth encoding on the first input sub-image to obtain a first input sub-image encoding; perform a fifth encoding on the first question text to obtain a first question text encoding; and determine a first visual prompt instruction based on the second input image encoding, the first input sub-image encoding and the first question text encoding.

[0129] In some embodiments, the first determination unit 701 is further used to use a splicing function to splice the second input image code and the first input sub-image code to determine the fused input image code; use preset query information to extract the visual features of the first question text from the fused input image code and the first question text code; and map the visual features based on the diffusion model to obtain the first visual prompt instruction.

[0130] In summary, the visual question answering model training device proposed in the present disclosure includes: a first determination unit, which is used to determine the first output image based on the first input image and the first visual prompt instruction of the first input image by using a diffusion model; a second determination unit, which is used to determine the instruction learning target of the diffusion model based on the first input image and the first output image; a training unit, which is used to train the diffusion model based on the first input image, the first output image, the first visual prompt instruction and the instruction learning target to obtain a pre-trained diffusion model. The device of the present disclosure uses the instruction learning target as the learning guide of the diffusion model, so that the diffusion model can understand the first visual prompt instruction through training, so that the pre-trained model can generate an output image that meets the question description according to the first visual prompt instruction, avoiding the analysis of the question text in the diffusion model training, and improving the training efficiency of the diffusion model.

[0131] In order to implement the visual question answering model training method provided by the embodiment of the present disclosure, the embodiment of the present disclosure also provides a visual question answering model application device, such as Figure 8 As shown, the visual question answering model application device 800 includes: An acquisition unit 801 acquires an image to be processed and a second question text corresponding to the image to be processed; The processing unit 802 is used to determine a second visual prompt instruction corresponding to the second question text based on the image to be processed and the second question text; The generating unit 803 is used to generate a response image based on the image to be processed and the second visual prompt instruction by using a pre-trained diffusion model.

[0132] In some embodiments, the generating unit 803 is further used to determine the editing area and editing direction of the image to be processed based on the second visual prompt instruction; based on the editing area and editing direction, the image to be processed is edited to generate a response image.

[0133] In summary, the visual question-answering model application device proposed in the present disclosure includes: an acquisition unit, which acquires the image to be processed and the second question text corresponding to the image to be processed; a processing unit, which is used to determine the second visual prompt instruction corresponding to the second question text based on the image to be processed and the second question text; and a generation unit, which is used to generate an answer image based on the image to be processed and the second visual prompt instruction using a pre-trained diffusion model. The device of the present disclosure generates an answer image using the second visual prompt instruction, thereby avoiding the direct analysis of the question text by the pre-trained diffusion model, reducing the requirements of the pre-trained model for understanding complex text when generating the answer image, improving the efficiency of generating the answer image, and reducing the resource consumption of the pre-trained diffusion model when generating the answer image.

[0134] It should be noted that: the visual question and answer model training device provided in the above embodiment, when performing visual question and answer model training, or the visual question and answer model application device provided in the above embodiment, when performing visual question and answer model application, only uses the division of the above program modules as an example. In actual application, the above processing can be assigned to different program modules as needed, that is, the internal structure of the visual question and answer model training device is divided into different program modules to complete all or part of the processing described above. In addition, the visual question and answer model training device provided in the above embodiment and the visual question and answer model training method embodiment provided in the embodiment of the present disclosure belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0135] Fig. 9 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure is shown in FIG. Fig. 9 As shown, the electronic device 900 includes at least one processor 902; and a memory 901 communicatively connected to the at least one processor 902; wherein the memory 901 stores instructions executable by the at least one processor 902, and the instructions are executed by the at least one processor 902 to implement the steps of the visual question answering model training method described in the embodiment of the present disclosure.

[0136] Optionally, the electronic device may specifically be the visual question answering model training device of the embodiment of the present application, and the electronic device may implement the corresponding processes implemented by the visual question answering model training device in each method of the embodiment of the present application, which will not be described in detail here for the sake of brevity.

[0137] It is understood that the electronic device also includes a communication interface 903. The various components in the electronic device are coupled together through a bus system 904. It is understood that the bus system 904 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 904 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Fig. 9 Various buses are labeled as bus system 904.

[0138] It can be understood that the memory 901 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM, SyncLink Dynamic Random Access Memory), and direct RAM bus random access memory (DRRAM, Direct Rambus Random Access Memory).The memory 901 described in the embodiment of the present invention is intended to include but is not limited to these and any other suitable types of memories.

[0139] The method disclosed in the above embodiment of the present disclosure can be applied to the processor 902, or implemented by the processor 902. The processor 902 has the ability to process signals. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 902 or the instruction in the form of software. The above processor 902 can be a general-purpose processor, a DSP, or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 902 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present invention. The general-purpose processor can be a microprocessor or any conventional processor, etc. In combination with the steps of the method disclosed in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in the memory 901, and the processor 902 reads the information in the memory 901 and completes the steps of the above method in combination with its hardware.

[0140] In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), FPGAs, general purpose processors, controllers, MCUs, microprocessors, or other electronic components to execute the aforementioned method.

[0141] The embodiment of the present disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to implement the steps of the visual question answering model training method described in the embodiment of the present disclosure when executed.

[0142] The embodiments of the present disclosure also provide a computer program product, including a computer program, which, when executed by a processor, implements the steps of the visual question answering model training method described in the embodiments of the present disclosure.

[0143] Optionally, the computer-readable storage medium can be applied to the visual question answering model training device in the embodiments of the present application, and the computer instructions enable the computer to execute the corresponding processes implemented by the visual question answering model training device in the various methods of the embodiments of the present application. For the sake of brevity, they will not be repeated here.

[0144] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0145] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0146] In addition, all functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may be separately used as a unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0147] A person of ordinary skill in the art can understand that: all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium, which, when executed, executes the steps of the above-mentioned method embodiment; and the aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, disks or optical disks.

[0148] Alternatively, if the above-mentioned integrated unit of the present invention is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present invention can be essentially or partly reflected in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROM, RAM, magnetic disks or optical disks.

[0149] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A visual question answering model training method, characterized in that: The method comprises: Determine a first output image based on a first input image and a first visual cue instruction of the first input image using a diffusion model; determining an instruction learning target of the diffusion model based on the first input image and the first output image; The diffusion model is trained based on the first input image, the first output image, the first visual cue instruction, and the instruction learning target to obtain a pre-trained diffusion model.

2. The method according to claim 1, characterized in that The step of determining the instruction learning target of the diffusion model based on the first input image and the first output image includes: Performing a first encoding on the first input image to obtain a first input image code, and performing a second encoding on the first output image to obtain a first output image code; An instruction learning target for the diffusion model is determined based on a parameter difference between the first input image encoding and the first output image encoding.

3. The method according to claim 1, characterized in that The step of training the diffusion model based on the first input image, the first output image, the first visual cue instruction, and the instruction learning target to obtain a pre-trained diffusion model includes: Sampling the preset noise to obtain a first sampled noise; determining a first noise image code based on the first output image code and the first sampled noise; Determining a predicted added noise based on the first noise image encoding, the first visual cue instruction, and the first input image encoding; The diffusion model is iterated based on the predicted added noise, the first sampled noise, the first visual cue instruction and the instruction learning target to obtain the pre-trained diffusion model.

4. The method according to claim 3, characterized in that The iterating the diffusion model based on the predicted added noise, the first sampled noise, the first visual cue instruction and the instruction learning target to obtain the pre-trained diffusion model comprises: Multiplying the norm of the vector difference between the predicted added noise and the first sampled noise by a preset first loss coefficient to determine a noise loss parameter; Multiplying the cosine similarity between the first visual cue instruction and the instruction learning target by a preset second loss coefficient to determine an instruction learning loss parameter; Performing a weighted summation on the noise loss parameter and the instruction learning loss parameter to determine a fusion loss parameter; Based on the first visual prompt instruction and the fusion loss parameter, updating the first visual prompt instruction to determine an updated visual prompt instruction; Based on the update visual prompt instruction, the diffusion model is iteratively trained until the iteration reaches a preset maximum number of iteration steps to obtain a pre-trained diffusion model.

5. The method according to claim 4, characterized in that The updating of the first visual prompt instruction based on the first visual prompt instruction and the fusion loss parameter to determine the updated visual prompt instruction comprises: Performing gradient calculation on the fusion loss parameter to obtain a model gradient parameter of the diffusion model; The updated visual prompt instruction is determined by taking the difference between the first visual prompt instruction and the model gradient parameter.

6. The method according to claim 1, characterized in that The method further comprises: The first visual prompt instruction is determined based on the first input image and a first question text of the first input image.

7. The method according to claim 6, characterized in that The determining the first visual prompt instruction based on the first input image and the first question text of the first input image comprises: Dividing the first input image into equal parts to obtain first input sub-images; Performing a third encoding on the first input image to obtain a second input image code, and performing a fourth encoding on the first input sub-image to obtain a first input sub-image code; Performing a fifth encoding on the first question text to obtain a first question text encoding; The first visual prompt instruction is determined based on the second input image code, the first input sub-image code and the first question text code.

8. The method according to claim 7, characterized in that The determining the first visual prompt instruction based on the second input image code, the first input sub-image code and the first question text code comprises: Using a splicing function, splicing the second input image code and the first input sub-image code to determine a fused input image code; Extracting visual features of the first question text from the fused input image code and the first question text code using preset query information; Based on the diffusion model, the visual features are mapped to obtain the first visual prompt instruction.

9. A method for applying a visual question answering model, characterized in that: The method comprises: Acquire an image to be processed and a second question text corresponding to the image to be processed; Determining a second visual prompt instruction corresponding to the second question text based on the image to be processed and the second question text; Based on the image to be processed and the second visual prompt instruction, a response image is generated using a pre-trained diffusion model.

10. The method according to claim 9, characterized in that The determining, based on the image to be processed and the second question text, a second visual prompt instruction corresponding to the second question text comprises: Determining an editing area and an editing direction of the image to be processed based on the second visual prompt instruction; Based on the editing area and the editing direction, the image to be processed is edited to generate the answer image.

11. A visual question answering model training device, characterized in that: include: A first determining unit, configured to determine a first output image based on a first input image and a first visual cue instruction of the first input image by using a diffusion model; a second determining unit, configured to determine an instruction learning target of the diffusion model based on the first input image and the first output image; A training unit is used to train the diffusion model based on the first input image, the first output image, the first visual cue instruction and the instruction learning target to obtain a pre-trained diffusion model.

12. A visual question answering model application device, characterized in that: include: An acquisition unit, which acquires an image to be processed and a second question text corresponding to the image to be processed; A processing unit, configured to determine a second visual prompt instruction corresponding to the second question text based on the image to be processed and the second question text; A generating unit is used to generate a response image based on the image to be processed and the second visual prompt instruction by using a pre-trained diffusion model.

13. An electronic device, characterized in that: include: a processor and a memory for storing a computer program capable of being executed on the processor, Wherein, when the processor is used to run the computer program, it executes the visual question answering model training method according to any one of claims 1-8, or executes the visual question answering model application method according to claim 9 or 10.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the visual question answering model training method according to any one of claims 1-8, or to execute the visual question answering model application method according to claim 9 or 10.

15. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, implements the visual question answering model training method according to any one of claims 1 to 8, or implements the visual question answering model application method according to claim 9 or 10.

Images