Training method and training device of visual question answering model and visual question answering system

Abstract

The application provides a training method and device of a visual question answering model and a visual question answering system. The method comprises: a first obtaining step of obtaining an image-answer pair; an input step of inputting the image-answer pair into a CLIP model to obtain a first loss value; a second obtaining step of obtaining a standard answer and calculating a second loss value of a text answer in the image-answer pair and the standard answer; a calculating step of performing weighted calculation according to the first loss value and the second loss value to obtain a target loss value; a third obtaining step of obtaining a plurality of model parameter groups and correcting the plurality of model parameter groups through an ADAM algorithm according to the target loss values to obtain a plurality of corrected parameter groups; and repeating the first obtaining step, the input step, the second obtaining step, the calculating step and the third obtaining step until the target loss values are all less than a first threshold. The method solves the problem that the visual question answering model in the prior art tends to text fitting and has low image fitting, resulting in poor model robustness.

Description

Technical Field

[0001] This invention relates to the field of visual question answering technology, and more specifically, to a training method, training device, computer-readable storage medium, and visual question answering system for a visual question answering model. Background Technology

[0002] Visual question answering tasks involve being given an image and a question about that image, and then having to provide the answer based on the image. The types of questions are varied, including multiple-choice questions and open-ended questions.

[0003] Existing techniques often use two network structures to extract information from images and text separately, and then interact the two information in some way to arrive at an answer. However, methods that extract features separately struggle to effectively fuse the two types of information. This results in the model giving little consideration to image information when answering questions. For example, the model's robustness is poor. For a given question, the model tends to choose the answer that appears most frequently in the training set, making it prone to errors in real-world scenarios. This indicates that the model overfits to the text in the training set and pays little attention to the corresponding images.

[0004] The CLIP model, released by OpenAI in early 2021, is a pre-trained neural network for matching images and text, achieving state-of-the-art (SOTA) results on many tasks in multimodal research. It uses contrastive learning to shorten the distance between similar images and text, and was trained using 400 million image-text pairs to achieve image-text alignment. However, the CLIP model cannot be directly used for visual question answering tasks. It is only a pre-trained model for image-text alignment and does not have the capability for visual question answering. Summary of the Invention

[0005] The main objective of this application is to provide a training method, training device, computer-readable storage medium, and visual question answering system for a visual question answering model, so as to at least solve the problem that in the prior art, visual question answering models tend to fit text and have poor image fitting, resulting in poor robustness of the model.

[0006] To achieve the above objectives, according to one aspect of this application, a training method for a visual question answering model is provided, comprising: a first acquisition step, acquiring multiple image-answer pairs, each image-answer pair including an input image and a text answer, wherein the input image in each image-answer pair is consistent, and the answer in different image-answer pairs is obtained by inputting the input image into different visual question answering models, the visual question answering models including Web-QA, V-Doc, TGIF-QA, and NuScenes-QA; an input step, inputting each image-answer pair into a CLIP model to obtain corresponding loss function values ​​to obtain multiple first loss values; and a second acquisition step, acquiring a standard answer. The process involves calculating multiple second loss values ​​by BLEU between the text answer and the standard answer in each image-answer pair; a calculation step involves weighting each first loss value and the corresponding second loss value to obtain multiple target loss values; a third acquisition step involves acquiring multiple model parameter sets and correcting each model parameter set using the ADAM algorithm based on each target loss value to obtain multiple corrected parameter sets, wherein the multiple model parameter sets include model parameters of different visual question answering models; and repeating the first acquisition step, the input step, the second acquisition step, the calculation step, and the third acquisition step at least once in sequence until each target loss value is less than a first threshold.

[0007] Optionally, after repeating the first acquisition step, the input step, the second acquisition step, the calculation step, and the third acquisition step at least once in sequence until each of the target loss values ​​is less than a first threshold, the method further includes: acquiring a first target image, wherein the first target image is the input image to be question-answered; inputting the first target image into each of the visual question-answering models to obtain the corresponding text answer; constructing image-answer pairs based on each of the text answers and the first target image to obtain multiple target image-answer pairs; inputting each target image-answer pair into the CLIP model to obtain multiple first similarities, wherein the first similarity is the cosine similarity between the first target image and the text answer in each target image-answer pair; and determining the text answer in the target image-answer pair with the largest first similarity as the target answer and outputting it.

[0008] Optionally, inputting each image-answer pair into the CLIP model to obtain a corresponding loss function value yields multiple first loss values, including: inputting the input image in the image-answer pair into a first encoder to obtain a first vector, the first encoder being used to encode the input image into a vector representation; inputting the text answer in the image-answer pair into a second encoder to obtain a second vector, the second encoder being used to encode the text answer into a vector representation; calculating a cosine similarity based on the first vector and the second vector to obtain a target similarity, and substituting the target similarity into a first preset formula to obtain the first loss value.

[0009] Optionally, inputting the input image in the image answer pair into a first encoder to obtain a first vector includes: cropping the input image in the image answer pair to a preset size to obtain a first image; normalizing the pixel values ​​of each pixel block in the first image to obtain a second image; performing data augmentation on the second image to obtain a third image, wherein the data augmentation includes at least random cropping, flipping, and rotation; encoding the third image and converting the third image into a vector form to obtain the first vector.

[0010] Optionally, calculating the BLEU between the text answer and the standard answer in each image-answer pair yields multiple second loss values, including: segmenting the text answer and the standard answer into words and combining them to obtain multiple sub-texts, where the number of words in each sub-text is a preset number; determining the frequency of occurrence of each sub-text in the text answer by matching each sub-text with the text answer to obtain multiple first statistical values; determining the frequency of occurrence of each sub-text in the standard answer by matching each sub-text with the standard answer to obtain second statistical values; determining a third statistical value based on the text answer, and calculating each... The ratio of the first statistical value to the third statistical value yields multiple fourth statistical values, where the third statistical value is the number of subtexts contained in the text answer; each of the fourth statistical values ​​is substituted into a second preset formula to obtain the candidate BLEUs corresponding to the text answer; a fifth statistical value is determined based on the standard answer, where the fifth statistical value is the number of subtexts contained in the standard answer; the difference between the third statistical value and the fifth statistical value is calculated to obtain a sixth statistical value; the harmonic mean of the candidate BLEUs is calculated to obtain a seventh statistical value; and the product of the sixth statistical value and the seventh statistical value is calculated to obtain the second loss value.

[0011] Optionally, the ADAM algorithm is used to correct each of the target loss values, including: calculating the gradient of the target loss value with respect to the corresponding model parameter set of the visual question answering model using the backpropagation algorithm to obtain the target gradient; and updating the model parameter set using the ADAM algorithm based on the target gradient to obtain the corresponding corrected parameter set.

[0012] Optionally, multiple target loss values ​​are obtained by weighted calculation based on each of the first loss values ​​and the corresponding second loss values, including: obtaining a first weight and a second weight, wherein the first weight is the weight corresponding to the first loss value and the second weight is the weight corresponding to the second loss value; and calculating the sum of the product of the first loss value and the first weight and the product of the second loss value and the second weight to obtain the target loss value.

[0013] According to another aspect of this application, a training apparatus for a visual question answering model is provided. The apparatus includes: a first acquisition unit, configured to perform a first acquisition step, acquiring multiple image-answer pairs, each image-answer pair including an input image and a text answer, wherein the input image in each image-answer pair is consistent, and the answer in different image-answer pairs is obtained by inputting the input image into different visual question answering models, the visual question answering models including Web-QA, V-Doc, TGIF-QA, and NuScenes-QA; a first input unit, configured to perform an input step, inputting each image-answer pair into a CLIP model to obtain corresponding loss function values ​​and thus multiple first loss values; and a second acquisition unit, configured to perform a second acquisition step, acquiring a target... The system first obtains a standard answer and calculates the BLEU between the text answer and the standard answer in each image answer pair to obtain multiple second loss values; a calculation unit is used to calculate multiple target loss values ​​by weighting each first loss value and the corresponding second loss value; a third acquisition unit is used to execute a third acquisition step, acquire multiple model parameter sets, and correct each model parameter set according to each target loss value using the ADAM algorithm to obtain multiple corrected parameter sets, wherein the multiple model parameter sets include model parameters of different visual question answering models; a repetition unit is used to repeat the first acquisition step, input step, second acquisition step, calculation step and third acquisition step at least once in sequence until each target loss value is less than a first threshold.

[0014] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform any of the methods described.

[0015] According to another aspect of this application, a visual question-answering system is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for performing any one of the methods described.

[0016] Applying the technical solution of this application, in the training method of the above-mentioned visual question answering model, firstly, a first acquisition step is performed to acquire multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input images in each image-answer pair are consistent, and the answers in different image-answer pairs are obtained by inputting the input images into different visual question answering models, including Web-QA, V-Doc, TGIF-QA, and NuScenes-QA. Then, an input step is performed to input each of the above-mentioned image-answer pairs into the CLIP model to obtain the corresponding loss function values ​​and obtain multiple first loss values. Afterwards, a second acquisition step is performed to acquire the standard answer and calculate each of the above-mentioned... The BLEU between the text answer and the standard answer in the image answer pair yields multiple second loss values. Then, a calculation step is performed to calculate multiple target loss values ​​by weighting each of the first loss values ​​and the corresponding second loss values. Next, a third acquisition step is performed to acquire multiple model parameter sets, and these sets are corrected using the ADAM algorithm based on the target loss values ​​to obtain multiple corrected parameter sets, which include model parameters from different visual question answering models. Finally, the first acquisition step, the input step, the second acquisition step, the calculation step, and the third acquisition step are repeated at least once in sequence until all the target loss values ​​are less than a first threshold. This application is based on multiple existing question-answering models, providing visual question-answering answers based on the input image. It uses the CLIP model for text and image alignment, and then calculates the total loss value based on the loss values ​​from both text and image aspects. The model parameters of each model are updated based on the loss value to improve the accuracy of each model. Compared to existing technologies that use two network structures to process images and text separately, this application uses the CLIP model for text alignment and fuses text and image information. This solves the problem in existing visual question-answering models that tend to fit text more closely and have lower image fitting, resulting in poor robustness. Attached Figure Description

[0017] Figure 1 A hardware structure block diagram of a mobile terminal for a training method of a visual question answering model provided in an embodiment of this application is shown.

[0018] Figure 2A flowchart illustrating a training method for a visual question-answering model according to an embodiment of this application is shown.

[0019] Figure 3 A flowchart illustrating a specific method for training a visual question-answering model according to an embodiment of this application is shown.

[0020] Figure 4 A structural block diagram of a training device for a visual question-answering model provided according to an embodiment of this application is shown.

[0021] The above figures include the following reference numerals:

[0022] 102. Processor; 104. Memory; 106. Transmission device; 108. Input / output device. Detailed Implementation

[0023] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] As described in the background section, existing visual question answering models tend to fit text information, resulting in poor combination of images and text, leading to poor robustness and low accuracy. To address the problem of existing visual question answering models tending to fit text and having poor image fitting, resulting in poor model robustness, embodiments of this application provide a training method, training device, computer-readable storage medium, and visual question answering system for a visual question answering model.

[0027] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0028] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a training method of a visual question-answering model according to an embodiment of the present invention. For example... Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0029] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the device information display method in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0030] This embodiment provides a method for training a visual question-answering model that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0031] Figure 2 This is a flowchart of a training method for a visual question-answering model according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0032] Step S201, the first acquisition step, acquires multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input image in each image-answer pair is the same. The answers in different image-answer pairs are obtained by inputting the input image into different visual question-answering models. The visual question-answering models include Web-QA, V-Doc, TGIF-QA, and NuScenes-QA.

[0033] Specifically, during the training process, an image to be used for visual question answering is first selected, i.e., the input image mentioned above. Then, the input image is input into the visual question answering model to obtain the answer, i.e., the text answer mentioned above. Then, the image answer pair is constructed based on the input image and the text answer mentioned above.

[0034] In practice, the more visual question answering models set up in this application, the higher the accuracy of the final trained comprehensive model. However, as the number of visual question answering models increases, the training difficulty and computational complexity also increase. Therefore, this application sets up at least Web-QA, V-Doc, TGIF-QA and NuScenes-QA as the aforementioned visual question answering models to ensure the performance of the comprehensive visual question answering model. Models can be added and trained according to the requirements for question answering accuracy.

[0035] Step S202: Input the above image answer pairs into the CLIP model to obtain the corresponding loss function values ​​and obtain multiple first loss values;

[0036] Specifically, to address the issue that existing models tend to fit text and have a low degree of integration between images and text, this application proposes to use the CLIP model to perform text alignment on the basis of a basic visual question answering model, thereby enhancing the integration of text and images in the visual question answering model. Then, based on the similarity between text and images, the loss value of the loss function is determined to obtain the aforementioned first loss value.

[0037] Step S203, the second acquisition step, is to acquire the standard answer and calculate the BLEU between the text answer and the standard answer in each of the above image answer pairs to obtain multiple second loss values;

[0038] Specifically, a BLEU value is calculated based on the standard answer and the text answers output by each of the aforementioned visual question answering models. This value is used to characterize the similarity between the text answer and the standard answer, and serves as the second loss value.

[0039] Step S204, calculation step: multiple target loss values ​​are obtained by weighted calculation based on each of the above-mentioned first loss values ​​and the corresponding above-mentioned second loss values;

[0040] Specifically, by combining the similarity between the input images and corresponding text answers of each of the above-mentioned visual question answering models, as well as the similarity between the text answers and standard answers of the above-mentioned visual question answering models, the loss value of each visual question answering model in the integrated visual question answering model is determined to obtain the above-mentioned target loss value.

[0041] Step S205, the third acquisition step, acquire multiple model parameter sets, and correct each of the above-mentioned model parameter sets according to the above-mentioned target loss values ​​through the ADAM algorithm to obtain multiple corrected parameter sets. The multiple above-mentioned model parameter sets include model parameters of different above-mentioned visual question answering models.

[0042] Specifically, in order to enhance the connection between the input image and the text answer, this application sets the target loss value as a guide, replacing the loss value calculated by the loss function in the prior art, and adjusts the parameters of each of the above visual question answering models through the ADAM algorithm to minimize the target loss value.

[0043] Step S206: Repeat the first acquisition step, the input step, the second acquisition step, the calculation step, and the third acquisition step at least once in sequence until each of the target loss values ​​is less than the first threshold.

[0044] Specifically, if the target loss value corresponding to the above-mentioned visual question answering model is greater than the preset threshold, it is determined that the accuracy of the above-mentioned visual question answering model has not met the usage requirements. The first acquisition step, input step, second acquisition step, calculation step and third acquisition step are repeated. Based on the above-mentioned target loss value, each visual question answering model is further optimized by the ADAM algorithm until the target loss value corresponding to all visual question answering models is less than the above-mentioned first threshold, and it is determined that the accuracy of the comprehensive visual question answering model has met the requirements.

[0045] In practice, as the number of iterations increases, the optimization level of the model gradually decreases. To avoid wasting computing resources, this application sets a maximum number of iterations. When the maximum number of iterations is reached, the modification of the above visual question answering model is terminated.

[0046] In this embodiment, firstly, a first acquisition step is performed to acquire multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input images in each image-answer pair are the same, while the answers in different image-answer pairs are obtained by inputting the input images into different visual question-answering models, including Web-QA, V-Doc, TGIF-QA, and NuScenes-QA. Then, an input step is performed to input each image-answer pair into a CLIP model to obtain corresponding loss function values ​​and multiple first loss values. Afterward, a second acquisition step is performed to acquire the standard answer and calculate the text answer in each image-answer pair. The BLEU between this answer and the aforementioned standard answer yields multiple second loss values. Next, a calculation step is performed, weighting each of the aforementioned first loss values ​​and the corresponding second loss values ​​to obtain multiple target loss values. Then, a third acquisition step is executed, acquiring multiple model parameter sets, and correcting each of the aforementioned model parameter sets using the ADAM algorithm based on each of the aforementioned target loss values ​​to obtain multiple corrected parameter sets. These multiple model parameter sets include model parameters from different visual question-answering models. Finally, the aforementioned first acquisition step, input step, second acquisition step, calculation step, and third acquisition step are repeated at least once in sequence until each of the aforementioned target loss values ​​is less than a first threshold. This application is based on multiple existing question-answering models, providing visual question-answering answers based on the input image. It uses the CLIP model for text and image alignment, and then calculates the total loss value based on the loss values ​​from both text and image aspects. The model parameters of each model are updated based on the loss value to improve the accuracy of each model. Compared to existing technologies that use two network structures to process images and text separately, this application uses the CLIP model for text alignment and fuses text and image information. This solves the problem in existing visual question-answering models that tend to fit text more closely and have lower image fitting, resulting in poor robustness.

[0047] To determine the text answer of the aforementioned comprehensive visual question answering model, in an optional implementation, after repeating the first acquisition step, input step, second acquisition step, calculation step, and third acquisition step at least once in sequence until each of the aforementioned target loss values ​​is less than a first threshold, the method further includes:

[0048] Step S301: Obtain the first target image, which is the input image to be used for question answering;

[0049] Specifically, upon receiving a visual question-answering task, the image in the visual question-answering task is extracted to obtain the aforementioned first target image, and then the aforementioned first target image is...

[0050] Step S302: Input the first target image into each of the above-mentioned visual question answering models to obtain the corresponding text answers;

[0051] Specifically, the first target image is input into the integrated visual question answering model. The integrated visual question answering model processes the first target image and then inputs it into each of the visual question answering models in sequence. The output results of each of the visual question answering models are collected to obtain multiple text answers.

[0052] Step S303: Construct image-answer pairs based on each of the above-mentioned text answers and the above-mentioned first target image to obtain multiple target image-answer pairs;

[0053] Specifically, based on the text answer of the above visual question answering model and the above first target image, the above image answer pair is constructed to obtain multiple above target image answer pairs.

[0054] Step S304: Input each of the above target image answer pairs into the above CLIP model to obtain multiple first similarities. The first similarity is the cosine similarity between the first target image and the text answer in each of the above target image answer pairs.

[0055] Specifically, each of the aforementioned target image answer pairs is input into the CLIP model. The CLIP model encodes the text answer and the first target image in each target image answer pair to obtain corresponding text vectors and image vectors. Then, cosine similarity is calculated based on the text vectors and image vectors to obtain the aforementioned first similarity.

[0056] Step S305: Determine the text answer in the target image answer pair with the highest first similarity as the target answer and output it.

[0057] Specifically, the text answer in the target image answer pair with the highest first similarity is the answer with the highest accuracy. Therefore, the text answer is determined as the final text answer, that is, the target answer is output to the user.

[0058] In order to calculate the first loss value mentioned above, in an optional implementation, step S202 includes:

[0059] Step S2021: Input the input image from the above image answer pair into the first encoder to obtain the first vector. The first encoder is used to encode the input image and convert it into a vector representation.

[0060] Specifically, the input image is fed into a first encoder, which preprocesses the input image and encodes it to obtain a vector representing the input image, resulting in the first vector. In specific implementations, the first encoder can employ CNN, Autoencoder, Sparse coding, VAE, or Feature extractor, among others.

[0061] Step S2022: Input the text answer in the above image answer pair into the second encoder to obtain the second vector. The second encoder is used to encode the text answer and convert it into a vector representation.

[0062] Specifically, the text answer is input into a second encoder, which segments, processes, and encodes the text answer to obtain the second vector representing the text answer. In specific implementations, the second encoder can employ TF-IDF, Word2Vec, GloVe, Doc2Vec, and Universal Sentence Encoder, among others.

[0063] Step S2023: Calculate the cosine similarity based on the first vector and the second vector to obtain the target similarity, and substitute the target similarity into the first preset formula to obtain the first loss value.

[0064] Specifically, let the first vector be A and the second vector be B, and then substitute the first vector and the second vector into the formula text. Similarity(A，B) =Cosine Similarity{f(A), g(B)}, that is, calculate the cosine similarity of the first vector and the second vector to obtain the target similarity. Then, let the first loss value be L1, and the first preset formula is L = -log{softmax[Sim(A,B)] / T}, where Sim(A,B) is the target similarity and T is the temperature parameter. Substituting the target similarity into the first preset formula will give the first loss value.

[0065] To obtain the aforementioned first vector, in one optional implementation, step S2021 includes:

[0066] Step S20211: Crop the input image in the above image answer pair to a preset size to obtain the first image;

[0067] Specifically, the encoder in the CLIP model first resizes the input image to the size required by the encoder to obtain the first image mentioned above. Depending on the encoder used, the corresponding size is different, usually 224*224 or 256*256 pixels to obtain the image.

[0068] Step S20212: Normalize the pixel values ​​of each pixel block in the first image to obtain the second image;

[0069] Specifically, for ease of encoding, after cropping the input image, this application sets up a method to standardize the pixel values ​​of the first image to obtain the second image. In specific implementations, this can be to scale the pixel values ​​to between 0 and 1, or to perform zero-mean and standardization processing.

[0070] Step S20213: Perform data augmentation on the second image to obtain a third image. The data augmentation includes at least random cropping, flipping, and rotation.

[0071] Specifically, to enhance the robustness and generalization ability of the model, this application sets up a method to perform random cropping, horizontal flipping, or rotation on the second image after adjusting the pixel values ​​of the image to obtain the third image.

[0072] Step S20214: Encode the third image and convert it into a vector form to obtain the first vector.

[0073] Specifically, the first vector is obtained by encoding the third image using an encoder.

[0074] To obtain the aforementioned second loss value, in one optional implementation, step S203 includes:

[0075] Step S2031: The above text answer and the above standard answer are segmented and combined to obtain multiple subtexts, and the number of the above segments in each subtext is a preset number;

[0076] Specifically, firstly, each of the above text answers and the above standard answers is segmented into multiple grams. Then, the number of segments n contained in the above subtext is set, and consecutive above grams are combined to obtain multiple n-grams, which is the above subtext.

[0077] Step S2032: Based on matching each of the above subtexts with the above text answer, determine the number of occurrences of each of the above subtexts in the above text answer to obtain multiple first statistical values; based on matching each of the above subtexts with the above standard answer, determine the number of occurrences of each of the above subtexts in the above standard answer to obtain second statistical values.

[0078] Specifically, by matching each of the above n-grams with each of the above text answers, the frequency of each n-gram in each of the above text answers is determined, thus obtaining the first statistical value. Then, by matching each of the above n-grams with each of the above standard answers, the frequency of each n-gram in each of the above standard answers is determined, thus obtaining the second statistical value.

[0079] Step S2033: Determine the third statistical value based on the above text answer, calculate the ratio of each of the above first statistical values ​​and the above third statistical values ​​to obtain multiple fourth statistical values, where the above third statistical value is the number of the above subtexts contained in the above text answer;

[0080] Specifically, the number of the aforementioned n-grams contained in each of the aforementioned text answers is determined to obtain the aforementioned third statistical value. The difference between the aforementioned first statistical value and the aforementioned third statistical value is then calculated to obtain the aforementioned fourth statistical value.

[0081] Step S2034: Substitute each of the above fourth statistical values ​​into the second preset formula to obtain the alternative BLEU corresponding to the above text answer;

[0082] Specifically, substituting the fourth statistical value mentioned above into the formula BLEU=BP*exp(1 / n*∑(1to n)w n *log(p n ), where BP short text penalty, w n The weight is usually 1 / n, p n为 The fourth statistical value mentioned above.

[0083] Step S2035: Determine the fifth statistical value based on the above standard answer. The fifth statistical value is the number of the above sub-texts contained in the above standard answer.

[0084] Specifically, the fifth statistical value is obtained by determining the number of the aforementioned n-grams contained in the above standard answer.

[0085] Step S2036: Calculate the difference between the third statistical value and the fifth statistical value to obtain the sixth statistical value; calculate the harmonic mean of the candidate BLEUs to obtain the seventh statistical value; and calculate the product of the sixth statistical value and the seventh statistical value to obtain the second loss value.

[0086] Specifically, based on the third and fifth statistical values ​​mentioned above, the length difference between the text answer and the standard answer is calculated to obtain the sixth statistical value. The harmonic mean is calculated based on the candidate BLEU to obtain the seventh statistical value. Then, the sixth and seventh statistical values ​​are used to calculate the second loss value L2.

[0087] In order to update the aforementioned visual question-answering models, in one optional implementation, step S205 includes:

[0088] Step S2051: Based on the target loss value, calculate the gradient of the target loss value with respect to the model parameter set of the corresponding visual question answering model using the backpropagation algorithm to obtain the target gradient;

[0089] Specifically, based on the target loss value, the gradient of the visual question answering model is calculated layer by layer using the gradient descent method and the reverberation propagation algorithm to obtain multiple target gradients.

[0090] Step S2052: Update the model parameter set according to the target gradient using the ADAM algorithm to obtain the corresponding corrected parameter set.

[0091] Specifically, the above-mentioned modified parameter set is obtained by updating the corresponding parameters according to the target gradient of each layer of the model based on the ADAM algorithm described above.

[0092] To obtain the target loss value, in one optional implementation, step S204 includes:

[0093] Step S2041: Obtain the first weight and the second weight, wherein the first weight is the weight corresponding to the first loss value and the second weight is the weight corresponding to the second loss value.

[0094] Specifically, the first weight mentioned above is obtained and set as W1, and the second weight mentioned above is obtained and set as W2.

[0095] Step S2042: Calculate the sum of the product of the first loss value and the first weight and the product of the second loss value and the second weight to obtain the target loss value.

[0096] Specifically, by substituting the first loss value, the first weight, the second weight, and the second loss value into the formula L = L1 * W1 + L2 * W2, the target loss value L is obtained.

[0097] To enable those skilled in the art to better understand the technical solution of this application, the implementation process of the training method of the visual question answering model of this application will be described in detail below with reference to specific embodiments.

[0098] This embodiment relates to a specific training method for a visual question-answering model, such as... Figure 3 As shown, it includes the following steps:

[0099] Step S1: Select multiple existing visual question answering models, input a unified visual question answering task, and generate text answers one by one;

[0100] Step S2: Input the image and the text answer from the above visual question answering task into the CLIP model, calculate the similarity between the image and the text, and then take the negative log value of the similarity as the image loss.

[0101] Step S3: Obtain the standard answer, calculate the BLEU score based on the standard answer and each of the above text answers, and obtain the text loss;

[0102] Step S4: Calculate the final loss by weighting the image loss and text loss, and feed the final loss back to each of the above visual question answering models. Adjust the parameters of the above visual question answering models according to the final loss.

[0103] Step S5: Repeat steps S1 to S4 until the final loss of each of the above visual question answering models is less than the set value.

[0104] Step S6: When the visual question answering task arrives, input the above visual question answering task into each of the above visual question answering models to obtain multiple text answers. Calculate the similarity between the image and each text answer based on the CLIP model, and take the text answer with the highest similarity as the final answer.

[0105] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0106] This application also provides a training apparatus for a visual question-answering model. It should be noted that the training apparatus for the visual question-answering model in this application can be used to execute the training method for the visual question-answering model provided in this application. This apparatus is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0107] The following describes the training device for the visual question-answering model provided in the embodiments of this application.

[0108] Figure 4 This is a structural block diagram of a training device for a visual question-answering model according to an embodiment of this application. Figure 4 As shown, the device includes:

[0109] The first acquisition unit 10 is used to perform a first acquisition step to acquire multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input images in each image-answer pair are the same. The answers in different image-answer pairs are obtained by inputting the input images into different visual question-answering models. The visual question-answering models include Web-QA, V-Doc, TGIF-QA, and NuScenes-QA.

[0110] Specifically, during the training process, an image to be used for visual question answering is first selected, i.e., the input image mentioned above. Then, the input image is input into the visual question answering model to obtain the answer, i.e., the text answer mentioned above. Then, the image answer pair is constructed based on the input image and the text answer mentioned above.

[0111] In practice, the more visual question answering models set up in this application, the higher the accuracy of the final trained comprehensive model. However, as the number of visual question answering models increases, the training difficulty and computational complexity also increase. Therefore, this application sets up at least Web-QA, V-Doc, TGIF-QA and NuScenes-QA as the aforementioned visual question answering models to ensure the performance of the comprehensive visual question answering model. Models can be added and trained according to the requirements for question answering accuracy.

[0112] The first input unit 20 is used to input steps, and input the above image answers into the CLIP model to obtain the corresponding loss function values ​​to obtain multiple first loss values;

[0113] Specifically, to address the issue that existing models tend to fit text and have a low degree of integration between images and text, this application proposes to use the CLIP model to perform text alignment on the basis of a basic visual question answering model, thereby enhancing the integration of text and images in the visual question answering model. Then, based on the similarity between text and images, the loss value of the loss function is determined to obtain the aforementioned first loss value.

[0114] The second acquisition unit 30 is used to perform the second acquisition step, acquire the standard answer, and calculate the BLEU between the text answer and the standard answer in each of the above image answer pairs to obtain multiple second loss values;

[0115] Specifically, a BLEU value is calculated based on the standard answer and the text answers output by each of the aforementioned visual question answering models. This value is used to characterize the similarity between the text answer and the standard answer, and serves as the second loss value.

[0116] The calculation unit 40 is used for the calculation step, which performs a weighted calculation based on each of the first loss values ​​and the corresponding second loss values ​​to obtain multiple target loss values;

[0117] Specifically, by combining the similarity between the input images and corresponding text answers of each of the above-mentioned visual question answering models, as well as the similarity between the text answers and standard answers of the above-mentioned visual question answering models, the loss value of each visual question answering model in the integrated visual question answering model is determined to obtain the above-mentioned target loss value.

[0118] The third acquisition unit 50 is used to perform the third acquisition step, acquire multiple model parameter sets, and correct each of the above-mentioned model parameter sets according to each of the above-mentioned target loss values ​​through the ADAM algorithm to obtain multiple corrected parameter sets. The multiple above-mentioned model parameter sets include model parameters of different above-mentioned visual question answering models.

[0119] Specifically, in order to enhance the connection between the input image and the text answer, this application sets the target loss value as a guide, replacing the loss value calculated by the loss function in the prior art, and adjusts the parameters of each of the above visual question answering models through the ADAM algorithm to minimize the target loss value.

[0120] The repeating unit 60 is used to repeat the first acquisition step, the input step, the second acquisition step, the calculation step, and the third acquisition step at least once in sequence until each of the target loss values ​​is less than the first threshold.

[0121] Specifically, if the target loss value corresponding to the above-mentioned visual question answering model is greater than the preset threshold, it is determined that the accuracy of the above-mentioned visual question answering model has not met the usage requirements. The first acquisition step, input step, second acquisition step, calculation step and third acquisition step are repeated. Based on the above-mentioned target loss value, each visual question answering model is further optimized by the ADAM algorithm until the target loss value corresponding to all visual question answering models is less than the above-mentioned first threshold, and it is determined that the accuracy of the comprehensive visual question answering model has met the requirements.

[0122] In practice, as the number of iterations increases, the optimization level of the model gradually decreases. To avoid wasting computing resources, this application sets a maximum number of iterations. When the maximum number of iterations is reached, the modification of the above visual question answering model is terminated.

[0123] In the above embodiments, the first acquisition unit performs a first acquisition step to acquire multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input images in each image-answer pair are the same, and the answers in different image-answer pairs are obtained by inputting the input images into different visual question-answering models, including Web-QA, V-Doc, TGIF-QA, and NuScenes-QA. The first input unit performs an input step to input each image-answer pair into a CLIP model to obtain the corresponding loss function values ​​and multiple first loss values. The second acquisition unit performs a second acquisition step to acquire the standard answer and calculate the first loss value in each image-answer pair. The BLEU between the above text answer and the above standard answer yields multiple second loss values; the calculation unit performs a calculation step, performing weighted calculations based on each of the above first loss values ​​and the corresponding above second loss values ​​to obtain multiple target loss values; the third acquisition unit performs a third acquisition step, acquiring multiple model parameter sets, and correcting each of the above model parameter sets using the ADAM algorithm based on each of the above target loss values ​​to obtain multiple corrected parameter sets, the multiple above model parameter sets including model parameters of different above visual question answering models; the repeating unit sequentially repeats the above first acquisition step, the above input step, the above second acquisition step, the above calculation step, and the above third acquisition step at least once, until each of the above target loss values ​​is less than the first threshold. This application is based on multiple existing question-answering models, providing visual question-answering answers based on the input image. It uses the CLIP model for text and image alignment, and then calculates the total loss value based on the loss values ​​from both text and image aspects. The model parameters of each model are updated based on the loss value to improve the accuracy of each model. Compared to existing technologies that use two network structures to process images and text separately, this application uses the CLIP model for text alignment and fuses text and image information. This solves the problem in existing visual question-answering models that tend to fit text more closely and have lower image fitting, resulting in poor robustness.

[0124] To determine the text answer of the aforementioned integrated visual question-answering model, in one optional embodiment, the apparatus further includes:

[0125] The fourth acquisition unit is used to acquire a first target image after repeating the first acquisition step, input step, second acquisition step, calculation step and third acquisition step at least once in sequence until each of the above target loss values ​​is less than the first threshold. The first target image is the above input image to be used for question answering.

[0126] Specifically, upon receiving a visual question-answering task, the image in the visual question-answering task is extracted to obtain the aforementioned first target image, and then the aforementioned first target image is...

[0127] The second input unit is used to input the first target image into each of the above-mentioned visual question answering models to obtain the corresponding text answer.

[0128] Specifically, the first target image is input into the integrated visual question answering model. The integrated visual question answering model processes the first target image and then inputs it into each of the visual question answering models in sequence. The output results of each of the visual question answering models are collected to obtain multiple text answers.

[0129] The construction unit is used to construct the image answer pairs based on each of the above-mentioned text answers and the above-mentioned first target image to obtain multiple target image answer pairs;

[0130] Specifically, based on the text answer of the above visual question answering model and the above first target image, the above image answer pair is constructed to obtain multiple above target image answer pairs.

[0131] The third input unit is used to input each of the above target image answer pairs into the above CLIP model to obtain multiple first similarities, wherein the first similarity is the cosine similarity between the first target image and the text answer in each of the above target image answer pairs;

[0132] Specifically, each of the aforementioned target image answer pairs is input into the CLIP model. The CLIP model encodes the text answer and the first target image in each target image answer pair to obtain corresponding text vectors and image vectors. Then, cosine similarity is calculated based on the text vectors and image vectors to obtain the aforementioned first similarity.

[0133] The determining unit is used to determine the text answer in the target image answer pair with the highest first similarity as the target answer and output it.

[0134] Specifically, the text answer in the target image answer pair with the highest first similarity is the answer with the highest accuracy. Therefore, the text answer is determined as the final text answer, that is, the target answer is output to the user.

[0135] In order to calculate the first loss value, in one optional implementation, the first input unit includes:

[0136] The first input module is used to input the input image from the above image answer pair into the first encoder to obtain a first vector, and the first encoder is used to encode the above input image into a vector representation;

[0137] Specifically, the input image is fed into a first encoder, which preprocesses the input image and encodes it to obtain a vector representing the input image, resulting in the first vector. In specific implementations, the first encoder can employ CNN, Autoencoder, Sparse coding, VAE, or Feature extractor, among others.

[0138] The second input module is used to input the text answer from the above image answer pair into the second encoder to obtain the second vector. The second encoder is used to encode the above text answer and convert it into a vector representation.

[0139] Specifically, the text answer is input into a second encoder, which segments, processes, and encodes the text answer to obtain the second vector representing the text answer. In specific implementations, the second encoder can employ TF-IDF, Word2Vec, GloVe, Doc2Vec, and Universal Sentence Encoder, among others.

[0140] The first calculation module is used to calculate the cosine similarity based on the first vector and the second vector to obtain the target similarity, and to substitute the target similarity into the first preset formula to obtain the first loss value.

[0141] Specifically, let the first vector be A and the second vector be B, and then substitute the first vector and the second vector into the formula text. Similarity(A，B) =Cosine Similarity{f(A), g(B)}, that is, calculate the cosine similarity of the first vector and the second vector to obtain the target similarity. Then, let the first loss value be L1, and the first preset formula is L = -log{softmax[Sim(A,B)] / T}, where Sim(A,B) is the target similarity and T is the temperature parameter. Substituting the target similarity into the first preset formula will give the first loss value.

[0142] To obtain the aforementioned first vector, in one optional implementation, the first input module includes:

[0143] The cropping submodule is used to crop the input image in the above image answer pair to a preset size to obtain the first image;

[0144] Specifically, the encoder in the CLIP model first resizes the input image to the size required by the encoder to obtain the first image mentioned above. Depending on the encoder used, the corresponding size is different, usually 224*224 or 256*256 pixels to obtain the image.

[0145] The calculation submodule is used to normalize the pixel values ​​of each pixel block in the first image to obtain the second image;

[0146] Specifically, for ease of encoding, after cropping the input image, this application sets up a method to standardize the pixel values ​​of the first image to obtain the second image. In specific implementations, this can be to scale the pixel values ​​to between 0 and 1, or to perform zero-mean and standardization processing.

[0147] The processing submodule is used to perform data augmentation on the second image to obtain a third image, wherein the data augmentation includes at least random cropping, flipping, and rotation.

[0148] Specifically, to enhance the robustness and generalization ability of the model, this application sets up a method to perform random cropping, horizontal flipping, or rotation on the second image after adjusting the pixel values ​​of the image to obtain the third image.

[0149] The encoding submodule is used to encode the third image and convert it into a vector form to obtain the first vector.

[0150] Specifically, the first vector is obtained by encoding the third image using an encoder.

[0151] To obtain the aforementioned second loss value, in one optional implementation, the second acquisition unit includes:

[0152] The word segmentation module is used to segment the above text answer and the above standard answer into words and combine them to obtain multiple subtexts. The number of words in each of the above subtexts is a preset number.

[0153] Specifically, firstly, each of the above text answers and the above standard answers is segmented into multiple grams. Then, the number of segments n contained in the above subtext is set, and consecutive above grams are combined to obtain multiple n-grams, which is the above subtext.

[0154] The first determining module is used to determine the number of times each of the aforementioned subtexts appears in the aforementioned text answer by matching each of the aforementioned subtexts with the aforementioned text answer to obtain multiple first statistical values, and to determine the number of times each of the aforementioned subtexts appears in the aforementioned standard answer by matching each of the aforementioned subtexts with the aforementioned standard answer to obtain second statistical values;

[0155] Specifically, by matching each of the above n-grams with each of the above text answers, the frequency of each n-gram in each of the above text answers is determined, thus obtaining the first statistical value. Then, by matching each of the above n-grams with each of the above standard answers, the frequency of each n-gram in each of the above standard answers is determined, thus obtaining the second statistical value.

[0156] The second determining module is used to determine a third statistical value based on the above text answer, and to calculate a plurality of fourth statistical values ​​by calculating the ratio of each of the above first statistical values ​​and the above third statistical values. The above third statistical value is the number of the above sub-texts contained in the above text answer.

[0157] Specifically, the number of the aforementioned n-grams contained in each of the aforementioned text answers is determined to obtain the aforementioned third statistical value. The difference between the aforementioned first statistical value and the aforementioned third statistical value is then calculated to obtain the aforementioned fourth statistical value.

[0158] The second calculation module is used to substitute each of the above-mentioned fourth statistical values ​​into the second preset formula to obtain the alternative BLEU corresponding to the above-mentioned text answer;

[0159] Specifically, substituting the fourth statistical value mentioned above into the formula BLEU=BP*exp(1 / n*∑(1to n)w n *log(p n ), where BP short text penalty, w n The weight is usually 1 / n, p n为 The fourth statistical value mentioned above.

[0160] The third determining module is used to determine the fifth statistical value based on the above standard answer. The fifth statistical value is the number of the above sub-texts contained in the above standard answer.

[0161] Specifically, the fifth statistical value is obtained by determining the number of the aforementioned n-grams contained in the above standard answer.

[0162] The third calculation module is used to calculate the difference between the third statistical value and the fifth statistical value to obtain the sixth statistical value, calculate the harmonic mean of the candidate BLEUs to obtain the seventh statistical value, and calculate the product of the sixth statistical value and the seventh statistical value to obtain the second loss value.

[0163] Specifically, based on the third and fifth statistical values ​​mentioned above, the length difference between the text answer and the standard answer is calculated to obtain the sixth statistical value. The harmonic mean is calculated based on the candidate BLEU to obtain the seventh statistical value. Then, the sixth and seventh statistical values ​​are used to calculate the second loss value L2.

[0164] In order to update the aforementioned visual question-answering models, in one optional implementation, the third acquisition unit includes:

[0165] The fourth calculation module is used to calculate the gradient of the target loss value with respect to the model parameter set of the corresponding visual question answering model using the backpropagation algorithm to obtain the target gradient.

[0166] Specifically, based on the target loss value, the gradient of the visual question answering model is calculated layer by layer using the gradient descent method and the reverberation propagation algorithm to obtain multiple target gradients.

[0167] The update module is used to update the model parameter set according to the target gradient using the ADAM algorithm to obtain the corresponding corrected parameter set.

[0168] Specifically, the above-mentioned modified parameter set is obtained by updating the corresponding parameters according to the target gradient of each layer of the model based on the ADAM algorithm described above.

[0169] To obtain the target loss value, in one optional implementation, the calculation unit includes:

[0170] The acquisition module is used to acquire a first weight and a second weight, wherein the first weight is the weight corresponding to the first loss value and the second weight is the weight corresponding to the second loss value;

[0171] Specifically, the first weight mentioned above is obtained and set as W1, and the second weight mentioned above is obtained and set as W2.

[0172] The fifth calculation module is used to calculate the sum of the product of the first loss value and the first weight and the product of the second loss value and the second weight to obtain the target loss value.

[0173] Specifically, by substituting the first loss value, the first weight, the second weight, and the second loss value into the formula L = L1 * W1 + L2 * W2, the target loss value L is obtained.

[0174] The training device for the aforementioned visual question-answering model includes a processor and a memory. The first acquisition unit, first input unit, second acquisition unit, calculation unit, third acquisition unit, and repetition unit are all stored as program units in the memory. The processor executes these program units stored in the memory to implement the corresponding functions. All of the above modules reside in the same processor; alternatively, the modules may be located in different processors in any combination.

[0175] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured; adjusting kernel parameters can improve the robustness and accuracy of the visual question-answering model.

[0176] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0177] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the training method for the visual question-answering model.

[0178] Specifically, the training methods for visual question answering models include:

[0179] Step S201, the first acquisition step, acquires multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input image in each image-answer pair is the same. The answers in different image-answer pairs are obtained by inputting the input image into different visual question-answering models. The visual question-answering models include Web-QA, V-Doc, TGIF-QA, and NuScenes-QA.

[0180] Specifically, during the training process, an image to be used for visual question answering is first selected, i.e., the input image mentioned above. Then, the input image is input into the visual question answering model to obtain the answer, i.e., the text answer mentioned above. Then, the image answer pair is constructed based on the input image and the text answer mentioned above.

[0181] In practice, the more visual question answering models set up in this application, the higher the accuracy of the final trained comprehensive model. However, as the number of visual question answering models increases, the training difficulty and computational complexity also increase. Therefore, this application sets up at least Web-QA, V-Doc, TGIF-QA and NuScenes-QA as the aforementioned visual question answering models to ensure the performance of the comprehensive visual question answering model. Models can be added and trained according to the requirements for question answering accuracy.

[0182] Step S202: Input the above image answer pairs into the CLIP model to obtain the corresponding loss function values ​​and obtain multiple first loss values;

[0183] Specifically, to address the issue that existing models tend to fit text and have a low degree of integration between images and text, this application proposes to use the CLIP model to perform text alignment on the basis of a basic visual question answering model, thereby enhancing the integration of text and images in the visual question answering model. Then, based on the similarity between text and images, the loss value of the loss function is determined to obtain the aforementioned first loss value.

[0184] Step S203, the second acquisition step, is to acquire the standard answer and calculate the BLEU between the text answer and the standard answer in each of the above image answer pairs to obtain multiple second loss values;

[0185] Specifically, a BLEU value is calculated based on the standard answer and the text answers output by each of the aforementioned visual question answering models. This value is used to characterize the similarity between the text answer and the standard answer, and serves as the second loss value.

[0186] Step S204, calculation step: multiple target loss values ​​are obtained by weighted calculation based on each of the above-mentioned first loss values ​​and the corresponding above-mentioned second loss values;

[0187] Specifically, by combining the similarity between the input images and corresponding text answers of each of the above-mentioned visual question answering models, as well as the similarity between the text answers and standard answers of the above-mentioned visual question answering models, the loss value of each visual question answering model in the integrated visual question answering model is determined to obtain the above-mentioned target loss value.

[0188] Step S205, the third acquisition step, acquire multiple model parameter sets, and correct each of the above-mentioned model parameter sets according to the above-mentioned target loss values ​​through the ADAM algorithm to obtain multiple corrected parameter sets. The multiple above-mentioned model parameter sets include model parameters of different above-mentioned visual question answering models.

[0189] Specifically, in order to enhance the connection between the input image and the text answer, this application sets the target loss value as a guide, replacing the loss value calculated by the loss function in the prior art, and adjusts the parameters of each of the above visual question answering models through the ADAM algorithm to minimize the target loss value.

[0190] Step S206: Repeat the first acquisition step, the input step, the second acquisition step, the calculation step, and the third acquisition step at least once in sequence until each of the target loss values ​​is less than the first threshold.

[0191] Specifically, if the target loss value corresponding to the above-mentioned visual question answering model is greater than the preset threshold, it is determined that the accuracy of the above-mentioned visual question answering model has not met the usage requirements. The first acquisition step, input step, second acquisition step, calculation step and third acquisition step are repeated. Based on the above-mentioned target loss value, each visual question answering model is further optimized by the ADAM algorithm until the target loss value corresponding to all visual question answering models is less than the above-mentioned first threshold, and it is determined that the accuracy of the comprehensive visual question answering model has met the requirements.

[0192] In practice, as the number of iterations increases, the optimization level of the model gradually decreases. To avoid wasting computing resources, this application sets a maximum number of iterations. When the maximum number of iterations is reached, the modification of the above visual question answering model is terminated.

[0193] This invention provides a processor for running a program, wherein the program executes the training method of the visual question answering model.

[0194] Specifically, the training methods for visual question answering models include:

[0195] Step S201, the first acquisition step, acquires multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input image in each image-answer pair is the same. The answers in different image-answer pairs are obtained by inputting the input image into different visual question-answering models. The visual question-answering models include Web-QA, V-Doc, TGIF-QA, and NuScenes-QA.

[0196] Specifically, during the training process, an image to be used for visual question answering is first selected, i.e., the input image mentioned above. Then, the input image is input into the visual question answering model to obtain the answer, i.e., the text answer mentioned above. Then, the image answer pair is constructed based on the input image and the text answer mentioned above.

[0197] In practice, the more visual question answering models set up in this application, the higher the accuracy of the final trained comprehensive model. However, as the number of visual question answering models increases, the training difficulty and computational complexity also increase. Therefore, this application sets up at least Web-QA, V-Doc, TGIF-QA and NuScenes-QA as the aforementioned visual question answering models to ensure the performance of the comprehensive visual question answering model. Models can be added and trained according to the requirements for question answering accuracy.

[0198] Step S202: Input the above image answer pairs into the CLIP model to obtain the corresponding loss function values ​​and obtain multiple first loss values;

[0199] Specifically, to address the issue that existing models tend to fit text and have a low degree of integration between images and text, this application proposes to use the CLIP model to perform text alignment on the basis of a basic visual question answering model, thereby enhancing the integration of text and images in the visual question answering model. Then, based on the similarity between text and images, the loss value of the loss function is determined to obtain the aforementioned first loss value.

[0200] Step S203, the second acquisition step, is to acquire the standard answer and calculate the BLEU between the text answer and the standard answer in each of the above image answer pairs to obtain multiple second loss values;

[0201] Specifically, a BLEU value is calculated based on the standard answer and the text answers output by each of the aforementioned visual question answering models. This value is used to characterize the similarity between the text answer and the standard answer, and serves as the second loss value.

[0202] Step S204, calculation step: multiple target loss values ​​are obtained by weighted calculation based on each of the above-mentioned first loss values ​​and the corresponding above-mentioned second loss values;

[0203] Specifically, by combining the similarity between the input images and corresponding text answers of each of the above-mentioned visual question answering models, as well as the similarity between the text answers and standard answers of the above-mentioned visual question answering models, the loss value of each visual question answering model in the integrated visual question answering model is determined to obtain the above-mentioned target loss value.

[0204] Step S205, the third acquisition step, acquire multiple model parameter sets, and correct each of the above-mentioned model parameter sets according to the above-mentioned target loss values ​​through the ADAM algorithm to obtain multiple corrected parameter sets. The multiple above-mentioned model parameter sets include model parameters of different above-mentioned visual question answering models.

[0205] Specifically, in order to enhance the connection between the input image and the text answer, this application sets the target loss value as a guide, replacing the loss value calculated by the loss function in the prior art, and adjusts the parameters of each of the above visual question answering models through the ADAM algorithm to minimize the target loss value.

[0206] Step S206: Repeat the first acquisition step, the input step, the second acquisition step, the calculation step, and the third acquisition step at least once in sequence until each of the target loss values ​​is less than the first threshold.

[0207] Specifically, if the target loss value corresponding to the above-mentioned visual question answering model is greater than the preset threshold, it is determined that the accuracy of the above-mentioned visual question answering model has not met the usage requirements. The first acquisition step, input step, second acquisition step, calculation step and third acquisition step are repeated. Based on the above-mentioned target loss value, each visual question answering model is further optimized by the ADAM algorithm until the target loss value corresponding to all visual question answering models is less than the above-mentioned first threshold, and it is determined that the accuracy of the comprehensive visual question answering model has met the requirements.

[0208] In practice, as the number of iterations increases, the optimization level of the model gradually decreases. To avoid wasting computing resources, this application sets a maximum number of iterations. When the maximum number of iterations is reached, the modification of the above visual question answering model is terminated.

[0209] This invention provides a visual question-answering system, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0210] Step S201, the first acquisition step, acquires multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input image in each image-answer pair is the same. The answers in different image-answer pairs are obtained by inputting the input image into different visual question-answering models. The visual question-answering models include Web-QA, V-Doc, TGIF-QA, and NuScenes-QA.

[0211] Step S202: Input the above image answer pairs into the CLIP model to obtain the corresponding loss function values ​​and obtain multiple first loss values;

[0212] Step S203, the second acquisition step, is to acquire the standard answer and calculate the BLEU between the text answer and the standard answer in each of the above image answer pairs to obtain multiple second loss values;

[0213] Step S204, calculation step: multiple target loss values ​​are obtained by weighted calculation based on each of the above-mentioned first loss values ​​and the corresponding above-mentioned second loss values;

[0214] Step S205, the third acquisition step, acquire multiple model parameter sets, and correct each of the above-mentioned model parameter sets according to the above-mentioned target loss values ​​through the ADAM algorithm to obtain multiple corrected parameter sets. The multiple above-mentioned model parameter sets include model parameters of different above-mentioned visual question answering models.

[0215] Step S206: Repeat the first acquisition step, the input step, the second acquisition step, the calculation step, and the third acquisition step at least once in sequence until each of the target loss values ​​is less than the first threshold.

[0216] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0217] Step S201, the first acquisition step, acquires multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input image in each image-answer pair is the same. The answers in different image-answer pairs are obtained by inputting the input image into different visual question-answering models. The visual question-answering models include Web-QA, V-Doc, TGIF-QA, and NuScenes-QA.

[0218] Step S202: Input the above image answer pairs into the CLIP model to obtain the corresponding loss function values ​​and obtain multiple first loss values;

[0219] Step S203, the second acquisition step, is to acquire the standard answer and calculate the BLEU between the text answer and the standard answer in each of the above image answer pairs to obtain multiple second loss values;

[0220] Step S204, calculation step: multiple target loss values ​​are obtained by weighted calculation based on each of the above-mentioned first loss values ​​and the corresponding above-mentioned second loss values;

[0221] Step S205, the third acquisition step, acquire multiple model parameter sets, and correct each of the above-mentioned model parameter sets according to the above-mentioned target loss values ​​through the ADAM algorithm to obtain multiple corrected parameter sets. The multiple above-mentioned model parameter sets include model parameters of different above-mentioned visual question answering models.

[0222] Step S206: Repeat the first acquisition step, the input step, the second acquisition step, the calculation step, and the third acquisition step at least once in sequence until each of the target loss values ​​is less than the first threshold.

[0223] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0224] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0225] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0226] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0227] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0228] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0229] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0230] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0231] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0232] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0233] 1) The training method of the visual question answering model of this application firstly involves performing a first acquisition step to acquire multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input images in each image-answer pair are the same, while the answers in different image-answer pairs are obtained by inputting the input images into different visual question answering models, including Web-QA, V-Doc, TGIF-QA, and NuScenes-QA. Then, an input step is performed to input each image-answer pair into a CLIP model to obtain the corresponding loss function values ​​and multiple first loss values. After that, a second acquisition step is performed to acquire the standard answer and calculate the image answer for each pair. Multiple second loss values ​​are obtained by BLEU between the above-mentioned text answer and the above-mentioned standard answer; then, a calculation step is performed to calculate multiple target loss values ​​by weighting each of the above-mentioned first loss values ​​and the corresponding above-mentioned second loss values; then, a third acquisition step is performed to acquire multiple model parameter sets, and each of the above-mentioned model parameter sets is corrected by the ADAM algorithm according to each of the above-mentioned target loss values ​​to obtain multiple corrected parameter sets, the multiple above-mentioned model parameter sets including model parameters of different above-mentioned visual question answering models; finally, the above-mentioned first acquisition step, the above-mentioned input step, the above-mentioned second acquisition step, the above-mentioned calculation step and the above-mentioned third acquisition step are repeated at least once in sequence until each of the above-mentioned target loss values ​​is less than a first threshold. This application is based on multiple existing question-answering models, providing visual question-answering answers based on the input image. It uses the CLIP model for text and image alignment, and then calculates the total loss value based on the loss values ​​from both text and image aspects. The model parameters of each model are updated based on the loss value to improve the accuracy of each model. Compared to existing technologies that use two network structures to process images and text separately, this application uses the CLIP model for text alignment and fuses text and image information. This solves the problem in existing visual question-answering models that tend to fit text more closely and have lower image fitting, resulting in poor robustness.

[0234] 2) The training device for the visual question answering model of this application includes a first acquisition unit that performs a first acquisition step to acquire multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input images in each image-answer pair are the same, and the answers in different image-answer pairs are obtained by inputting the input images into different visual question answering models, including Web-QA, V-Doc, TGIF-QA, and NuScenes-QA. The first input unit performs an input step to input each image-answer pair into a CLIP model to obtain the corresponding loss function values ​​and obtain multiple first loss values. The second acquisition unit performs a second acquisition step to acquire the standard answer and calculate the first loss value for each image-answer pair. The BLEU between the text answer and the standard answer in the answer pair yields multiple second loss values; the calculation unit performs a calculation step, weighting each of the first loss values ​​and the corresponding second loss values ​​to obtain multiple target loss values; the third acquisition unit performs a third acquisition step, acquiring multiple model parameter sets, and correcting each of the model parameter sets using the ADAM algorithm based on each of the target loss values ​​to obtain multiple corrected parameter sets, the multiple model parameter sets including model parameters of different visual question answering models; the repeating unit repeats the first acquisition step, the input step, the second acquisition step, the calculation step, and the third acquisition step at least once in sequence until each of the target loss values ​​is less than the first threshold. This application is based on multiple existing question-answering models, providing visual question-answering answers based on the input image. It uses the CLIP model for text and image alignment, and then calculates the total loss value based on the loss values ​​from both text and image aspects. The model parameters of each model are updated based on the loss value to improve the accuracy of each model. Compared to existing technologies that use two network structures to process images and text separately, this application uses the CLIP model for text alignment and fuses text and image information. This solves the problem in existing visual question-answering models that tend to fit text more closely and have lower image fitting, resulting in poor robustness.

[0235] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for a visual question-answering model, characterized in that, include: The first acquisition step involves acquiring multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input image is consistent across all image-answer pairs. The answers in different image-answer pairs are obtained by inputting the input image into different visual question-answering models. The visual question-answering models include Web-QA, V-Doc, TGIF-QA, and NuScenes-QA. The input step involves inputting each image answer into the CLIP model to obtain the corresponding loss function value and thus multiple first loss values. The second acquisition step involves obtaining the standard answer and calculating multiple second loss values ​​by comparing the text answer and the standard answer in each image-answer pair. The calculation steps involve weighting each of the first loss values ​​and the corresponding second loss values ​​to obtain multiple target loss values; The third step involves obtaining multiple model parameter sets and correcting each model parameter set using the ADAM algorithm based on the target loss value to obtain multiple corrected parameter sets. The multiple model parameter sets include model parameters of different visual question answering models. Repeat the first acquisition step, input step, second acquisition step, calculation step and third acquisition step at least once in sequence until each of the target loss values ​​is less than the first threshold. After repeating the first acquisition step, input step, second acquisition step, calculation step and third acquisition step at least once in sequence until each of the target loss values ​​is less than the first threshold, the method further includes: acquiring a first target image, wherein the first target image is the input image to be question-answered; The first target image is input into each of the visual question answering models to obtain the corresponding text answer; image answer pairs are constructed based on each text answer and the first target image to obtain multiple target image answer pairs; each target image answer pair is input into the CLIP model to obtain multiple first similarities, where the first similarity is the cosine similarity between the first target image and the text answer in each target image answer pair; the text answer in the target image answer pair with the largest first similarity is determined as the target answer and output.

2. The method according to claim 1, characterized in that, The image answers are input into the CLIP model to obtain the corresponding loss function values, resulting in multiple first loss values, including: The input image in the image-answer pair is input into a first encoder to obtain a first vector. The first encoder is used to encode the input image into a vector representation. The text answer in the image answer pair is input into the second encoder to obtain the second vector. The second encoder is used to encode the text answer and convert it into a vector representation. The target similarity is obtained by calculating the cosine similarity based on the first vector and the second vector, and the target similarity is substituted into the first preset formula to obtain the first loss value.

3. The method according to claim 2, characterized in that, The input image in the image-answer pair is input into the first encoder to obtain the first vector, including: The input image in the image answer pair is cropped to a preset size to obtain the first image; The second image is obtained by normalizing the pixel values ​​of each pixel block in the first image; The second image is augmented to obtain a third image, wherein the data augmentation includes at least random cropping, flipping, and rotation; The third image is encoded and converted into a vector form to obtain the first vector.

4. The method according to claim 1, characterized in that, Calculating the BLEU between the text answer and the standard answer in each of the image-answer pairs yields several second loss values, including: The text answer and the standard answer are segmented and combined to obtain multiple subtexts, and the number of segments in each subtext is a preset number; Multiple first statistical values ​​are obtained by matching each of the subtexts with the text answer to determine the number of times each of the subtexts appears in the text answer; and a second statistical value is obtained by matching each of the subtexts with the standard answer to determine the number of times each of the subtexts appears in the standard answer. A third statistical value is determined based on the text answer, and multiple fourth statistical values ​​are obtained by calculating the ratio of each first statistical value to the third statistical value. The third statistical value is the number of subtexts contained in the text answer. Substituting each of the fourth statistical values ​​into the second preset formula yields the alternative BLEU values ​​corresponding to the text answer; A fifth statistical value is determined based on the standard answer, wherein the fifth statistical value is the number of subtexts contained in the standard answer; The difference between the third and fifth statistical values ​​is used to obtain the sixth statistical value. The harmonic mean of the candidate BLEUs is calculated to obtain the seventh statistical value. The product of the sixth and seventh statistical values ​​is used to obtain the second loss value.

5. The method according to claim 1, characterized in that, Based on the target loss values, the ADAM algorithm is used to correct the model parameter sets, including: The target gradient is obtained by calculating the gradient of the target loss value with respect to the model parameter set of the corresponding visual question answering model using the backpropagation algorithm. The corrected parameter set is obtained by updating the model parameter set according to the target gradient using the ADAM algorithm.

6. The method according to claim 1, characterized in that, Multiple target loss values ​​are obtained by weighting each of the first loss values ​​and the corresponding second loss values, including: Obtain a first weight and a second weight, wherein the first weight is the weight corresponding to the first loss value, and the second weight is the weight corresponding to the second loss value; The target loss value is obtained by summing the product of the first loss value and the first weight and the product of the second loss value and the second weight.

7. A training device for a visual question-answering model, characterized in that, The device includes: The first acquisition unit is used to perform a first acquisition step to acquire multiple image-answer pairs. Each image-answer pair includes an input image and a text answer. The input image in each image-answer pair is the same. The answer in different image-answer pairs is obtained by inputting the input image into different visual question-answering models. The visual question-answering models include Web-QA, V-Doc, TGIF-QA, and NuScenes-QA. The first input unit is used to input the steps, and inputs the image answer pairs into the CLIP model to obtain the corresponding loss function values ​​to obtain multiple first loss values; The second acquisition unit is used to perform the second acquisition step, acquire the standard answer, and calculate the BLEU between the text answer and the standard answer in each image answer pair to obtain multiple second loss values; The calculation unit is used to calculate multiple target loss values ​​by weighting each of the first loss values ​​and the corresponding second loss values. The third acquisition unit is used to perform the third acquisition step, acquire multiple model parameter groups, and correct each model parameter group according to each target loss value through the ADAM algorithm to obtain multiple corrected parameter groups. The multiple model parameter groups include model parameters of different visual question answering models. The repeating unit is used to repeat the first acquisition step, the input step, the second acquisition step, the calculation step and the third acquisition step at least once in sequence until each of the target loss values ​​is less than the first threshold. The apparatus further includes: a fourth acquisition unit, configured to acquire a first target image after sequentially repeating the first acquisition step, input step, second acquisition step, calculation step, and third acquisition step at least once until each target loss value is less than a first threshold, wherein the first target image is the input image to be used for question answering; a second input unit, configured to input the first target image into each of the visual question answering models to obtain the corresponding text answer; a construction unit, configured to construct image answer pairs based on each text answer and the first target image to obtain multiple target image answer pairs; a third input unit, configured to input each target image answer pair into the CLIP model to obtain multiple first similarities, wherein the first similarity is the cosine similarity between the first target image and the text answer in each target image answer pair; and a determination unit, configured to determine the text answer in the target image answer pair with the largest first similarity as the target answer and output it.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 6.

9. A visual question-answering system, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising methods for performing any one of claims 1 to 6.

Images