Model processing method and device, storage medium and computer program product

Abstract

The embodiment of the invention discloses a model training method and device, a storage medium and a computer program product. The model training method comprises the steps of selecting a high-information-content sample image corresponding to each type of standard visual answers based on visual uncertainty; based on the selected high-information-content sample image, constructing a sample set with standard visual answer distribution meeting expected answer distribution; carrying out region-level visual information labeling on each high-information-content sample image in the sample set; performing visual alignment training on the visual question and answer model by utilizing each labeled high-information-content sample image; wherein the visual question and answer model is used for understanding the image and answering a visual question proposed for the image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular, to a model processing method, device, storage medium, and computer program product. Background Art

[0002] Visual Question Answering (VQA) aims to accurately answer natural language questions based on visual images. Compared with traditional text question answering systems, visual question answering systems need to understand both image and text modal information at the same time. Visual question answering systems connect the two major fields of computer vision and natural language processing, enabling machines to have the ability to see and read at the same time like humans. Visual question answering technology has broad application prospects. For example, it can help visually impaired users perceive and understand the image world, provide road and direction guidance for self-driving cars, and assist in building an intelligent government service platform that integrates vision and language.

[0003] The existing classic visual question answering method is a method based on deep learning, which first projects the image and question text into the same feature space, then interacts with information in the feature space to obtain multimodal key features, and then predicts the answer based on the multimodal features. The above methods are usually optimized based on large-scale training data and empirical risk minimization objectives. Data in real scenes often have natural biases. For example, most bananas in supermarkets are yellow. Therefore, there will be modal bias in the manually collected training data, which is mainly reflected in the problem of excessive language prior knowledge (language bias), that is, most questions and answers have a strong correlation. This modal bias will make the model only need to follow the statistical rules between questions and answers to quickly achieve good performance on the training set, but the performance on the test set with a large distribution difference from the training set is very poor. The existing classic solution is to perform adversarial training based on counterfactual samples, so that the model can overcome the influence of language prior knowledge. The existing solution needs to introduce pixel-level fine-grained annotations when constructing counterfactual samples, or can only produce low-quality samples or pseudo-negative examples. For the above reasons, the current model optimization efficiency is low and the generalization ability of the model is poor. Summary of the invention

[0004] The embodiments of the present application provide a model processing method, device, storage medium and computer program product, which introduce visual information by actively learning to annotate sample images with high information content, enhance the model's ability to learn the alignment of multimodal content, and improve not only the model optimization efficiency but also the model generalization ability.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application embodiment provides a model processing method, including:

[0007] Select high-information sample images corresponding to each type of standard visual answer based on visual uncertainty;

[0008] Based on the selected high-information sample images, a sample set whose standard visual answer distribution meets the expected answer distribution is constructed;

[0009] For each high-information sample image in the sample set, performing region-level visual information annotation;

[0010] Using each labeled high-information sample image, visual alignment training is performed on a visual question answering model; wherein the visual question answering model is used to understand the image and answer visual questions posed to the image.

[0011] In the above method, selecting a high-information sample image corresponding to each type of standard visual answer based on visual uncertainty includes:

[0012] Inputting a first image corresponding to a first standard visual answer into the visual question answering model; wherein the first standard visual answer is a class of standard visual answers, and the first image is a sample image corresponding to the first standard visual answer;

[0013] Determining, by means of an attention mechanism of the visual question answering model, a visual attention distribution of an image region in the first image;

[0014] determining an uncertainty entropy of the first image based on a visual attention distribution of an image region in the first image;

[0015] If the uncertainty entropy of the first image is greater than or equal to a threshold, the first image is determined as a high-information sample image corresponding to a first standard visual answer.

[0016] In the above method, the step of constructing a sample set whose standard visual answer distribution satisfies the expected answer distribution based on the selected high-information sample images includes:

[0017] Construct a sampling set based on selected high-information samples;

[0018] For the sampling set, the difference between the standard visual answer distribution of the set and the expected answer distribution is minimized to obtain the sample set.

[0019] In the above method, the step of labeling each high-information sample image in the sample set with region-level visual information includes:

[0020] Obtaining the importance ranking of K image regions in the second image; wherein the importance of the image region is positively correlated with the correlation between the visual information of the image region and the standard visual answer, the second image is a high-information sample image in the sample set, the importance of the K image regions is greater than the importance of other image regions in the second image, and K is a positive integer;

[0021] The importance ranking is marked for the K image regions to complete region-level visual information labeling of the second image.

[0022] In the above method, using each labeled high-information sample image to perform visual alignment training on the visual question answering model includes:

[0023] Inputting the second image into the visual question answering model;

[0024] Determining the visual attention distribution of the K image regions through the attention mechanism of the visual question answering model;

[0025] According to the principle that the importance of the image region is positively correlated with the visual attention, the importance ranking and the visual attention distribution of the K image regions are compared to determine the visual loss function;

[0026] The attention mechanism of the visual question answering model is adjusted based on the visual loss function to complete visual alignment training of the visual question answering model based on the second image.

[0027] In the above method, after performing visual alignment training on the visual question answering model using each labeled high-information sample image, the method further includes:

[0028] Inputting the third image and the fourth image into the trained visual question answering model respectively; wherein the third image and the fourth image are respectively a high-information sample image in the sample set;

[0029] Determining visual attention distribution of image regions in the third image and the fourth image respectively through the attention mechanism of the trained visual question answering model;

[0030] Based on the visual attention distribution of the image regions in the third image and the fourth image, performing region-level fusion on the third image and the fourth image to generate a high-information counterfactual image;

[0031] The trained visual question answering model is adversarially trained using the high-information counterfactual image.

[0032] In the above method, the performing region-level fusion of the third image and the fourth image based on the visual attention distribution of the image regions in the third image and the fourth image to generate a high-information counterfactual image includes:

[0033] Based on the visual attention distribution of the image areas in the third image, determining M image areas from the third image; wherein M is a positive integer, and the visual attention of the M image areas is less than the visual attention of other image areas in the third image;

[0034] Based on the visual attention distribution of the image areas in the fourth image, determining N image areas from the fourth image; wherein N is a positive integer, and the visual attention of the N image areas is greater than the visual attention of other image areas in the fourth image;

[0035] The M image regions are fused with the N image regions to generate a high-information counterfactual image.

[0036] The present application provides a model processing device, including:

[0037] A selection module, used to select high-information sample images corresponding to each type of standard visual answer based on visual uncertainty;

[0038] A construction module, for constructing a sample set whose standard visual answer distribution satisfies the expected answer distribution based on the selected high-information sample images;

[0039] A labeling module, used for labeling each high-information sample image in the sample set with region-level visual information;

[0040] A training module is used to perform visual alignment training on a visual question answering model using each labeled high-information sample image; wherein the visual question answering model is used to understand the image and answer visual questions posed to the image.

[0041] The embodiment of the present application provides a model processing device, including: a processor, a memory and a communication bus;

[0042] The communication bus is used to realize the communication connection between the processor and the memory;

[0043] The processor is used to execute one or more computer programs stored in the memory to implement the above-mentioned model processing method.

[0044] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned model training method is implemented.

[0045] An embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned model processing method when being executed.

[0046] The embodiments of the present application provide a model training method, device, storage medium and computer program product, and the model processing method includes: selecting high-information sample images corresponding to each type of standard visual answer based on visual uncertainty; constructing a sample set whose standard visual answer distribution meets the expected answer distribution based on the selected high-information sample images; performing regional-level visual information annotation on each high-information sample image in the sample set; using each annotated high-information sample image to perform visual alignment training on the visual question-answering model; wherein the visual question-answering model is used to understand the image and answer visual questions posed to the image. The technical solution provided by the embodiments of the present application introduces visual information by actively learning to annotate high-information sample images, enhances the model's ability to learn the alignment of multimodal content, and not only improves the model optimization efficiency, but also improves the model generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 A schematic diagram of a framework of an exemplary visual question answering model provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a language bias problem in an exemplary visual question answering task provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of a framework of an exemplary model processing process provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of a framework of an exemplary model visual alignment process provided in an embodiment of the present application;

[0051] Figure 5 A flow chart of a model processing method provided in an embodiment of the present application;

[0052] Figure 6 An exemplary schematic diagram of sorting image regions provided in an embodiment of the present application;

[0053] Figure 7 A schematic diagram of a model processing device provided in an embodiment of the present application Figure 1

[0054] Figure 8 A schematic diagram of a model processing device provided in an embodiment of the present application Figure 2 . DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] The following will specifically describe the technical solution of the present application and how the technical solution of the present application solves the above technical problems through embodiments and in conjunction with the accompanying drawings. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0057] In addition, the technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0058] The model processing method provided in the embodiment of the present application is suitable for processing a visual question answering model. The relevant contents of the visual question answering are explained below.

[0059] The visual question answering system aims to answer natural language questions Q based on given visual images and give natural language answers A. Figure 1 As shown in Figure 1, the existing mainstream visual question answering model usually consists of four modules: question encoder, image encoder, multimodal feature fusion and answer predictor. The question encoder uses a recurrent neural network to extract question features q, while the image encoder extracts visual object features V = {v1, v2, ..., v |V|}, where v i represents the features of the i-th object in the image, and |V| represents the total number of objects identified in image I; the multimodal feature fuser first extracts visual features related to the question based on the attention mechanism, and then fuses them with the question features to obtain multimodal features; the answer predictor predicts the answer based on the multimodal features. The whole process can be expressed as P(A|I,Q)=f vqa (I,Q), where f_vqa(·) represents a visual question answering model whose input is question Q and image I and whose output is the distribution of answers; P(A|I,Q) represents the probability of predicting answer A given question Q and image I.

[0060] The visual question answering task can be viewed as a multi-classification problem in the answer space Ω, where each answer corresponds to a category, and the model is optimized using the multi-label soft probability cross entropy loss function shown in formula (1):

[0061]

[0062] Among them, I i Denotes a given image, Q i Represents a given problem i, a ij represents the j-th candidate answer to question i, represents the standard score of answer j on question i.

[0063] Ideally, a visual question-answering model should answer questions based on the content of the image and give the correct answer. The training data used to train the model usually has a serious imbalance problem, that is, the distribution of questions and answers in the training set is seriously imbalanced. For example, for the question "What color is the banana in the picture?", 90% of the standard answers are "yellow", only 5% of the standard answers are "green", 3% of the standard answers are "white", and 2% of the standard answers are "black". Based on this imbalanced training data, the model can achieve a high accuracy rate on the training set simply by learning the dependency between the question "What color is...?" and the answer "yellow" without looking at the image content. Figure 2 As shown in Figure 1, this model that over-relies on language prior knowledge and ignores the understanding of image content works well in scenarios with the same distribution as the training data, but performs poorly in scenarios with inconsistent distribution. This situation is called excessive language prior knowledge or language bias in visual question answering tasks.

[0064] The counterfactual sample method is currently the mainstream method to overcome language bias. Its essence is a data augmentation method. Specifically, the original sample in a given dataset<I,Q,A> (i.e. set the corresponding sample and the answer), if the image I ′ does not contain visual clues that support answering question Q with answer A, then image I is considered ′ is a counterfactual image, and its sample with the original question and answer QA ′ ,Q,A> is the counterfactual sample.

[0065] The model can predict the answer A for question Q if and only if image I contains the key information to answer question Q; and the counterfactual image I ′ If the model does not contain relevant information, the model should refuse to feedback the so-called correct answer A for question Q. Therefore, the model should minimize the probability that the model predicts the answer A for question Q given the counterfactual image, which is formalized as the counterfactual loss function shown in the following formula (2):

[0066]

[0067] Since the given image I contains relevant information to answer the question and image I' does not contain relevant information, and ​This leads to a completely opposite optimization direction, which allows the model to distinguish the subtle differences between image I and counterfactual image I'. By jointly optimizing the above two loss functions, the model can focus more on understanding the image content, thereby overcoming the model's reliance on language prior bias. The quality of the counterfactual image I' is crucial to the optimization goal. Too simple samples can make Fast convergence means that the model can easily distinguish counterfactual samples without deep understanding of the image content; however, too high difficulty or pseudo-negative examples may make it difficult for the loss function to converge. Therefore, generating high-quality counterfactual samples is crucial for training robust visual question answering models.

[0068] The technical solution of this application aims to promote the alignment learning of the two modalities of visual content and question text by introducing a small amount of visual supervision information and generating high-quality counterfactual samples, so that the model can answer questions based on the correct visual clues, thereby overcoming the dependence on prior language knowledge. Figure 3 As shown, the technical solution of this application may include two parts:

[0069] (1) Based on visual alignment with balanced active learning, a small amount of visual supervision signals with high information content are introduced, so that the model can focus on the correct image area when answering questions.

[0070] Affected by language prior knowledge, the model can answer questions without understanding the content of the image, which makes it difficult for the model to focus on the correct image area. Considering the serious language prior knowledge in the dataset and the lack of fine-grained visual supervision information, it is difficult for the model to focus on the correct key areas during the learning process. This application proposes a balanced active learning strategy to learn a more accurate text-visual alignment relationship by selectively providing fine-grained visual supervision for training.

[0071] Active learning aims to select as few data samples as possible from an unlabeled set for annotation, thereby maximizing the performance of the model. In visual question answering tasks, given a question-image pair, usually only the answer is annotated while ignoring the visual evidence marking, that is, which areas or objects in the image should be used as the basis for feeding back the correct answer. In the absence of visual evidence, it will be difficult to ensure that the trained model can focus on the correct image area. In this application, based on active learning, balanced samples with high information content are selected for region-level annotation, which can achieve more efficient visual alignment relationship learning. The specific framework is as follows: Figure 4 shown.

[0072] (2) Construct high-quality counterfactual samples for adversarial training. Based on the learned correct visual attention, a counterfactual sample construction algorithm based on regional feature fusion is designed to generate high-quality samples to ensure that the model can carry out efficient adversarial training, thereby helping the model to overcome language prior bias.

[0073] Based on the above content, the technical solution provided by this application is described in detail below.

[0074] Figure 5 A flow chart of a model processing method provided in an embodiment of the present application. Figure 5 As shown, in the embodiment of the present application, the model processing method mainly includes the following steps:

[0075] S101. Select high-information sample images corresponding to each type of standard visual answer based on visual uncertainty.

[0076] In an embodiment of the present application, the model processing device can select high-information sample images corresponding to each type of standard visual answer based on visual uncertainty.

[0077] It should be noted that in the embodiments of the present application, each type of standard visual answer is the same standard and correct visual answer generated for a question based on a type of image. For example, each of the five images contains a dog, and the dogs in the images are all white. The question is: "What color is the dog in the picture?" For these images, which are actually a type of image, the corresponding type of standard visual answer is "white". The specific standard visual answer and the number of standard visual answers can be selected or set according to actual needs and application scenarios, and are not limited in the embodiments of the present application.

[0078] It is understandable that not all image regions are useful when answering questions, so for the visual question answering model used to understand images and answer visual questions about images, a differentiated visual attention distribution should be given, where relevant regions should be allocated greater attention, while irrelevant regions should be allocated less attention. If the visual question answering model's attention distribution to each image region in a sample image is relatively smooth, that is, the visual evidence is weak, then the uncertainty of predicting the standard visual answer is greater. If such sample images are used, it is helpful to enhance the model's learning of the alignment relationship between text and visual image regions. Such sample images are high-information sample images corresponding to the standard visual answer.

[0079] In an embodiment of the present application, a model processing device selects a high-information sample image corresponding to each type of standard visual answer based on visual uncertainty, including: inputting a first image corresponding to a first standard visual answer into a visual question-answering model; wherein the first standard visual answer is a type of standard visual answer, and the first image is a sample image corresponding to the first standard visual answer; determining the visual attention distribution of an image area in the first image through the attention mechanism of the visual question-answering model; determining the uncertainty entropy of the first image based on the visual attention distribution of the image area in the first image; if the uncertainty entropy of the first image is greater than or equal to a threshold, determining the first image as a high-information sample image corresponding to the first standard visual answer.

[0080] It should be noted that in the embodiment of the present application, for the first standard visual answer, the model processing device can obtain a corresponding sample image, that is, the first image, and the correct answer to the question asked by the first image should be the first standard visual answer. The specific first image and the number of first images are not limited in the embodiment of the present application.

[0081] It should be noted that in an embodiment of the present application, the model processing device inputs the first image into the visual question answering model. The attention mechanism of the visual question answering model can determine the visual attention distribution of the image area in the first image. Based on this, the entropy of the visual attention distribution can be used to measure the uncertainty of its visual attention distribution. Entropy can capture the amount of information contained in the visual attention distribution, and reaches its maximum value when all image areas are equally likely. Specifically, the visual attention distribution of the first image is a=[a1,a2,...,a K ], the uncertainty entropy H of the first image can be calculated according to the following formula (3):

[0082]

[0083] Among them, a1, a2, ..., a K represents the visual attention of different image regions among the K image regions in the first image, that is, the attention score.

[0084] It should be noted that in the embodiments of the present application, sample images with larger entropy are selected as high-information sample images, that is, high-information sample images have a high degree of uncertainty, wherein the size of the uncertainty entropy can be measured by a set threshold, and the specific threshold can be set according to actual needs and application scenarios, and is not limited in the embodiments of the present application.

[0085] In an embodiment of the present application, for each type of visual standard answer, the above-mentioned method can be used to select the corresponding high-information sample image from the corresponding sample images. By way of example, the above-mentioned method can be used to select 5 high-information sample images corresponding to each type of visual standard answer. The specific number of high-information sample images selected is not limited in the embodiment of the present application.

[0086] S102: Based on the selected high-information sample images, construct a sample set whose standard visual answer distribution satisfies the expected answer distribution.

[0087] In an embodiment of the present application, after selecting high-information sample images corresponding to each type of standard visual answer based on visual uncertainty, the model processing device constructs a sample set whose standard visual answer distribution meets the expected answer distribution based on the selected high-information sample images.

[0088] It is understandable that a balanced sample set requires that the number of samples corresponding to each type of standard visual answer is equal. However, there is a serious imbalance in the number of current visual question-answering samples, where most questions correspond to high-frequency answers. To avoid oversampling of sample images corresponding to high-frequency answers, this application proposes a sampling strategy that takes into account the balance of answer distribution and constructs a sample set whose standard visual answer distribution meets the expected answer distribution.

[0089] In an embodiment of the present application, a model processing device constructs a sample set whose standard visual answer distribution satisfies the expected answer distribution based on selected high-information sample images, including: constructing a sampling set based on the selected high-information samples; for the sampling set, minimizing the difference between the standard visual answer distribution of the set and the expected answer distribution to obtain the sample set.

[0090] It should be noted that in the embodiment of the present application, by minimizing the difference between the standard visual answer distribution and the expected answer distribution of the sampling set, a sample image with a balanced answer category distribution is selected to obtain a sample set, as shown in the following formula (4):

[0091]

[0092] Among them, A T z represents the standard visual answer distribution of the sample set, z includes z i ∈{0,1} indicates whether the i-th high-information sample image in the sampling set is selected. A includes A i The answer vector represents the standard visual answer corresponding to the i-th high-information sample. Considering that the sample set construction and model enhancement can be multi-round, is the expected answer distribution of the cth sampling period for active learning, where w j =max[cb / Cn j,0] is the expected number of samples of the jth type of standard visual answer, b is the total number of samples to be sampled in the period, c is the number of sampling periods, n j is the total number of samples corresponding to the jth standard visual answer that has been sampled in the previous c-1 sampling cycles, and C represents the total number of standard visual answer categories. Through the above constraints on answer distribution, oversampling from certain specific answer categories can be avoided, thereby achieving balanced sampling and obtaining a sample set.

[0093] S103: perform region-level visual information annotation on each high-information sample image in the sample set.

[0094] In an embodiment of the present application, after constructing the sample set, the model processing device can perform region-level visual information annotation on each high-information sample image in the sample set.

[0095] In an embodiment of the present application, a model processing device performs region-level visual information annotation on each high-information sample image in a sample set, including: obtaining an importance ranking of K image regions in a second image; wherein the importance of an image region is positively correlated with the correlation between the visual information of the image region and a standard visual answer, the second image is a high-information sample image in the sample set, the importance of the K image regions is greater than the importance of other image regions in the second image, and K is a positive integer; marking the importance ranking of the K image regions to complete region-level visual information annotation of the second image.

[0096] It should be noted that in the embodiments of the present application, for high-information sample images in the sample set, such as the second image, the image regions can be sorted according to the question and the standard visual answer to annotate the image, thereby realizing region-level visual information annotation. Since it is difficult to assign an accurate attention score to each region, the image regions can be sorted according to the importance of each image region in answering the question. Figure 4 As shown, for the question "What color is the dog?" and the answer "yellow", the image region containing the "whole dog" can be the most important, followed by the image region containing only the "dog head", and so on, and the background region is the least important image region. Since there are background regions and a large number of irrelevant regions in the image, in order to speed up the labeling efficiency, only the importance ranking of the more important K image regions can be obtained. The model processing device can mark the importance ranking of the K image regions, and the ranked image regions can be expressed as {r1>r2>...>r k >[r k+1 ,...]}, where r1 to r kIndicates the labeled image area. Of course, K can also be the total number of image areas. In this case, the importance ranking and labeling of all image areas are obtained. When labeling, different K values ​​can be adaptively set for different images to improve efficiency.

[0097] S104. Perform visual alignment training on the visual question answering model using each labeled high-information sample image; wherein the visual question answering model is used to understand the image and answer visual questions posed to the image.

[0098] In an embodiment of the present application, after completing the region-level visual information annotation of each high-information sample image in the sample set, the model processing device uses each annotated high-information sample image to perform visual alignment training on the visual question answering model.

[0099] In an embodiment of the present application, a model processing device uses each labeled high-information sample image to perform visual alignment training on a visual question answering model, including: inputting a second image into the visual question answering model; determining the visual attention distribution of K image regions through the attention mechanism of the visual question answering model; according to the principle that the importance of the image region is positively correlated with the visual attention, comparing the importance ranking and the visual attention distribution of the K image regions, and determining a visual loss function; adjusting the attention mechanism of the visual question answering model based on the visual loss function, and completing the visual alignment training of the visual question answering model based on the second image.

[0100] It should be noted that, in an embodiment of the present application, based on the labeled high-information samples, the model processing device can align the attention of the visual question answering model and the image area annotation information through sorting learning. Taking into account that it is usually difficult to directly optimize the absolute value difference of the attention of different image regions, the visual question answering model can be trained to sort the image regions according to the importance ranking of the image regions presented by the region-level visual information annotation, so that more important regions can obtain greater visual attention, that is, higher attention scores. Exemplarily, given the importance ranking of image regions r1>r2, it means that image region r1 is more important than r2 when answering questions, and the attention score a1 of the visual question answering model to r1 should be greater than the attention score a2 to r2. Therefore, the visual loss function can be determined for adjusting the attention mechanism of the visual question answering model, and the visual loss function is shown in the following formula (5):

[0101]

[0102] In order to realize multiple image regions (the order of annotation is {r1>r2>...>r k >[r k+1 ,...,r K]}), the pair-wise ranking can be extended to list-wise ranking, and the visual loss function can be expressed as follows:

[0103]

[0104] Among them, the current top-ranked image region is recursively removed, and then the next image region is considered to be more important than the remaining other regions until all image regions are traversed.

[0105] In the examples of this application, see Figure 6 , which shows in detail the principle of performing sorting on multiple image regions. In this way, the model's attention and visual information can be aligned, so that the model focuses on the correct image region when answering questions.

[0106] It should be noted that, in the embodiment of the present application, combining the loss formula shown in formula (1) and formula (6), the total loss function can be expressed as the following formula (7):

[0107]

[0108] Among them, α is a given parameter.

[0109] In an embodiment of the present application, the model processing device can not only perform visual alignment training on the visual question answering model, but also perform adversarial training, which is described in detail below.

[0110] In an embodiment of the present application, after the model processing device uses each labeled high-information sample image to perform visual alignment training on the visual question answering model, it can also perform the following steps: input the third image and the fourth image into the trained visual question answering model respectively; wherein the third image and the fourth image are respectively a high-information sample image in the sample set; through the attention mechanism of the trained visual question answering model, determine the visual attention distribution of the image areas in the third image and the fourth image respectively; based on the visual attention distribution of the image areas in the third image and the fourth image, perform region-level fusion on the third image and the fourth image to generate a high-information counterfactual image; use the high-information counterfactual image to perform adversarial training on the trained visual question answering model.

[0111] It should be noted that the existing counterfactual image construction method synthesizes the negative image of the counterfactual sample by random sampling or masking the key area, which may introduce low-information counterfactual samples or pseudo-negative examples. Based on this, the present application proposes to use a regional fusion algorithm to generate more diverse high-quality counterfactual samples. Fusion (Mix-up) is a classic method in the field of data enhancement. Given a data distribution, it interpolates the anchor image x with the randomly selected image x' at the pixel level to obtain an enhanced sample of the anchor point x. Since it is difficult to ensure that the fused image is semantically explanatory and may introduce pseudo-negative examples or noise data, pixel-level fusion has certain limitations in constructing counterfactual samples. Considering this problem, the present application designs a fusion strategy based on image regions, combined with accurate image region attention distribution to generate higher quality negative images. The basic idea of ​​​​the method comes from the following facts: 1) Most of the existing high-performance models are based on fine-grained regional features extracted from pre-trained models; 2) If the given image does not contain key image areas that are irrelevant to the problem, the model should not derive the corresponding standard visual answer from it. Therefore, compared with pixel-level fusion, the present application proposes to perform region-level fusion of two images, namely the third image and the fourth image, to better control the quality of the interpolated image. The specific third image and the fourth image are not limited in the present embodiment.

[0112] In an embodiment of the present application, a model processing device performs region-level fusion of the third image and the fourth image based on the visual attention distribution of image areas in the third image and the fourth image to generate a high-information counterfactual image, including: determining M image areas from the third image based on the visual attention distribution of image areas in the third image; wherein M is a positive integer, and the visual attention of the M image areas is less than the visual attention of other image areas in the third image; determining N image areas from the fourth image based on the visual attention distribution of image areas in the fourth image; wherein N is a positive integer, and the visual attention of the N image areas is greater than the visual attention of other image areas in the fourth image; and fusing the M image areas with the N image areas to generate a high-information counterfactual image.

[0113] It can be understood that in the embodiment of the present application, the model processing device performs regional fusion of images based on the visual attention distribution of the image areas in the third image and the fourth image, wherein M image areas with greater visual attention in the third image are selected, and these M image areas have greater visual attention, that is, the key areas in the image, and N image areas with less visual attention in the fourth image are selected, and these N image areas have less visual attention, that is, the non-key areas in the image, such as the background area. Based on this, the two are fused to obtain a high-information counterfactual image. Specifically, the M image areas in the third image are represented by V +, the N image regions in the fourth image are represented as V - , the high-information counterfactual image I' is synthesized by region-level fusion using the following formula (8):

[0114] I ′ =V - ∪ V + (8)

[0115] Based on this, the region-level fusion proposed in this application can not only improve the efficiency of image generation, but also ensure higher quality of fused images and more stable training.

[0116] It can be understood that in the embodiment of the present application, since the high-information counterfactual image I' is the product of region-level fusion, the third image differs from I' in only some image regions, and the relevant loss function is shown in the above formula (2).

[0117] It can be understood that in the embodiments of the present application, following the counterfactual training strategy, and Different optimization directions can be guided respectively. The model needs to predict different answers based on the high-information counterfactual image and the image used to generate it. The model must distinguish the subtle differences between the images, and this process will further enhance the model's understanding of the image. Compared with the counterfactual images constructed based on the erasure strategy, the high-information counterfactual images constructed in the present application contain high-information image areas, which can avoid the situation where low-information samples cause training collapse. In addition, combined with the aforementioned visual alignment strategy based on active learning, high-quality samples can be generated based on more accurate attention distribution, avoiding the situation where traditional methods cause false negatives due to mistaken erasure and rubbing.

[0118] In summary, this solution provides an efficient and economical method to generate high-quality counterfactual samples, which can improve the model's ability to understand visual image information, promote robust model training, and overcome language bias. In addition to the visual alignment method based on fine-grained visual annotation information, adversarial training based on high-information counterfactual samples can also improve the model's ability to understand image content. Based on this, the above two strategies are combined to jointly improve the model effect. The joint loss function is shown in the following formula (9):

[0119]

[0120] Among them, α and β are given parameters.

[0121] The technical solution provided by the embodiment of the present application has the following advantages: First, the existing robust visual question-answering method based on visual information enhancement needs to rely on large-scale manual annotation information, and does not consider the distribution of the annotated data. The present application proposes a annotation scheme based on balanced active learning, which selects some difficult samples with balanced answer distribution in the data set for regional visual annotation, which not only improves the information gain brought by the annotated data but also improves the annotation efficiency; Second, the existing unbiased visual question-answering method based on counterfactual samples faces the situation of low-quality or pseudo-negative counterfactual samples, and the proposal proposes a regional-level image fusion strategy to construct high-quality counterfactual samples, which benefits from the precise visual alignment effect and attention distribution brought by active learning annotated data, so that the present application can produce high-quality counterfactual images that are globally similar to the original image but locally different, thereby effectively improving the effect of the unbiased visual question-answering method based on counterfactual samples. Compared with the integrated learning method that achieves good results on the test set by adjusting the sample weights, the present application introduces the alignment ability of multimodal content (image and text) by directly (active learning annotation) or indirectly (high-information counterfactual images) through the visual information enhancement model, which has better results and stronger interpretability.

[0122] An embodiment of the present application provides a model processing device. Figure 7 A schematic diagram of a model processing device provided in an embodiment of the present application Figure 1 .like Figure 7 As shown, in an embodiment of the present application, the model processing device 1 includes:

[0123] A selection module 11, configured to select a high-information sample image corresponding to each type of standard visual answer based on visual uncertainty;

[0124] A construction module 12 is used to construct a sample set whose standard visual answer distribution meets the expected answer distribution based on the selected high-information sample images;

[0125] A labeling module 13, used for labeling each high-information sample image in the sample set with region-level visual information;

[0126] The training module 14 is used to perform visual alignment training on the visual question answering model using each labeled high-information sample image; wherein the visual question answering model is used to understand the image and answer the visual questions posed to the image.

[0127] In one embodiment of the present application, the selection module 11 is used to input a first image corresponding to a first standard visual answer into the visual question answering model; wherein the first standard visual answer is a type of standard visual answer, and the first image is a sample image corresponding to the first standard visual answer; through the attention mechanism of the visual question answering model, the visual attention distribution of the image area in the first image is determined; based on the visual attention distribution of the image area in the first image, the uncertainty entropy of the first image is determined; if the uncertainty entropy of the first image is greater than or equal to a threshold, the first image is determined as a high-information sample image corresponding to the first standard visual answer.

[0128] In one embodiment of the present application, the construction module 12 is used to construct a sampling set based on selected high-information samples; for the sampling set, the difference between the standard visual answer distribution of the set and the expected answer distribution is minimized to obtain the sample set.

[0129] In one embodiment of the present application, the labeling module 13 is used to obtain the importance ranking of K image areas in the second image; wherein the importance of the image area is positively correlated with the correlation between the visual information of the image area and the standard visual answer, the second image is a high-information sample image in the sample set, the importance of the K image areas is greater than the importance of other image areas in the second image, and K is a positive integer; the importance ranking is marked for the K image areas to complete the region-level visual information labeling of the second image.

[0130] In one embodiment of the present application, the training module 14 is used to input the second image into the visual question answering model; determine the visual attention distribution of the K image regions through the attention mechanism of the visual question answering model; according to the principle that the importance of the image region is positively correlated with the visual attention, compare the importance ranking and the visual attention distribution of the K image regions to determine the visual loss function; adjust the attention mechanism of the visual question answering model based on the visual loss function to complete the visual alignment training of the visual question answering model based on the second image.

[0131] In one embodiment of the present application, the construction module 12 is further used to input the third image and the fourth image into the trained visual question answering model respectively; wherein the third image and the fourth image are respectively a high-information sample image in the sample set; through the attention mechanism of the trained visual question answering model, the visual attention distribution of the image regions in the third image and the fourth image is determined respectively; based on the visual attention distribution of the image regions in the third image and the fourth image, the third image and the fourth image are region-level fused to generate a high-information counterfactual image;

[0132] The training module 14 is further configured to perform adversarial training on the trained visual question answering model using the high-information counterfactual image.

[0133] In one embodiment of the present application, the construction module 12 is used to determine M image areas from the third image based on the visual attention distribution of image areas in the third image; wherein M is a positive integer, and the visual attention of the M image areas is less than the visual attention of other image areas in the third image; based on the visual attention distribution of image areas in the fourth image, determine N image areas from the fourth image; wherein N is a positive integer, and the visual attention of the N image areas is greater than the visual attention of other image areas in the fourth image; and fuse the M image areas with the N image areas to generate a high-information counterfactual image.

[0134] Based on the same inventive concept, Figure 8 A schematic diagram of a model processing device provided in an embodiment of the present application Figure 2 .like Figure 8 As shown, in the embodiment of the present application, the model processing device 1 includes: a processor 15, a memory 16 and a communication bus 17;

[0135] The communication bus 17 is used to realize the communication connection between the processor 15 and the memory 16;

[0136] The processor 15 is used to execute one or more computer programs stored in the memory 16 to implement the above-mentioned model processing method.

[0137] An embodiment of the present application provides a computer program product, including a computer program, which implements the above-mentioned model processing method when being executed.

[0138] The embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the above-mentioned model training method when the computer program is executed by a processor. The computer-readable storage medium may be a volatile memory (volatile memory), such as a random access memory (Random-Access Memory, RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (Read-Only Memory, ROM), a flash memory (flash memory), a hard disk (Hard Disk Drive, HDD) or a solid-state drive (Solid-State Drive, SSD); or may be a respective device including one or any combination of the above-mentioned memories, such as a mobile phone, a computer, a tablet device, a personal digital assistant, etc.

[0139] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0140] The present application is described with reference to implementation flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0141] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which is implemented in the implementation flow diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0142] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing the steps in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0143] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed in the utility application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model processing method, characterized in that: include: Select high-information sample images corresponding to each type of standard visual answer based on visual uncertainty; Based on the selected high-information sample images, a sample set whose standard visual answer distribution meets the expected answer distribution is constructed; For each high-information sample image in the sample set, performing region-level visual information annotation; Using each labeled high-information sample image, visual alignment training is performed on a visual question answering model; wherein the visual question answering model is used to understand the image and answer visual questions posed to the image.

2. The method according to claim 1, characterized in that: The method of selecting a high-information sample image corresponding to each type of standard visual answer based on visual uncertainty includes: Inputting a first image corresponding to a first standard visual answer into the visual question answering model; wherein the first standard visual answer is a class of standard visual answers, and the first image is a sample image corresponding to the first standard visual answer; Determining, by means of an attention mechanism of the visual question answering model, a visual attention distribution of an image region in the first image; determining an uncertainty entropy of the first image based on a visual attention distribution of an image region in the first image; If the uncertainty entropy of the first image is greater than or equal to a threshold, the first image is determined as a high-information sample image corresponding to a first standard visual answer.

3. The method according to claim 1, characterized in that The method of constructing a sample set whose standard visual answer distribution satisfies the expected answer distribution based on the selected high-information sample images includes: Construct a sampling set based on selected high-information samples; For the sampling set, the difference between the standard visual answer distribution of the set and the expected answer distribution is minimized to obtain the sample set.

4. The method according to claim 1, characterized in that: The step of labeling each high-information sample image in the sample set with region-level visual information includes: Obtaining the importance ranking of K image regions in the second image; wherein the importance of the image region is positively correlated with the correlation between the visual information of the image region and the standard visual answer, the second image is a high-information sample image in the sample set, the importance of the K image regions is greater than the importance of other image regions in the second image, and K is a positive integer; The importance ranking is marked for the K image regions to complete region-level visual information labeling of the second image.

5. The method according to claim 4, characterized in that The method of using each labeled high-information sample image to perform visual alignment training on the visual question answering model includes: Inputting the second image into the visual question answering model; Determining the visual attention distribution of the K image regions through the attention mechanism of the visual question answering model; According to the principle that the importance of the image region is positively correlated with the visual attention, the importance ranking and the visual attention distribution of the K image regions are compared to determine the visual loss function; The attention mechanism of the visual question answering model is adjusted based on the visual loss function to complete visual alignment training of the visual question answering model based on the second image.

6. The method according to claim 1, characterized in that After performing visual alignment training on the visual question answering model using each labeled high-information sample image, the method further includes: Inputting the third image and the fourth image into the trained visual question answering model respectively; wherein the third image and the fourth image are respectively a high-information sample image in the sample set; Determining visual attention distribution of image regions in the third image and the fourth image respectively through the attention mechanism of the trained visual question answering model; Based on the visual attention distribution of the image regions in the third image and the fourth image, performing region-level fusion on the third image and the fourth image to generate a high-information counterfactual image; The trained visual question answering model is adversarially trained using the high-information counterfactual image.

7. The method according to claim 6, characterized in that The step of performing region-level fusion on the third image and the fourth image based on the visual attention distribution of the image regions in the third image and the fourth image to generate a high-information counterfactual image includes: Based on the visual attention distribution of the image areas in the third image, determining M image areas from the third image; wherein M is a positive integer, and the visual attention of the M image areas is less than the visual attention of other image areas in the third image; Based on the visual attention distribution of the image areas in the fourth image, determining N image areas from the fourth image; wherein N is a positive integer, and the visual attention of the N image areas is greater than the visual attention of other image areas in the fourth image; The M image regions are fused with the N image regions to generate a high-information counterfactual image.

8. A model processing device, characterized in that: include: A selection module, used to select high-information sample images corresponding to each type of standard visual answer based on visual uncertainty; A construction module, for constructing a sample set whose standard visual answer distribution satisfies the expected answer distribution based on the selected high-information sample images; A labeling module, used for labeling each high-information sample image in the sample set with region-level visual information; A training module is used to perform visual alignment training on a visual question answering model using each labeled high-information sample image; wherein the visual question answering model is used to understand the image and answer visual questions posed to the image.

9. A model processing device, characterized in that: include: processor, memory, and communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute one or more computer programs stored in the memory to implement the model processing method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the model training method as described in any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed, the model processing method according to any one of claims 1 to 7 is implemented.

Images