A triple asymmetry and principle generated knowledge visual question and answer method and system

Abstract

The application discloses a kind of triple asymmetric and principle generated knowledge visual question and answer method and system, belong to knowledge visual question and answer field.Picture, question corresponding to picture and its option are acquired, question and each option are regarded as a text pair, record the reason behind answer;Utilize pre-training model to extract picture embedding feature vector and the first text embedding feature vector of text pair;Lightweight text encoder is used to obtain the second text embedding feature vector of text pair;Loss function item of text relative to picture, loss function item of picture relative to text, loss function item of text relative to text, principle generated cross-entropy loss function item are combined, lightweight text encoder and decoder are trained, and knowledge visual question and answer task is completed using the trained lightweight text encoder and pre-training model.The application realizes accurate and efficient knowledge visual question and answer by jointly using triple asymmetric feature alignment mechanism and reason generation mechanism.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of knowledge visual question answering, and in particular to a knowledge visual question answering method and system based on triple asymmetry and principle generation. BACKGROUND

[0002] With the rapid development of deep learning, people are increasingly interested in multi-modal recognition and reasoning. Knowledge visual question answering is a challenging task that not only requires a given image, but also requires external knowledge to answer questions.

[0003] Existing knowledge visual question answering methods can be roughly divided into two categories: one is a method that relies on an external knowledge base, and the other is a method that uses a large language model as an implicit knowledge engine.

[0004] However, the former method relies heavily on the quality of external knowledge base information retrieval, which can easily introduce additional information bias into the entire system. The latter method uses a large pure language model as a knowledge engine, which has extremely high computational cost and loss of image information. If these two problems are solved, the inference efficiency can be effectively improved while ensuring the accuracy of knowledge visual answers. SUMMARY

[0005] In order to overcome the defects of relying on an external knowledge base and a large pure language model in the prior art, the present application provides a knowledge visual question answering method and system based on triple asymmetry and principle generation, which uses a pre-trained model (such as CLIP) and a lightweight language model to solve the knowledge visual question answering task.

[0006] The specific technical solutions adopted by the present application are as follows:

[0007] In a first aspect, the present application provides a knowledge visual question answering method based on triple asymmetry and principle generation, comprising:

[0008] obtaining a picture, a question corresponding to the picture and options thereof, taking the question and each option as a text pair, and recording the reason behind the answer;

[0009] extracting a picture embedding feature vector and a first text embedding feature vector of the text pair using a pre-trained model, and obtaining a second text embedding feature vector of the text pair using a lightweight text encoder;

[0010] A bijective relationship between the second text embedding feature vector and the picture embedding feature vector is used to obtain a loss function item of the text relative to the picture; a one-to-many but not onto relationship between the picture embedding feature vector and the second text embedding feature vector is used to obtain a loss function item of the picture relative to the text; a bijective relationship between the second text embedding feature vector and the first text embedding feature vector is used to obtain a loss function item of the text relative to the text; a cross-entropy loss function item of principle generation is obtained by using the decoder autoregressive generation answer behind the reason text.

[0011] The loss function items are combined to train the lightweight text encoder and the decoder, and the trained lightweight text encoder and the pre-trained model are used to complete the knowledge visual question and answer task.

[0012] Further, the pre-trained model adopts a CLIP model, which includes a pre-trained visual encoder and a text encoder, respectively used to extract a picture embedding feature vector and a first text embedding feature vector of a text pair.

[0013] Further, the text pair is preprocessed before the text embedding feature vector is extracted, and the preprocessing includes:

[0014] For each given text pair s i , first, a special symbol <bos>, the 0th word of the text pair sentence is marked to indicate the beginning of the text pair sentence; then a special symbol is added after the text pair sentence <eos>, the last word of the text pair sentence, to mark the end of the text pair sentence; and then using special symbols <pad>The text pair sentence is padded to a preset maximum text pair sentence length to obtain a preprocessed text pair s' i .

[0015] Further, the lightweight text encoder comprises a Transformer encoder and a mapping layer, the Transformer encoder is used to obtain a text encoding feature sequence of the text pair, only the last encoding layer of the Transformer encoder is trained, and the parameters of the remaining encoding layers are frozen; the mapping layer is used to convert the text encoding feature sequence into a second text embedding feature vector consistent with the format of the first text embedding feature vector.

[0016] Further, the calculation formula of the mapping layer is:

[0017]

[0018]

[0019] wherein, denotes the second text embedding feature vector corresponding to the i-th text pair, denotes the average encoding feature corresponding to the i-th text pair, e j denotes the text encoding feature of the j-th word in the text pair, n denotes the number of words in the text pair, e0=e bos when j=n+1, e n+1 =W eos ; W e is a mapping matrix, and ||·||2 is an average operator with a norm of 2.

[0020] Further, the bijective relationship between the second text embedding feature vector and the picture embedding feature vector is used to obtain a text-to-picture loss function term, and the calculation formula is as follows:

[0021]

[0022] wherein, denotes the text-to-picture loss function term, S denotes the number of pictures in a training batch, denotes the second text embedding feature vector of the text pair composed of the i-th correct option in the training batch, denotes the k-th picture in the training batch, τ denotes a temperature hyperparameter of a contrastive learning, and exp is an exponential operator.

[0023] Further, the injective but non-surjective relationship between the picture embedding feature vector and the second text embedding feature vector is used to obtain a picture-to-text loss function term, and the calculation formula is as follows:

[0024]

[0025] wherein, denotes the picture-to-text loss function term, B denotes the number of pictures in the training batch, N denotes the number of options corresponding to the question of each picture; a k ∈A, a k denotes the answer index corresponding to the kth picture, A denotes the correct answer index set, denotes the kth picture embedding feature vector, denotes the second text embedding feature vector of the text pair corresponding to the correct option of the kth picture, denotes the second text embedding feature vector of the ith text pair in the training batch, τ is a temperature hyperparameter of contrastive learning, exp is an exponential operator,

[0026] Further, the bijection relationship between the second text embedding feature vector and the first text embedding feature vector is used to obtain the text-to-text loss function term, and the calculation formula is as follows:

[0027]

[0028] wherein, denotes the text-to-text loss function term, B denotes the number of pictures in the training batch, N denotes the number of options corresponding to the question of each picture, denotes the second text embedding feature vector of the ith text pair in the training batch, denotes the first text embedding feature vector of the ith and jth text pairs in the training batch, τ is a temperature hyperparameter of contrastive learning, exp is an exponential operator.

[0029] Further, the decoder autoregressive generation of the reason text behind the answer is used to obtain the cross-entropy loss function term generated by the principle, which comprises:

[0030] The cross-modal affinity matrix M between the picture embedding feature vector and the text encoding feature sequence of the text pair in which the correct option is located is calculated, and the calculation method of each row of the matrix is as follows:

[0031]

[0032] wherein, and b s are trainable weight vectors and offset vectors, tanh(·) is a tangent function, denotes the kth picture embedding feature vector, e j denotes the text encoding feature of the jth word in the text pair in which the correct option is located, W v and W e respectively represent the trainable visual weight matrix and the text weight matrix;

[0033] normalizing the cross-modal affinity matrix M to obtain a relative affinity strength matrix from the image modality to the text modality

[0034] according to the picture embedding feature vector and the text encoding feature sequence of the text pair of the correct option and the relative affinity strength matrix constructing a fused semantic feature C:

[0035]

[0036] wherein concat(·) is a vector concatenation operation;

[0037] the cross-entropy loss function item generated by the principle generation:

[0038]

[0039] wherein, the loss function item generated by the reason, P(·|·) represents the conditional probability, w<k represents the first k words in the reason, and K represents the length of the reason text behind the answer.

[0040] In a second aspect, the present application provides a triple asymmetric and principle generation knowledge visual question and answer system, comprising:

[0041] a data acquisition module for acquiring a picture, a question corresponding to the picture and options thereof, taking the question and each option as a text pair, and recording the reason behind the answer;

[0042] a feature extraction module for extracting a picture embedding feature vector and a first text embedding feature vector of the text pair by using a pre-trained model; and obtaining a second text embedding feature vector of the text pair by using a lightweight text encoder;

[0043] a loss term calculation module for obtaining a loss function item of the text relative to the picture by using the bijective relationship between the second text embedding feature vector and the picture embedding feature vector; obtaining a loss function item of the picture relative to the text by using the injective but non-surjective relationship between the picture embedding feature vector and the second text embedding feature vector; obtaining a loss function item of the text relative to the text by using the bijective relationship between the second text embedding feature vector and the first text embedding feature vector; and obtaining a cross-entropy loss function item of the principle generation by using the decoder autoregressive generation of the reason text behind the answer;

[0044] a training module for training the lightweight text encoder and the decoder in combination with the loss function items;

[0045] A knowledge visual question answering module is configured to complete a knowledge visual question answering task by using a trained lightweight text encoder and a pre-trained model.

[0046] Compared with the prior art, the present application has the beneficial effects that:

[0047] The present application uses a triple asymmetric feature alignment mechanism and a principle generation mechanism to implement the knowledge visual question answering task, wherein the triple asymmetric feature alignment mechanism aligns the lightweight text encoder with the visual encoder and the text encoder of the pre-trained model in space at a very low training cost (0.0152B trainable parameters), and enhances the understanding ability of the semantic granularity, so as to realize accurate and efficient knowledge visual question answering. The principle generation mechanism forces the model to output the reason behind the answer during the training process, thereby avoiding the shortest path dependence, enabling the model to truly understand the entire reasoning chain, and realizing accurate and efficient knowledge visual question answering.

[0048] In summary, by jointly using the triple asymmetric feature alignment mechanism and the reason generation mechanism, the present application can fully utilize the image and text modal alignment properties extracted by the pre-trained model and the lightweight of the trainable text encoder, replace the external knowledge base and the pre-trained large-scale language model, and has obvious computational efficiency and answer effectiveness. The present application is evaluated on the most challenging and largest knowledge-based visual question answering dataset A-OKVQA. The results show that the present application is significantly better than the existing knowledge visual question answering method, and only uses about 1 / 400 parameters to achieve a performance comparable to the method of the large-scale language model (such as GPT-3). BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 is the overall framework diagram of the knowledge visual question answering method using triple asymmetric feature alignment and principle generation according to the embodiment of the present application;

[0050] Figure 2 is a schematic diagram of triple asymmetric alignment;

[0051] Figure 3 is a schematic diagram of picture feature and text feature sequence fusion;

[0052] 2) Align the lightweight text encoder with the image space of the pre-trained visual text model at a very low training cost (0.0152B trainable parameters), and enhance the understanding ability of the semantic granularity;

[0053] 3) Introduce a text generator, combine picture information and question and answer text information, force the model to output the reason behind the answer during the training process, thereby avoiding the shortest path dependence, enabling the model to truly understand the entire reasoning chain. DETAILED DESCRIPTION

[0054] The present application is further described and explained with reference to the accompanying drawings and detailed description.

[0055] A method for generating solutions to knowledge visual question answering by using triple asymmetric feature alignment and principle is disclosed, and the overall flow is as shown in Figure 1 The method mainly includes the following steps:

[0056] Step 1, constructing a text pair of a question and an answer for a picture;

[0057] Step 2, obtaining a picture embedding feature vector by using a pre-trained visual encoder;

[0058] Step 3, obtaining a first text embedding feature vector of the text pair by using a pre-trained text encoder;

[0059] Step 4, encoding the text pair by using a lightweight text encoder to obtain a text encoding feature sequence;

[0060] Step 5, converting the text encoding feature sequence obtained in step 4 into a structure with the same format as the first text embedding feature vector obtained in step 3 to obtain a converted second text embedding feature vector;

[0061] Step 6, obtaining a loss function item of the text relative to the picture by using the bijective relationship between the second text embedding feature vector and the picture embedding feature vector;

[0062] Step 7, obtaining a loss function item of the picture relative to the text by using the injective but not surjective relationship between the picture embedding feature vector and the second text embedding feature vector;

[0063] Step 8, obtaining a loss function item of the text relative to the text by using the bijective relationship between the second text embedding feature vector and the first text embedding feature vector;

[0064] Step 9, obtaining a cross-entropy loss function item of principle generation by using a decoder autoregressive to generate a reason text behind an answer;

[0065] Step 10, combining the loss function items obtained in steps 6 to 9 to obtain a complete loss function, and completing the knowledge visual question answering task by training.

[0066] In step 1, the text pair of the question and the answer can be constructed by using the following implementation mode.

[0067] In the training set, the validation set and the test set, each picture will have a question q based on knowledge and N options C=(c1, c2, …, c N To facilitate the model's understanding of the internal logic of the question options, this invention constructs text pairs by concatenating text pairs at the word level rather than the feature level. This involves adding the prompt word "Question" before the question and the prompt word "Answer" before the options.

[0068] s i =[Question: q Answer: c] i ]

[0069] Where q represents the question content of the image, which is a sentence composed of multiple words and punctuation marks; c i c represents the i-th option corresponding to the question. i It could be a single word or a phrase consisting of multiple words; s i It is the i-th text pair corresponding to the image.

[0070] After constructing the text as described above, each image has N corresponding text pairs, one of which is the text pair corresponding to the correct answer, and its index is recorded as 'a'. Each training batch has B images, which yields B*N text pairs, B answer indices, and B reasons corresponding to each answer.

[0071] The training set is organized as follows, with D train Zhang picture and D train *N corresponding text pairs, where each image corresponds to N text pairs. For ease of explanation, the image sets will be labeled V = (v1, v2, ..., v3) in the following content, using training batches as the unit. B The set of text pairs is S = (s1, s2, ..., s...). B*N The set of answer indices is A = (a1, a2, ..., a...). B The set of reasons is R = (r1, r2, ..., r...). B ), where the index of the answer corresponding to the k-th image is a. k The reason is r k .

[0072] In steps 2 and 3, the image embedding feature vector is obtained using a pre-trained visual encoder, and the text embedding feature vector is obtained using a pre-trained text encoder, respectively. The specific implementation methods are as follows.

[0073] During network training, each image and N text pairs are simultaneously input into the pre-trained visual encoder and pre-trained text encoder. In this embodiment, the pre-trained visual encoder and pre-trained text encoder use the visual encoder and text encoder from the CLIP model.

[0074] For each image, obtain the image embedding feature vector encoded by the pre-trained visual encoder in the CLIP model. The calculation formula is as follows:

[0075]

[0076] where v k represents the kth picture, CLIP I (·) represents a pre-trained visual encoder in the CLIP model, and the picture embedding feature vectors of all pictures in the training batch constitute a set

[0077] For each given text pair s i Preprocessing, first add special symbols before the text pair sentence <bos>, the 0th word of the text pair sentence is marked to indicate the beginning of the text pair sentence; then a special symbol is added after the text pair sentence <eos>, the last word of the text pair sentence, to mark the end of the text pair sentence; and then using special symbols <pad>filling the text pair sentence to a preset maximum text pair sentence length to obtain s' i ; and finally inputting the pre-trained text encoder in the CLIP model to obtain the representation of the text:

[0078]

[0079] wherein s' i represents the i-th text pair sentence after the above preprocessing, CLIP T (·) represents the pre-trained text encoder in the CLIP model.

[0080] The pre-trained visual encoder and the text encoder of the CLIP model mentioned above are initialized using pre-trained parameters, and the parameters are frozen and do not participate in training, so as to reduce the training cost and maintain the original image-text consistency of the CLIP.

[0081] In step 4, the text is encoded using a lightweight text encoder to obtain a text embedding feature vector sequence, which can be specifically implemented as follows.

[0082] Similarly, for each given text pair s i is preprocessed, and the preprocessing method is the same as step 3. For the convenience of description, special symbols are ignored in the following representation <pad>, obtaining pre-processed text pair of sentences s' i = [ <bos> ，...， <eos>], and then a sequence of text encoding features E is extracted using a lightweight text encoder that has been pre-trained on a large amount of text i :

[0083] E i = LM enc (s′ i ) = [e bos , e1,..., en, e n ] eos

[0084] where s′ i represents the i-th text pair sentence after the above preprocessing, LM enc (·) represents a lightweight text encoder trained on pure text data, n represents the length of the text pair sentence, e n is the text encoding feature corresponding to the n-th word in the text pair sentence, e bos , e eos are special symbols <bos>and <eos>Corresponding text encoding features.

[0085] In a specific implementation of the present application, the lightweight text encoder adopts a Transformer encoder structure, which can be initialized using pre-trained models of different sizes, including bart-base and bart-large. During training, to reduce training costs and prevent overfitting, only the last layer of the Transformer encoder is trained, and the parameters of the remaining layers are fixed.

[0086] In step 5, the text encoding feature sequence is converted into the same structure as the first text embedding feature vector obtained in step 3, obtaining a converted second text embedding feature vector, specifically:

[0087] The first text embedding feature vector encoded by the pre-trained text encoder of the CLIP model is an aggregated semantic feature vector d1 is the dimension of the first text embedding feature vector of the pre-trained text encoder of the CLIP model. Similarly, the image embedding feature vector dimension of the pre-trained visual encoder of the CLIP model is also d1. The text encoding feature sequence obtained by the lightweight text encoder is d2 is the text feature dimension of the lightweight text encoder. To ensure that the output feature dimensions remain consistent, the average value of all output features of the trainable lightweight text encoder is calculated through the mapping layer as the text pair sentence representation, and it is normalized:

[0088]

[0089]

[0090] wherein, represents the second text embedding feature vector corresponding to the i-th text pair, e j represents the text encoding feature of the j-th word in the text pair, e0=e bos when j=n+1, e n+1 ; eos is a mapping matrix, representing a trainable weight matrix for mapping text encoding features into the latent space of the CLIP model, and ||·||2 is the average operator with a norm of 2. The above steps are to first convert the encoding feature sequence into a single feature, then normalize and convert it to the same feature dimension as the first text embedding feature vector.

[0091] ​In step 6, the bijection between the second text embedding feature vectors obtained by the lightweight text encoder and the image embedding feature vectors obtained by the pre-trained visual encoder of the CLIP model is used to obtain the loss function item of the text relative to the image, the purpose of which is to minimize the distance between each second text embedding feature and its corresponding target image feature, while maximizing the distance from other image features.

[0092] The process is shown in Figure 2 (a), which can be achieved by aligning the trainable second text embedding feature vectors with the fixed image embedding feature vectors. Through analysis, it is found that not every text has its corresponding image, so the second text embedding feature vectors are divided into two subsets: the positive sample set and the negative sample set represents the set of all text pairs corresponding to the second text embedding feature vectors in the training batch, represents the Bth positive sample, represents the (N-1)Bth negative sample. Then, and the set of image embedding feature vectors of all images in the training batch there is a bijection between them, and the contrast learning InfoNCE loss between and is optimized to obtain the loss function item of the text relative to the image:

[0093]

[0094] where, represents the loss function item of the text relative to the image, represents the dot product of the ith text embedding feature vector and the kth image embedding feature vector, τ is a temperature hyperparameter for contrast learning, and exp is the exponential operator.

[0095] In step 7, the injective but not surjective relationship between the image embedding feature vectors obtained by the pre-trained visual encoder in the CLIP model and the second text embedding feature vectors obtained by the lightweight text encoder is used to obtain the loss function item of the image relative to the text, the purpose of which is to distinguish the overall meaning of the text pair sentence and achieve fine-grained distinction at the word level.

[0096] The process is shown in Figure 2 (b), which can be achieved by one-to-one correspondence of the subscript set A to obtain the correct text pair corresponding to the image, and use the set of second text embedding feature vectors of the entire sentence to perform injective but not surjective alignment from to to obtain the loss function item of the image relative to the text:

[0097]

[0098] wherein a k ∈A, a k denotes the answer index corresponding to the k-th picture, denotes the loss function item of the picture relative to the text denotes the dot product of the k-th picture embedding feature vector and the second text embedding feature vector corresponding to the text pair of the correct answer of the k-th picture, τ is a temperature hyperparameter of contrastive learning, and exp is an exponential operator.

[0099] In step 8, the bijective relationship between the second text embedding feature vector obtained by the lightweight text encoder and the first text embedding feature vector obtained by the pre-trained text encoder of the CLIP model is used to obtain the loss function item of the text relative to the text, and the purpose is to constrain the features output by the trainable lightweight text encoder into the same feature space of the CLIP model.

[0100] The process is as shown in Figure 2 (c), and in the case of inputting the same text, the alignment relationship between the second text embedding feature vector obtained by the trainable lightweight text encoder and the first text embedding feature vector output by the CLIP model is used to obtain the loss function item of the text relative to the text:

[0101]

[0102] wherein, denotes the loss function item of the text relative to the text, denotes the dot product of the i-th second text embedding feature vector and the j-th first text embedding feature vector, τ is a temperature hyperparameter of contrastive learning, and exp is an exponential operator. In step 9, the cross-entropy loss function item generated by the principle is obtained by using the decoder autoregressive to generate the text behind the reason for the answer, which is specifically:

[0103] 9.1) Taking one picture in the training batch as an example, let the image representation be The text encoding feature sequence of the text pair sentence output by the trainable lightweight text encoder is The cross-modal affinity matrix M between and is calculated through the attention layer, and the calculation method of each row of the matrix is as follows:

[0104]

[0105] wherein, and b s are trainable weight vectors and offset vectors, tanh(·) is a tangent function, W v and W e Let e ​​represent the trainable visual weight matrix and text weight matrix, respectively. j express The text encoding feature of the j-th element, i.e., the j-th word, when j=0, e0=e bos When j = n+1, e n+1 =e eos ; Traversal For each element in the matrix, we obtain the cross-modal affinity matrix M;

[0106] 9.2) Obtain the relative affinity matrix from image mode to text mode by normalizing the cross-modal affinity matrix. The calculation method for each row of the matrix is ​​as follows:

[0107]

[0108] Where, ||·||2 is the average value operator with a normal form of 2;

[0109] 9.3) To preserve the original semantics of the image and text, as well as their fusion result, a fused semantic feature C is constructed:

[0110]

[0111] Here, concat(·) is a vector concatenation operation.

[0112] Steps 9.1) to 9.3) above are as follows: Figure 3 As shown.

[0113] 9.4) Decode the reasoning behind the answer to avoid shortest path dependence. Let the word-level representation of the reason of length K be r1 = (w1, w2, ..., w K The loss function term generated is obtained from this.

[0114]

[0115] in, Let P(·|·) represent the loss function term generated by the reason, and let w represent the conditional probability. <k The first k words in the reason.

[0116] In this step, the decoder has a multi-layer structure and can be a Transformer decoder. In this invention, only the last layer of the Transformer decoder is trained, and the parameters of the remaining layers are fixed.

[0117] In step 10, the loss function terms obtained in steps 6, 7, 8, and 9 are... and By minimizing the joint loss term The encoder-decoder network composed of the lightweight text encoder of training step 4 and the decoder of step 9 (as shown in Figure 1 The joint loss term is the weighted sum of the four loss function terms:

[0118]

[0119] where λ E2V , λ V2E , λ E2S and λ RG are weight parameters.

[0120] The process of knowledge visual question answering using the trained model is as follows: obtaining an image, a question and options, forming a text pair with the question and each option, obtaining a picture embedding feature vector using a pre-trained visual encoder, encoding the text pair using a lightweight text encoder, obtaining a text encoding feature sequence and performing format conversion to obtain a second text embedding feature vector; calculating the similarity between each second text embedding feature vector and the picture embedding feature vector, and outputting the option corresponding to the second text embedding feature vector with the highest similarity as the answer.

[0121] In this embodiment, the loss function is used to train the parameters of the model using the gradient descent learning method, complete the triple asymmetric feature alignment and principle generation, and thus complete the knowledge visual question answering task.

[0122] The above method will be applied to the following embodiments to embody the technical effects of the present application, and the specific steps in the embodiments will not be described again.

[0123] The present application is tested on the largest knowledge visual question answering dataset A-OKVQA. The present application uses a four-to-one correct rate evaluation standard to evaluate the effect of the present application, including the correct rate of the validation set (val-acc) and the correct rate of the test set (test-acc), and is compared with the following prior art models:

[0124] Comparison 1. Use external knowledge base method, such as KRISP and GPV-2. The model performance depends largely on the quality of the external knowledge base and the ability to retrieve knowledge, which greatly limits the practical application of this method.

[0125] Comparison 2. Methods such as ClipCap, Prophet and Prophe-MC use the CLIP model as an image feature extractor, then construct a prompt text for a large-scale language model, and thus complete the knowledge visual question answering. The larger the pre-trained language model, the better the performance, but this leads to resource waste and reduced inference efficiency.

[0126] Comparative 3. The multi-modal pre-training models such as ViLBERT, LXMERT and CLIP were fine-tuned on the A-OKVQA dataset to obtain their performance on the knowledge visual question answering task.

[0127] According to the steps described in the specific embodiment, the obtained experimental results are shown in Table 1, and the model of the application is represented as TAG.

[0128] Table 1: Test results of the application for the A-OKVQA dataset

[0129]

[0130]

[0131] From Table 1, it can be found that the lightweight language model based on few trainable parameters (0.387B) achieves the second highest accuracy among all methods, with an accuracy of 67.9% on the A-OKVQA validation set and a correct rate of 61.2% on the test set. Compared with Prophet and Prophet-MC, the parameter amount used by TAG is only one four-hundredth (0.387B vs 170B), but the performance on the A-OKVQA dataset is comparable. These results demonstrate the effectiveness of the TAG method and its potential to improve the performance of knowledge-based VQA systems.

[0132] In this embodiment, a knowledge visual question answering system generated by using triple asymmetric feature alignment and principle is also provided, which is used to implement the above-mentioned embodiments. The terms "module", "unit" and the like used below can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware implementation is also possible.

[0133] The knowledge visual question answering system provided by the application comprises:

[0134] The data acquisition module is used to acquire pictures, questions corresponding to the pictures and options thereof, take the question and each option as a text pair, and record the reason behind the answer;

[0135] The feature extraction module is used to extract a picture embedding feature vector and a first text embedding feature vector of the text pair by using a pre-training model; and obtain a second text embedding feature vector of the text pair by using a lightweight text encoder;

[0136] a loss term calculation module configured to obtain a loss function term of the text relative to the picture by using a bijective relationship between the second text embedding feature vector and the picture embedding feature vector, obtain a loss function term of the picture relative to the text by using a one-way but not full-coverage relationship between the picture embedding feature vector and the second text embedding feature vector, obtain a loss function term of the text relative to the text by using a bijective relationship between the second text embedding feature vector and the first text embedding feature vector, and obtain a cross-entropy loss function term of principle generation by using the decoder autoregressive generation answer behind the reason text;

[0137] a training module configured to train the lightweight text encoder and the decoder in combination with the loss function terms;

[0138] a knowledge visual question and answer module configured to complete a knowledge visual question and answer task by using the trained lightweight text encoder and the pre-training model.

[0139] In a specific implementation of the present application, the feature extraction module comprises:

[0140] a pre-training visual encoder module configured to obtain a picture embedding feature vector of the picture;

[0141] a pre-training text encoder module configured to obtain a first text embedding feature vector of a text pair corresponding to the picture;

[0142] a lightweight text encoder module configured to encode the text pair and convert the text encoding feature sequence obtained by encoding into a second text embedding feature vector consistent in dimension with the first text embedding feature vector.

[0143] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts are described in the method embodiment, and the implementation method of the remaining modules will not be described here. The system embodiment described above is only illustrative, wherein the units described as separate components can be or can not be physically separated, and the components displayed as units can be or can not be physical units, that is, they can be located in one place or distributed on multiple network units. According to actual needs, part or all of the modules can be selected to achieve the purpose of the present application scheme. Those skilled in the art can understand and implement without creative labor.

[0144] Embodiments of the system of the present application can be applied on any data processing capable device, which can be a device or apparatus such as a computer. The system embodiments can be implemented by software, or by hardware, or by a combination of software and hardware. Taking the software implementation as an example, as a logical sense apparatus, it is formed by the processor of the data processing capable device in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory for running.

[0145] The above only lists specific embodiments of the present application. Obviously, the present application is not limited to the above embodiments, and there can be many variations. All variations that can be directly derived or inferred from the disclosure of the present application by those of ordinary skill in the art should be considered as falling within the scope of the present application.< / eos> < / bos> ​< / eos> < / bos> < / pad> < / pad> < / eos> < / bos> < / pad> < / eos> < / bos>

Claims

1. A knowledge-based visual question-answering method based on triple asymmetry and its underlying principles, characterized in that... include: Retrieve the image, the corresponding question and its options, treat the question and each option as a text pair, and record the reasoning behind the answer; The image embedding feature vector and the first text embedding feature vector of the text pair are extracted using a pre-trained model; the second text embedding feature vector of the text pair is obtained using a lightweight text encoder. The loss function term for text relative to image is obtained by utilizing the bijective relationship between the second text embedding feature vector and the image embedding feature vector; the loss function term for image relative to text is obtained by utilizing the injective but not surjective relationship between the image embedding feature vector and the second text embedding feature vector. The calculation formula is as follows: ； in, This represents the loss function term relative to the text. This indicates the number of images in the training batch. This indicates the number of options corresponding to the question for each image; , This represents the index of the answer corresponding to the k-th image. This represents the set of indices of the correct answers. This represents the embedding feature vector of the k-th image. The second text embedding feature vector represents the text pair corresponding to the correct option of the k-th image. Let represent the second text embedding feature vector of the i-th text pair in the training batch. It is a temperature hyperparameter for comparative learning, and exp is the exponential operator; By utilizing the bijective relationship between the second text embedding feature vector and the first text embedding feature vector, the loss function term of text relative to text is obtained, and the calculation formula is as follows: ； in, This represents the loss function term relative to the text. This represents the first text embedding feature vector of the i-th and j-th text pairs in the training batch; The decoder is used to generate the reasoning text behind the answer through autoregression, and the cross-entropy loss function term generated by the principle is obtained. By combining the loss function term, a lightweight text encoder and decoder are trained, and the trained lightweight text encoder and pre-trained model are used to complete the knowledge visual question answering task.

2. The knowledge visual question-answering method based on the principle of triple asymmetry as described in claim 1, characterized in that, The pre-trained model uses the CLIP model, which includes a pre-trained visual encoder and a text encoder, used to extract image embedding feature vectors and the first text embedding feature vector of text pairs, respectively.

3. The knowledge visual question-answering method based on triple asymmetry and principle generation according to claim 1, characterized in that, The text pairs are preprocessed before text embedding feature vector extraction. The preprocessing includes: For each given text pair First, a special symbol `<bos>` is added before each text pair sentence to mark the beginning of the sentence as the 0th word. Then, a special symbol `<eos>` is added after each sentence to mark the end. Finally, the special symbol `<pad>` is used to pad the text pair sentences to the pre-defined maximum sentence length, resulting in the pre-processed text pairs. .

4. The knowledge visual question-answering method based on triple asymmetry and principle generation according to claim 1, characterized in that, The lightweight text encoder includes a Transformer encoder and a mapping layer. The Transformer encoder is used to obtain the text encoding feature sequence of text pairs. Only the last encoding layer of the Transformer encoder is trained, and the parameters of the remaining encoding layers are frozen. The mapping layer is used to convert the text encoding feature sequence into a second text embedding feature vector that is consistent with the format of the first text embedding feature vector.

5. The knowledge visual question-answering method based on triple asymmetry and principle generation according to claim 4, characterized in that, The calculation formula for the mapping layer is as follows: ； ； in, This represents the second text embedding feature vector corresponding to the i-th text pair. This represents the average encoding feature corresponding to the i-th text pair. This represents the text encoding feature of the j-th word in the text pair. This indicates the number of words in a text pair; when j=0, When j=n+1, ; For the mapping matrix, It is the average value operator with a normal form of 2.

6. The knowledge visual question-answering method based on the principle of triple asymmetry as described in claim 1, characterized in that, The loss function term for text relative to image is obtained by utilizing the bijective relationship between the second text embedding feature vector and the image embedding feature vector. The calculation formula is as follows: ； in, This represents the loss function term relative to the image. This indicates the number of images in the training batch. The second text embedding feature vector represents the text pair consisting of the i-th correct option in the training batch. This represents the k-th image in the training batch. This represents a temperature hyperparameter for comparative learning, where exp is the exponential operator.

7. The knowledge visual question answering method based on triple asymmetry and principle generation according to claim 4, characterized in that, The method of using a decoder to autoregressively generate the reasoning text behind the answer and obtaining the cross-entropy loss function term generated based on the principle includes: Calculate the cross-modal affinity matrix M between the image embedding feature vector and the text encoded feature sequence of the text pair containing the correct option. The calculation method for each row of the matrix is ​​as follows: ； in, and These are trainable weight vectors and offset vectors. , This represents the embedding feature vector of the k-th image. This represents the text encoding feature of the j-th word in the text pair containing the correct option. and These represent the trainable visual weight matrix and the text weight matrix, respectively. Normalizing the cross-modal affinity matrix M yields the relative affinity strength matrix from image mode to text mode. ; Based on image embedding feature vectors The text encoding feature sequence of the text pair containing the correct option. and the relative affinity matrix Construct fused semantic features C: ； in, It is a vector concatenation operation; The cross-entropy loss function term generated by the computational principle: ； in, This represents the loss function term generated based on the reason. Represents conditional probability. This refers to the first k words in the reason. The length of the text indicating the reasoning behind the answer.

8. A knowledge visual question-answering system based on triple asymmetry and its underlying principles, used to implement the knowledge visual question-answering method based on triple asymmetry and its underlying principles as described in claim 1, characterized in that... include: The data acquisition module is used to acquire images, the corresponding questions and their options, treat each question and option as a text pair, and record the reasons behind the answers. The feature extraction module is used to extract the image embedding feature vector and the first text embedding feature vector of the text pair using a pre-trained model; and to obtain the second text embedding feature vector of the text pair using a lightweight text encoder. The loss term calculation module is used to obtain the text-to-image loss function term by utilizing the bijective relationship between the second text embedding feature vector and the image embedding feature vector; to obtain the image-to-text loss function term by utilizing the injective but not surjective relationship between the image embedding feature vector and the second text embedding feature vector; to obtain the text-to-text loss function term by utilizing the bijective relationship between the second text embedding feature vector and the first text embedding feature vector; and to obtain the principle-generated cross-entropy loss function term by utilizing the decoder's autoregressive generation of the reasoning text behind the answer. The training module is used to train a lightweight text encoder and decoder by incorporating a loss function term; The knowledge visual question answering module is used to complete knowledge visual question answering tasks using a trained lightweight text encoder and a pre-trained model.

Images