Visual Question Answering Method with Intermediate-Process Supervision Strategy Based on Object Detection

Abstract

The present invention discloses a visual question-answering method based on an intermediate process supervision strategy of target detection, comprising the following steps: 1) extracting image features from visual images through a convolutional neural network; 2) parsing questions with answers into program trees using an encoder-decoder network model; 3) obtaining intermediate supervision signals corresponding to each execution step through a program execution network; image features and a program tree of L layers, the program execution is step-by-step reasoning from the 1st layer to the Lth layer, and the answer is obtained based on the image features. The present invention parses the execution program from the question, and then performs step-by-step reasoning on the scene graph to generate a complete intermediate result. The present invention uses the intermediate results to perform intermediate process supervision on the visual question-answering model. The present invention provides a more comprehensive supervision signal for the reasoning process, thereby improving the answer prediction performance.

Description

Technical Field

[0001] The present invention relates to artificial intelligence technology, and in particular to a visual question answering method based on an intermediate process supervision strategy of target detection. Background Art

[0002] Neural modular networks (NMNs) have achieved sustained success in answering combined visual questions, especially those involving multi-hop visual and logical reasoning. NMNs use instance modules on the reasoning path of the problem to decompose complex problems into multiple corresponding subtasks, and then use intermediate process supervision to guide answer prediction, thereby improving the interpretability of reasoning. However, due to the vague modeling of intermediate process supervision, the performance of previous solutions in answering combined questions involving complex visual and logical reasoning is mediocre. The reasons are as follows: (1) These solutions usually assume that each instance module involves only one associated target, while ignoring other potential associated targets, thus hindering the complete cross-modal alignment learning; (2) Intersection over Union (IoU)-based intermediate supervision methods may introduce noise signals because the problem of bounding box overlap may guide the model's focus to actually irrelevant targets. Summary of the invention

[0003] The technical problem to be solved by the present invention is to provide a visual question answering method based on an intermediate process supervision strategy of target detection in view of the defects in the prior art.

[0004] The technical solution adopted by the present invention to solve the technical problem is: a visual question answering method based on an intermediate process supervision strategy for target detection, comprising the following steps:

[0005] 1) Extract image features through convolutional neural network (CNN);

[0006] Given an image I, use the pre-trained CNN backbone of Faster R-CNN to extract feature maps Where H and W represent the height and width of the feature map respectively, v In order to give image features visual context and cross-modal text information, this paper adopts the MCAN method, which uses Transformer blocks to encode questions and images. Specifically, given a question Q of length T, it is embedded into the latent space Then a two-layer Transformer is used to encode the problem as follows:

[0007]

[0008] Among them, SA, LN, and FFN represent self-attention, layer normalization, and feed-forward network, respectively. Then, a two-layer Transformer is used to use the visual context and question semantics. To enrich the image features V, as follows:

[0009]

[0010] Among them, PosEmb represents position embedding. FC represents the fully connected layer, which transforms the feature dimension from d v Convert to d h GA stands for guided attention, which uses the semantics of the question to enhance relevant visual features. The resulting image representation Can be used for program execution

[0011] 2) Use an encoder-decoder network (T5) model to parse the question into a program tree;

[0012] 2.1) Follow the MMN method to generate the corresponding executable program

[0013] This embodiment uses the MMN method to generate the corresponding executable program from the problem. Specifically, the nodes of the program tree are formalized as "Function (Arg1, ... ArgN)", where "Function" can be divided into 10 different execution types (such as select, relate, exist, or, etc.), each of which can be further subdivided into more subtypes (such as relate: relate_attr, relate_name, relate_inv_name, etc.), which take a variable number of parameters as input.

[0014] 2.2) Based on the above program types, the complete program tree can be viewed as a sequence of function nodes and generated by an encoder-decoder network. A prompt is added in front of the question (i.e., "Convert the question into a program:") and fed to the T5 model for program generation. The output sequence consists of interdependent functions, where "[N]" indicates the dependency. Then, a structured program tree with L layers is built from the sequence according to these dependencies, which can be used to guide step-by-step program execution.

[0015] 3) Obtain the intermediate supervision signal corresponding to each execution step through the program execution network;

[0016] Image features and L-layer program tree, the program executes step-by-step reasoning from layer 1 to layer L to obtain answers based on image features;

[0017] The details are as follows: Assume that the program tree contains N nodes, each node corresponds to a program text (such as "Select (giraffe)", "Select (elephant)", etc.), and use the program semantics to initialize the state matrix As shown below:

[0018]

[0019] Among them, prog i represents the program text of the ith node and is truncated or padded to a fixed length k; GloVe represents the GloVe embedding layer. In the process of step-by-step reasoning, the matrix S implies the intermediate reasoning states and results and can be used to decode the output of the DIS algorithm proposed in this invention. The input state of the lth layer is marked as S l-1 , the present invention uses the Transformer framework to obtain the final output state S l .

[0020] Specifically,

[0021] First, a masked self-attention layer is used to collect dependencies from the state of the previous layer, and then a guided attention layer is used to collect dependencies from image features. Find the visual clues in the code, which are as follows:

[0022]

[0023] Among them, MaskSA and GA represent the mask self-attention layer and the guided attention layer respectively; MaskSA uses the weight matrix M l Mask the non-dependent nodes, expressed as follows:

[0024]

[0025] in, Represents the mask matrix; if and only if node i is the parent node of node j, M ij =0, otherwise M ij = -∞; Q S ,K S ,V S It is the result of mapping S through a three-layer fully connected layer; its representation is as follows:

[0026]

[0027] Different from SA which collects information from itself, it guides attention to obtain features from other sources (e.g., visual cues), as follows:

[0028]

[0029] in, and Depend on After L program reasoning, the final state is used to predict the answer through a multi-layer perceptron (MLP) layer:

[0030]

[0031] in Represents the answer set The predicted scores of the answers in , and the answer with the highest score is selected as the final answer. Θ represents the model parameters. Finally, the cross entropy loss function is used to optimize the model, which is expressed as follows:

[0032]

[0033] in, and a gt denote the visual question answering dataset and the true answer, respectively.

[0034] Based on the execution of the above program, the present invention proposes an intermediate supervision method to limit the reasoning process and improve the answer prediction performance. The existing method is to calculate the probability distribution based on the IoU between the target bounding box and the true bounding box, which easily causes the model to focus on targets that are not relevant to the task. To this end, the present invention proposes a detection-based intermediate process supervision method, which formulates the intermediate supervision signal into a unified sequence form, thereby giving the model the ability to use different supervision types (such as bounding boxes, logical words (true / false), and text).

[0035] The detection-based intermediate process supervision method consists of two steps:

[0036] (1) Design manual rules for program execution and perform operations on the program tree on the ground truth map;

[0037] Execute the program on the given real annotated scene graph. The GQA dataset only provides real scene graph annotation information for the training set and the validation set. Therefore, DIS is only used to optimize the model to produce intermediate results (such as targets, true / false, answers) during the training phase.

[0038] The present invention designs symbolic reasoning rules for each function type (such as Select, Relate, Verify, etc.), for example, Select(x) is defined as selecting the node corresponding to x, and Relate(x, to the left of) is defined as finding the node to the left of x. Through the program Program and the scene graph Scene Graph, intermediate supervision is gradually inferred.

[0039] Generally, supervision can be divided into three types: Objects with bounding boxes, True / False, and Answers. For Objects, the present invention uses the bin technology in the Pix2Seq network to convert continuous coordinate values ​​into discrete labels. If there are multiple associated targets, their corresponding bounding boxes will be randomly shuffled and connected to form a result sequence. For True / False and Answer, the present invention directly uses text tags to form a sequence. In addition, some special tags are added to the sequence for further generation, such as [BEG], [SEP], , etc.

[0040] (2) An intermediate result decoder is used to decode the intermediate output, and then an autoregressive loss is used to optimize the loss of the intermediate process supervision based on detection;

[0041] The present invention proposes a two-layer Transformer for transforming the state Decode intermediate output results;

[0042] First, the [BEG] tag and state Initialization prompts the decoder to generate an output sequence; in the training stage, the present invention uses teacher-forcing and autoregressive loss to optimize the model, the formula is as follows:

[0043]

[0044]

[0045] Where o represents the length of the resulting sequence. Represents the intermediate state result obtained from the i-th program node.

[0046] During the training phase of the model, the prediction loss of the answer and the loss of the intermediate supervision based on the detection are combined to optimize the model parameters by minimizing the following loss function:

[0047]

[0048] Among them, α represents the weight of the intermediate process supervision loss module in the entire optimization process. In the test phase, the model only needs to retain the last predicted answer module.

[0049] The beneficial effects produced by the present invention are:

[0050] 1. The present invention parses the execution program from the question, and then performs step-by-step reasoning on the scene graph to generate a complete intermediate result. The present invention uses the intermediate result to perform intermediate process supervision on the visual question answering model. Compared with previous methods, the present invention provides a more comprehensive supervision signal for the reasoning process, thereby improving the answer prediction performance.

[0051] 2. By taking into account the intermediate results, the logical consistency of the answers output by the model is significantly improved when answering questions with complex visual and logical combinations and their corresponding sub-questions. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The present invention will be further described below with reference to the accompanying drawings and embodiments, in which:

[0053] Figure 1 It is a method flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0055] like Figure 1 As shown, a visual question answering method based on an intermediate process supervision strategy for target detection includes the following steps:

[0056] A visual question answering method based on an intermediate process supervision strategy for object detection includes the following steps:

[0057] 1) Extract image features through convolutional neural network (CNN);

[0058] Given an image I, use the pre-trained CNN backbone of Faster R-CNN to extract feature maps Where H and W represent the height and width of the feature map respectively, v In order to give image features visual context and cross-modal text information, this paper adopts the MCAN method, which uses Transformer blocks to encode questions and images. Specifically, given a question Q of length T, it is embedded into the latent space Then a two-layer Transformer is used to encode the problem as follows:

[0059]

[0060] Among them, SA, LN, and FFN represent self-attention, layer normalization, and feed-forward network, respectively. Then, a two-layer Transformer is used to use the visual context and question semantics. To enrich the image features V, as follows:

[0061]

[0062] Among them, PosEmb represents position embedding. FC represents the fully connected layer, which transforms the feature dimension from d v Convert to d h GA stands for guided attention, which uses the semantics of the question to enhance relevant visual features. The resulting image representation Can be used for program execution

[0063] 2) Use an encoder-decoder network (T5) model to parse the question into a program tree;

[0064] 2.1) Follow the MMN method to generate the corresponding executable program

[0065] This embodiment uses the MMN method to generate the corresponding executable program from the problem. Specifically, the nodes of the program tree are formalized as "Function (Arg1, ... ArgN)", where "Function" can be divided into 10 different execution types (such as select, relate, exist, or, etc.), each of which can be further subdivided into more subtypes (such as relate: relate_attr, relate_name, relate_inv_name, etc.), which take a variable number of parameters as input.

[0066] 2.2) Based on the above program types, the complete program tree can be viewed as a sequence of function nodes and generated by an encoder-decoder network. A prompt is added in front of the question (i.e., "Convert the question into a program:") and fed to the T5 model for program generation. The output sequence consists of interdependent functions, where "[N]" indicates the dependency. Then, a structured program tree with L layers is built from the sequence according to these dependencies, which can be used to guide step-by-step program execution.

[0067] 3) Obtain the intermediate supervision signal corresponding to each execution step through the program execution network;

[0068] Image features and L-layer program tree, the program executes step-by-step reasoning from layer 1 to layer L to obtain answers based on image features;

[0069] The details are as follows: Assume that the program tree contains N nodes, each node corresponds to a program text (such as "Select (giraffe)", "Select (elephant)", etc.), and use the program semantics to initialize the state matrix As shown below:

[0070]

[0071] Among them, prog i represents the program text of the ith node and is truncated or padded to a fixed length k; GloVe represents the GloVe embedding layer. In the process of step-by-step reasoning, the matrix S implies the intermediate reasoning states and results and can be used to decode the output of the DIS algorithm proposed in this invention. The input state of the lth layer is marked as S l-1 , the present invention uses the Transformer framework to obtain the final output state S l .

[0072] Specifically,

[0073] First, a masked self-attention layer is used to collect dependencies from the state of the previous layer, and then a guided attention layer is used to collect dependencies from image features. Find the visual clues in the code, which are as follows:

[0074]

[0075] Among them, MaskSA and GA represent the mask self-attention layer and the guided attention layer respectively; MaskSA uses the weight matrix M l Mask the non-dependent nodes, expressed as follows:

[0076]

[0077] in, Represents the mask matrix; if and only if node i is the parent node of node j, M ij =0, otherwise M ij = -∞; Q S ,K S ,V S It is the result of mapping S through a three-layer fully connected layer; its representation is as follows:

[0078]

[0079] Different from SA which collects information from itself, it guides attention to obtain features from other sources (e.g., visual cues), as follows:

[0080]

[0081] in, and Depend on After L program reasoning, the final state is used to predict the answer through a multi-layer perceptron (MLP) layer:

[0082]

[0083] in Represents the answer set The predicted scores of the answers in , and the answer with the highest score is selected as the final answer. Θ represents the model parameters. Finally, the cross entropy loss function is used to optimize the model, which is expressed as follows:

[0084]

[0085] in, and a gt denote the visual question answering dataset and the true answer, respectively.

[0086] Based on the execution of the above program, the present invention proposes an intermediate supervision method to limit the reasoning process and improve the answer prediction performance. The existing method is to calculate the probability distribution based on the IoU between the target bounding box and the true bounding box, which easily causes the model to focus on targets that are not relevant to the task. To this end, the present invention proposes a detection-based intermediate process supervision method, which formulates the intermediate supervision signal into a unified sequence form, thereby giving the model the ability to use different supervision types (such as bounding boxes, logical words (true / false), and text).

[0087] The detection-based intermediate process supervision method consists of two steps:

[0088] (1) Design manual rules for program execution and perform operations on the program tree on the ground truth map;

[0089] Execute the program on the given real annotated scene graph. The GQA dataset only provides real scene graph annotation information for the training set and the validation set. Therefore, DIS is only used to optimize the model in the training stage to produce intermediate results (such as targets, true / false, answers);

[0090] The present invention designs symbolic reasoning rules for each function type (such as Select, Relate, Verify, etc.), for example, Select(x) is defined as selecting the node corresponding to x, and Relate(x, to the left of) is defined as finding the node to the left of x. Through the program Program and the scene graph Scene Graph, intermediate supervision is gradually inferred.

[0091] Generally, supervision can be divided into three types: Objects with bounding boxes, True / False, and Answers. For Objects, the present invention uses the bin technology in the Pix2Seq network to convert continuous coordinate values ​​into discrete labels. If there are multiple associated targets, their corresponding bounding boxes will be randomly shuffled and connected to form a result sequence. For True / False and Answer, the present invention directly uses text tags to form a sequence. In addition, some special tags are added to the sequence for further generation, such as [BEG], [SEP], , etc.

[0092] (2) An intermediate result decoder is used to decode the intermediate output, and then an autoregressive loss is used to optimize the loss of the intermediate process supervision based on detection;

[0093] The present invention proposes a two-layer Transformer for transforming the state Decode intermediate output results;

[0094] First, the [BEG] tag and state Initialization prompts the decoder to generate an output sequence; in the training stage, the present invention uses teacher-forcing and autoregressive loss to optimize the model, the formula is as follows:

[0095]

[0096]

[0097] Where o represents the length of the resulting sequence. Represents the intermediate state result obtained from the i-th program node.

[0098] During the training phase of the model, the prediction loss of the answer and the loss of the intermediate supervision based on the detection are combined to optimize the model parameters by minimizing the following loss function:

[0099]

[0100] Among them, α represents the weight of the intermediate process supervision loss module in the entire optimization process. In the test phase, the model only needs to retain the last predicted answer module.

[0101] Experiments show that the detection-based intermediate supervision method (DIS) of the present invention achieves better results than the existing mainstream methods. Two benchmark visual question answering datasets are used for evaluation in the experiment, namely the GQA dataset and the GQA-Sub dataset. The detailed information of the datasets is shown in Tables 1 and 2.

[0102] Table 1 GQA dataset information

[0103]

[0104] Table 2 GQA-Sub dataset information

[0105]

[0106]

[0107] The experimental part aims to evaluate the effectiveness of the detection-based intermediate process supervision model proposed in the present invention on the GQA and GQA-Sub datasets. Specifically, the present invention uses the accuracy of the model on different question types (such as Binary questions, Open questions) and the overall accuracy (Accuracy) on the GQA dataset as the evaluation indicators of the model, and the experimental comparison results are given in Table 3. The overall accuracy (Acc) and (Acc-(Sub)) on combined questions and sub-questions on the GQA-Sub dataset, as well as the answer consistency measurement index RC(k) are used as evaluation indicators of the model, and the experimental comparison results are given in Table 4. The consistency measurement index RC(k) is calculated in the following way:

[0108]

[0109] in, Represents a set of combined questions. Correct(Q) = 1 if and only if question Q can be correctly answered by the VQA model, otherwise Correct(Q) = 0. If and only if problem Q and all its subproblems can be answered correctly by the VQA model, otherwise Therefore, the range of RC(k) value is between 0 and 1, and a larger value indicates a higher logical consistency.

[0110] Table 3 Comparative experimental results on the GQA dataset

[0111]

[0112] In the experiment, the present invention mainly evaluates the performance of the model of the present invention in answering combination questions and reasoning consistency, and the corresponding experimental results are presented in Tables 3 and 4, respectively. Table 3 shows the test results of the online GQA official test platform of the current most advanced model and the DIS method proposed in the present invention on the GQA dataset. These results also reflect the performance of all models on combination questions. The required input represents the information required for the model to predict the answer, where V and L represent visual and language modality information, respectively, and DataAug represents data augmentation. As shown in Table 3, the DIS method proposed in the present invention performs best in both Binary and Overall accuracy among the methods listed in Table 3. Specifically, under basically the same input and setting conditions as MMN, the DIS method exceeds the MMN model by 0.46% and 0.47% in Binary and Open questions, respectively.

[0113] Table 4 Comparative experimental results on the GQA-Sub dataset, where DA means that the model is trained using sub-questions in a data-enhanced manner during training.

[0114]

[0115]

[0116] In addition, the present invention also evaluates the performance of the proposed DIS in terms of reasoning consistency. The results of the DIS proposed in the present invention and the most advanced methods in terms of accuracy (i.e., Acc and Acc-(Sub)) and reasoning consistency (i.e., RC(k)) are shown in Table 4. Acc and Acc-(Sub) represent the accuracy on the val and val-sub data partitions, respectively. DA indicates that sub-problems are used as data enhancement for model training. It can be observed from Table 4 that the DIS proposed in the present invention outperforms other state-of-the-art methods in terms of accuracy Acc and Acc-(Sub) and reasoning consistency index RC(k). Specifically, without sub-problem data enhancement, DIS is 4.66% and 5.85% higher than MMN in val and val-sub partitions, respectively. At the same time, it can be observed that the performance of the DIS algorithm of the present invention in reasoning consistency has also been significantly improved, i.e., it has been improved by 6.19%, 9.57% and 14.9% on RC(1), RC(2) and RC(3), respectively. This superiority stems from the comprehensive and noise-free detection-based intermediate supervision signal provided by DIS, which significantly enhances the model's ability to answer combinatorial questions and the consistency of their corresponding sub-questions. When DIS is compared with the method of using sub-questions for data augmentation in train-sub, the DIS method proposed in this invention even outperforms the current RCVQA model, which specifically enhances reasoning consistency by introducing consistency constraint loss and has the best performance. Specifically, DIS exceeds RCVQA by 3.55% in the Acc indicator, and exceeds the three reasoning consistency indicators of RC(1), RC(2) and RC(3) by 1.64%, 2.26% and 2.48% respectively, which shows the effectiveness of the method proposed in this invention.

[0117] It should be understood that those skilled in the art can make improvements or changes based on the above description, and all these improvements and changes should fall within the scope of protection of the appended claims of the present invention.

Claims

1. A visual question answering method based on an intermediate process supervision strategy for target detection, characterized in that: The following steps are involved: 1) Extract image features from visual images through convolutional neural networks; 2) Use an encoder-decoder network model to parse the question with answer into a program tree; 3) Obtain the intermediate supervision signal corresponding to each execution step through the program execution network; For the image features and the L-layer program tree, the program performs step-by-step reasoning from the 1st layer to the Lth layer to obtain the answer based on the image features; The details are as follows: Assume that the program tree contains N nodes, each node corresponds to a program text, and initialize the state matrix using program semantics As shown below: Among them, prog i represents the program text of the i-th node and is truncated or padded to a fixed length k; GloVe represents the GloVe embedding layer; The input state of the lth layer is labeled S l-1 , use the Transformer framework to obtain the final output state S l ; Specifically, First, a masked self-attention layer is used to collect dependencies from the state of the previous layer, and then a guided attention layer is used to collect dependencies from image features. Find the visual clues in the code, which are as follows: Among them, MaskSA and GA represent the mask self-attention layer and the guided attention layer respectively; MaskSA uses the weight matrix M l Mask the non-dependent nodes, expressed as follows: in, Represents the mask matrix; if and only if node i is the parent node of node j, M ij =0, otherwise M ij = -∞; Q S ,K S ,V S It is the result of mapping S through a three-layer fully connected layer; its representation is as follows: Q S =FC Q (S) K S =FC J (S); In S =FC V (S) The features that guide the attention layer are expressed as follows: in, and Depend on Derived; after L program reasoning, the final state is used to predict the answer through the multi-layer perception MLP layer: in, Represents the answer set The predicted score of the answer in , the answer with the highest score is selected as the final answer; Θ represents the model parameter; Finally, the cross entropy loss function is used to optimize the model, which is expressed as follows: in, and a gt denote the visual question answering dataset and the true answer, respectively.

2. The visual question answering method based on the intermediate process supervision strategy of target detection according to claim 1, characterized in that: The step 1) is specifically as follows: 1.1) Given an image I, extract feature maps using the pre-trained CNN backbone of Faster R-CNN Among them, H and W represent the height and width of the feature map respectively, and d v Represents the dimension of the feature; 1.2) Give image features visual context and cross-modal text information, using the MCAN method to encode questions and images through Transformer blocks; The details are as follows: For a given question Q of length T, embed it into the latent space Then a two-layer Transformer is used to encode the problem as follows: Among them, SA stands for self-attention; LN stands for layer normalization; FFN stands for feedforward neural network; Using two-layer Transformer to use visual context and question semantics To enrich the image features V, as follows: Among them, PosEmb represents position embedding; FC represents the fully connected layer, which transforms the feature dimension from d v Convert to d h ; GA represents guided attention, which utilizes question semantics to enhance relevant visual features.

3. The visual question answering method based on the intermediate process supervision strategy of target detection according to claim 1, characterized in that: The step 2) is as follows: 2.1) Follow the MMN method to generate the corresponding executable program; Use the MMN method to generate the corresponding executable program from the problem; 2.2) Based on the above program types, the encoder-decoder network generates a complete program tree.

4. The visual question answering method based on the intermediate process supervision strategy of target detection according to claim 1, characterized in that: In the step 3), during the step-by-step reasoning from the 1st layer to the Lth layer, an intermediate supervision method is used to restrict the reasoning process and improve the answer prediction performance; The detection-based intermediate process supervision method consists of two steps: 3.1) Design manual rules for program execution to perform operations on the program tree on the ground truth map; 3.2) Use the intermediate result decoder to decode the intermediate output, and then use the autoregressive loss to optimize the loss of the intermediate process supervision based on the detection; Using a two-layer Transformer, from the state Decode intermediate output results; First, the [BEG] tag and state Initialization prompts the decoder to generate output sequences; during the training phase, teacher-forcing and autoregressive loss are used to optimize the model, as follows: Among them, o represents the length of the result sequence, Represents the intermediate state result obtained from the i-th program node.

5. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images