A Generative Visual Commonsense Reasoning and Explanation Method Based on Scene Graph Generation

Abstract

The present invention discloses a generative visual common sense reasoning and explanation method based on scene graph generation. The method includes: adding a single-head attention and gating fusion mechanism to the language large model Flan-T5 for multimodal fusion; in the scene graph generation stage, using an image patch sequence and a prompt template containing object information to generate a scene graph triple sequence; in the visual common sense reasoning and explanation generation stage, combining the scene graph with the question to construct a prompt template, and designing an automatic scene graph selection module based on confidence scores, enabling the model to automatically select appropriate triples and generate answers and explanations. Better metric scores are obtained on the VCR dataset than models such as e-UG. The present invention addresses the problem of the single modality of the language large model, combines multimodal fusion techniques, and through constructing a prompt template and a confidence-based scene graph selection mechanism, gives full play to the semantic understanding characteristics of the large model, endows it with the ability to deeply understand visual information, and thus conducts reasoning and explanation.

Description

Technical Field

[0001] The present invention relates to a generative visual common sense reasoning and interpretation method based on scene graph generation, belonging to the field of natural language processing in the computer field. Background Art

[0002] Visual common sense reasoning is a classic task in the cross - field of natural language processing and computer vision. How to generate coherent and logical reasoning explanations under the condition of recognizing visual scenes has always been a very challenging problem. With the development of deep neural networks, deep neural network models with more parameters and more complex structures have been continuously proposed and applied to multi - modal scenarios. However, the models often have difficulty capturing the hidden relationships between things in visual information and making common - sense explanations. Limited by this, this task has been limited to making better answers through multiple - choice. However, such a setting is difficult to improve the interpretability of the model and has poor transferability.

[0003] Recently, the success of large models and the rise of tasks such as natural language explanations have triggered a new wave of enthusiasm for model interpretability research. More and more work has shifted its focus to model reasoning ability and interpretability. Methods such as knowledge graphs and chain of thought have been gradually adopted in various directions. At the same time, the related research on Prompt is in full swing. PromptLearning is based on language models and directly models text probabilities. A large number of works have shown that in the context of large language models, using Prompt combined with instruction fine - tuning can give full play to the powerful semantic understanding ability of the model to a greater extent and generate according to instructions. Therefore, the original input can be modified into a text string with some unfilled slots using the template Prompt, and then the large language model is used to probabilistically fill in the unfilled information to obtain the final output. Summary of the Invention

[0004] The technical problem to be solved by the present invention:

[0005] The purpose of the present invention is to design a generative visual common sense reasoning and interpretation method based on scene graph generation to solve the problems that the existing visual common sense reasoning models lack generative ability, have poor transferability and interpretability. In a visual scene, there are certain correlations between various things, and this relationship is very important when cognizing and understanding the environment. When humans observe a scene and make inferences, they can reasonably utilize these hidden relationships between things to make inferences and explanations, and this way of thinking is lacking in the model. The present invention provides a model framework with a similar way of thinking, which generates a scene graph and reasoning explanations in stages in a generative manner, can alleviate the problems of insufficient visual information granularity and lack of reasoning ability of the model, and improve transferability.

[0006] The present invention adopts the following technical solutions to solve the technical problems:

[0007] A generative visual commonsense reasoning and interpretation method based on scene graph generation, comprising the following steps:

[0008] (1) Scene graph generation: Obtain a patch sequence from an image and a prompt template containing object information to generate a corresponding scene graph triple sequence;

[0009] (2) Automatic scene graph selection based on confidence scores: The confidence scores of scene graph triple sequences are obtained through the CLIP model. The confidence scores are used to guide the visual common sense reasoning and interpretation generation model to automatically select scene graph triple sequences that conform to the object relationships in the image.

[0010] (3) Visual common sense reasoning and explanation generation: Using the Flan-T5 large language model, the scene graph generated by the scene graph triple sequence generated in step (1) and the question are constructed into a Prompt prompt template, and combined with the image patch sequence to generate answers and explanations to visual common sense reasoning questions.

[0011] Preferably, the implementation process of step (1) is: constructing a Prompt prompt template containing object information:<subject1,object1;subject2,object2;...> , obtain an improved Flan-T5 large language model, the improved Flan-T5 large language model includes an encoder-decoder structure, and a multimodal fusion module is connected in series at the front end of the encoder. The calculation formula of the multimodal fusion module is:

[0012] H c =Attention(H t W Q ,H v W K ,H v W V )

[0013] λ=Sigmoid(W t H t +W v H c )

[0014] H f =(1-λ)H t +λH c

[0015] Among them, H t It is a prompt template for text input, which contains object information. v is the visual input, i.e., a sequence of image patches, Hc is the hidden state, H f is a vector; W Q , W K , W V , W t , W v are learnable parameters, and λ is a weight hyperparameter;

[0016] The input of the Flan-T5 large language model is a sequence of image patches and a Prompt prompt template containing object information, and the output is a sequence of scene graph triples: <subject1, object1, relation1; subject2, object2, relation2;...>.

[0017] Preferably, the implementation process of step (2) is as follows:

[0018] (2.1) Obtain a visual common sense reasoning and explanation generation model, where the visual common sense reasoning and explanation generation model includes a CLIP module, the CLIP module is connected in series to the improved Flan-T5 large language model, the input of the CLIP module is the sequence of scene graph triples and the image output by the improved Flan-T5 large language model, and the output is the normalized confidence score of the sequence of scene graph triples;

[0019] (2.2) Obtain a weighted text representation according to the normalized confidence score:

[0020]

[0021] According to e ij is the constructed Prompt prompt template <Context, Question>, and c ij is the weight of the normalized confidence score.

[0022] Preferably, the implementation process of step (3) is as follows:

[0023] (3.1) Construct a Prompt prompt template: <Context, Question>, fill the generated sequence of scene graph triples and the questions in the dataset into the Prompt prompt template: <Context, Question> respectively as text inputs, and use the sequence of image patches in step (1) as visual inputs;

[0024] (3.2) Adopt the improved Flan-T5 large language model, input the patch sequence of the image in step (1), the Prompt prompt template constructed in step (3.1): <Context,Question>, and the normalized confidence score of the scene graph triple sequence, and output the answer and explanation for answering visual common sense questions.

[0025] Compared with the prior art, the present invention adopts the above technical solutions and has the following beneficial effects:

[0026] Adopt the patch sequence of the image as the visual input, which can reduce the information loss caused by compression during feature extraction. At the same time, using modality fusion endows the single-modal language model with the ability to solve multi-modal problems. Secondly, an automatic selection method of the scene graph based on the confidence score is provided, which enables the model to automatically screen out high-quality triples from the generated scene graphs and provide fine-grained visual common sense information. Experiments show that the model trained by the generative visual common sense reasoning and explanation method based on scene graph generation of the present invention can generate high-quality scene graph relationship sequences according to image information, and can reasonably select scene graphs to make reasoning answers and explanations, effectively enhancing the interpretability of the large language model. Description of the Drawings

[0027] Figure 1 It is a schematic diagram of the overall process of the proposed algorithm framework. Detailed Embodiments

[0028] The following further describes the present invention in detail with reference to the drawings.

[0029] To better understand the visual scene and make credible reasoning explanations, the overall framework of the generative visual common sense reasoning and explanation method based on scene graph generation proposed by the present invention is as Figure 1 shown, which is divided into three parts: 1. Construction and generation of the scene graph. 2. Automatic selection of the scene graph based on the confidence score. 3. Generation of visual common sense reasoning and explanation.

[0030] (1) Construction and generation of the scene graph

[0031] The construction and generation of the scene graph proposed by the present invention follows the setting without position information. To be precise, different from the existing scene graph generation methods, all cumbersome position information is removed. To provide simpler and more effective training data, we sort all the scene graphs corresponding to each image according to the average size of the objects and select a fixed number to participate in the training.

[0032] During the process of generating the scene graph, the Flan-T5 large language model is used. The Flan-T5 model is a single-modal language model with an encoder-decoder structure. Therefore, in order to endow it with the ability of multi-modal understanding and generation, a gating fusion mechanism is introduced to fuse the embedded text input H t and the visual input H v , and the process can be expressed as follows:

[0033] H c = Attention(H t W Q ,H v W K ,H v W V )

[0034] λ = Sigmoid(W t H t +W v H c )

[0035] H f =(1 - λ)H t +λH c

[0036] Among them, W Q , W K , W V , W t , W v are learnable parameters, and λ is a weight hyperparameter. H c is the hidden state obtained by cross-modal attention calculation of the text input and the visual input, and H f is a vector that fuses the two state information of H t and H c ;

[0037] Following the characteristics of the language model, in order to obtain the triple representation sequence of the scene graph, the instruction is designed as follows:

[0038] "Create a scene graph based on an image that includes multipleobjects.Your task is to identify the key elements andrelationships betweenthese objects in the image,as well as their spatial arrangement within thescene.Objects:{X c}Scene:"

[0039] Subsequently, add X containing object information c = <sub1 obj1; sub2 obj2; …; sub n obj n > sequence to form a Prompt template. By inputting an image patch sequence and a text Prompt, a scene graph containing object relationship information is generated.

[0040] (2) Automatic selection of scene graph based on confidence score

[0041] To ensure more accurate and reasonable reasoning, the scene graph triples used for assisting reasoning must be as accurate as possible. Therefore, high-quality generated triples must be selected. We propose two selection methods:

[0042] 1) Threshold-based scene graph selection is a manual selection method. According to the generated scene graph and the picture, obtain the normalized confidence scores of the scene graph from the CILP model and sort them in descending order. Manually set a threshold, and select the scene graph with the largest confidence score from the set of unselected scene graphs and add it to the set of selected scene graphs until the cumulative score of the selected scene graph sequence reaches or exceeds the threshold. Set multiple values and find a better threshold.

[0043] 2) Automatic selection of scene graph based on confidence score is a method that guides the model to automatically select scene graphs during training and prediction according to the confidence scores. According to the generated scene graph sequence and the original image, obtain the normalized confidence scores of the scene graph from the CILP model. These scores will be extended to the dimension of the matching text embedding tensor, and the weighted calculation of the embedded input will be performed. More precisely, the part associated with each triple consists of the same corresponding scores, while the part outside the scene graph sequence is assigned a default value of 1. According to the input representation e ij and the confidence weight c ij , obtain the confidence-weighted text representation through the following expression:

[0044]

[0045] Compared with the threshold-based selection method, the automatic selection method of scene graph based on confidence score can avoid the cumbersome process of selecting a threshold. Moreover, the automatic selection method can reduce human intervention and prompt the diversity of model performance. After comparison, the automatic selection method based on confidence is finally selected. After training, the model can rely on its own semantic understanding ability to select appropriate scene graphs.

[0046] Through this selection method, it helps the model to pay more attention to triples with higher confidence, while reducing the possibility of introducing additional noise due to the selection of triples with lower confidence scores.

[0047] (3) Visual Commonsense Reasoning and Explanation Generation

[0048] After obtaining the generated sequence of scene graph triples, visual commonsense reasoning answers and explanation generation are performed. The visual input uses a sequence of patches from the image, and the text input constructs a Prompt template based on the question and the scene graph:

[0049] "Context:{G}Question:{Q}"

[0050] Where G represents the sequence of scene graphs, Q represents the question. Also using the Flan-T5 large language model and the early multi-modal fusion mechanism, an automatic scene graph selection method based on confidence scores is added before decoding, and the model is supervised according to:

[0051] "{A}Because:{E}" format to generate answers and detailed explanations at the same time. Where A represents the answer and E represents the explanation. Through this process, the generated scene graph can be used to supplement visual information, integrate the semantic understanding ability of the large model, and generate complete, fluent, interpretable reasoning and answers.

[0052] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. A generative visual common sense reasoning and interpretation method based on scene graph generation, characterized in that It includes the following steps: (1) Generation of scene graph: Obtain a sequence of patches from an image and a Prompt template containing object information, and generate a corresponding sequence of scene graph triples; (2) Automatic selection of scene graph based on confidence scores: Obtain the confidence scores of the sequence of scene graph triples through the CLIP model, and use the confidence scores to guide the visual common sense reasoning and explanation generation model to automatically select the sequence of scene graph triples that conform to the object relationships in the image; (3) Visual common sense reasoning and explanation generation: Use the Flan-T5 large language model to construct a Prompt template with the scene graph generated from the sequence of scene graph triples generated in step (1) and a question, and combine it with the image patch sequence to generate answers and explanations to visual common sense reasoning questions; The implementation process of step (1) is as follows: Construct a Prompt template containing object information: <subject1,object1; subject2,object2;...>, obtain an improved Flan-T5 large language model, and the improved Flan-T5 large language model includes an encoder-decoder structure. A multimodal fusion module is connected in series at the front end of the encoder. The calculation formula of the multimodal fusion module is: H c = Attention(H t W Q , H v W K , H v W V ) λ = Sigmoid(W t H t + W v H c ) H f = (1 - λ)H t + λH c Among them, H t is the Prompt prompt template for text input that contains object information, H v is the patch sequence of the visual input, i.e., the image, H c is the hidden state, H f is the vector; W Q , W K , W V , W t , W v are learnable parameters, and λ is the weight hyperparameter; The input of the Flan-T5 large language model is the sequence of patches of the image and the Prompt template containing object information, and the output is the sequence of scene graph triples: <subject1,object1,relation1; subject2,object2,relation2;...>.

2. The generative visual common sense reasoning and explanation method based on scene graph generation according to claim 1, characterized in that, The implementation process of step (2) is as follows: (2.1) Obtain a visual common sense reasoning and explanation generation model, and the visual common sense reasoning and explanation generation model includes a CLIP module. The CLIP module is connected in series to the improved Flan-T5 large language model. The input of the CLIP module is the sequence of scene graph triples output by the improved Flan-T5 large language model and the image, and the output is the normalized confidence scores of the sequence of scene graph triples; (2.2) Obtain a weighted text representation according to the normalized confidence scores; According to e ij is the constructed Prompt template <Context,Question>, c ij is the weight of the normalized confidence score.

3. The generative visual common sense reasoning and explanation method based on scene graph generation according to claim 2, wherein, The implementation process of step (3) is as follows: (3.1) Construct a Prompt template: <Context,Question>, fill the generated sequence of scene graph triples and the questions in the dataset into the Prompt template: <Context,Question> respectively as text inputs, and use the sequence of patches of the image in step (1) as visual inputs; (3.2) Use the improved Flan-T5 large language model to input the sequence of patches of the image in step (1), the Prompt template <Context,Question> constructed in step (3.1), and the normalized confidence scores of the sequence of scene graph triples, and output answers and explanations to respond to visual common sense questions.

Images