Zero-shot visual question answering method and system based on boolean prompt enhancement

Abstract

The application discloses a zero sample visual question answering method and system based on Boolean prompt enhancement, relates to the technical field of zero sample visual question answering, and comprises the following steps: judging the complexity of an input question based on a pre-trained visual language model, outputting the answer to a simple question, and selecting a question with high complexity for the model; extracting keywords based on the selected complex question, and generating a description of an image corresponding to the question; generating a sub-question capturing global information based on the original question, generating a sub-question containing local information from the keywords of the question and the image description; constructing a redundancy value and a richness value of the sub-question based on the cosine similarity between the generated sub-question and the original question; deleting the sub-question with high redundancy by using the redundancy value, ensuring the richness of the content of the sub-question by using the richness value, using the obtained sub-question as a prompt to enhance the understanding of the complex question and the attention to local visual information of the model, and improving the accuracy of zero sample visual question answering.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of zero-shot visual question answering, in particular to a zero-shot visual question answering method and system based on Boolean prompt enhancement. BACKGROUND

[0002] With the rapid development of convolutional neural networks, the accuracy of visual question answering has been greatly improved. However, most current visual question answering methods are based on a large amount of labeled data for training, which covers specific question types and answer categories. However, in practical applications, the data processed by various industries will always have new question types and answer categories that are not included in the training data. Such visual question answering models trained based on labeled data can only handle known question and answer pairs in the training phase. When encountering completely new questions that have never appeared in the training set, the model often performs poorly. This limitation severely limits the application of visual question answering technology in more extensive and complex real-world environments. To overcome this challenge, researchers have begun to introduce zero-shot technology into the visual question answering task, allowing the model to attempt to answer questions that have never been seen based on existing knowledge and model structure without training for new question types and answer categories, thereby making the visual question answering system more versatile and generalizable.

[0003] Recently, many zero-shot visual question answering methods have attempted to utilize pre-trained visual language models to convert questions into model-predefined input templates to fully utilize the knowledge learned by pre-trained visual language models to achieve zero-shot visual question answering. However, these methods have two shortcomings: first, when encountering complex questions, existing methods often ignore the textual details and associations in the question, hindering a comprehensive understanding of the question; second, existing methods do not focus on local image information, resulting in the omission of important details in the image when the image content is particularly complex or requires detailed observation. Therefore, the key to the zero-shot visual question answering task lies in how to handle complex questions and how to enhance the attention to local image information. SUMMARY

[0004] To address the shortcomings mentioned in the background, the purpose of the present application is to provide a zero-shot visual question answering method and system based on Boolean prompt enhancement.

[0005] In a first aspect, the purpose of the present application can be achieved by the following technical solution: a zero-shot visual question answering method based on Boolean prompt enhancement, the method comprising the following steps:

[0006] An original question is obtained and input into a pre-trained visual language model, an original answer is output and input into the pre-trained visual language model, the confidence of the original answer is adaptively judged to determine the complexity of the original question, and if it is a simple question, the original answer is output.

[0007] If it is a complex problem, the complex problem is input into the pre-trained visual language model, and keywords and corresponding image descriptions of the complex problem are output, and two types of sub-problems are generated based on the complex problem and the keywords and corresponding image descriptions of the complex problem, wherein the two types of sub-problems respectively contain global information and local information;

[0008] Cosine similarity between the two types of sub-problems and the complex problem and cosine similarity between the two types of sub-problems are calculated, and redundancy and richness values of the two types of sub-problems are calculated based on the cosine similarity between the two types of sub-problems and the complex problem and the cosine similarity between the two types of sub-problems;

[0009] The two types of sub-problems are optimized based on the redundancy and richness values, and the optimized sub-problems are input into the pre-trained visual language model, thereby enhancing understanding of the complex problem and attention to local visual information and improving accuracy of zero-shot visual question answering.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further comprises that the pre-trained visual language model is as follows:

[0011] a i =PVLM(p(q i ,v i ),θ)

[0012] In the formula, PVLM represents a visual language model network Blip-2 or MiniGPT-4; θ represents pre-trained weight parameters; p represents a prompt input into the visual language model; q i represents the i-th question input into the model; v i represents the i-th image input into the model; and a i represents output corresponding to the prompt p, the i-th question and the image. In combination with the first aspect, in some implementations of the first aspect, the method further comprises that a mechanism for adaptively judging complexity of an original question based on confidence of an original answer is as follows:

[0013] bool i =CE(p d (q i ),a i )

[0014] In the formula, CE represents a visual language model confidence network; p d represents a prompt input into the visual language model, which makes the model output a Boolean type reply; q i represents the i-th question input into the model; and a i represents a corresponding question q ibool i the answer to the question q i the answer a to the question q i the degree of approval of the answer a, in the form of “Yes” or “No”;

[0015] if the answer is “Yes”, determining that the original question is a simple question and outputting the original answer;

[0016] if the answer is “No”, determining that the original question is a complex question.

[0017] In combination with the first aspect, in some implementations of the first aspect, the method further comprises the processes of keyword extraction, image description generation, and two types of sub-question generation:

[0018] inputting the original complex question into the pre-trained visual language model, setting an extraction question keyword prompt, and causing the pre-trained visual language model to extract keywords related to the complex question;

[0019] inputting the corresponding image of the original complex question into the pre-trained visual language model, setting a generation image description prompt, and causing the pre-trained visual language model to generate a description of the image;

[0020] inputting the original complex question and the corresponding image into the pre-trained visual language model, setting a question asking prompt, and causing the pre-trained visual language model to generate a question about the global information of the complex question based on the content of the complex question and the corresponding image, as a first type of sub-question;

[0021] inputting the keywords and the corresponding image into the pre-trained visual language model, setting a keyword asking prompt, and causing the pre-trained visual language model to generate a sub-question about the local information of the question based on the content of the keywords and the image, as a sub-question for extracting the local information of the question;

[0022] inputting the image description into the pre-trained visual language model, setting an image description asking prompt, and causing the pre-trained visual language model to generate a sub-question about the local information of the image based on the content of the image description, as a sub-question for extracting the local information of the image;

[0023] merging the sub-question for extracting the local information of the question and the sub-question for extracting the local information of the image, as a second type of sub-question.

[0024] In combination with the first aspect, in some implementations of the first aspect, the method further comprises the processes of calculating the redundancy value re and the richness value r:

[0025] the redundancy value re:

[0026]

[0027] Wherein, N1 represents the number of keywords; e i represents the cosine similarity between the i th keyword and the original question; s j represents the cosine similarity between the j th sub-question and the original question; the richness value r:

[0028]

[0029] Wherein, s i represents the cosine similarity between the i th sub-question and the original question; s j represents the cosine similarity between the j th sub-question and the original question; u ij represents the cosine similarity between the i th sub-question and the j th sub-question; n represents the number of sub-questions;

[0030] Cosine similarity calculation:

[0031]

[0032] Wherein, A i and B i are the i th components of vectors A and B respectively.

[0033] In combination with the first aspect, in some implementations of the first aspect, the method further comprises: the process of optimizing the two types of sub-questions based on the redundancy value and the richness value to obtain the optimized sub-questions is as follows:

[0034] Based on the redundancy value of each sub-question, it is determined whether the set redundancy threshold is exceeded, and if the threshold is exceeded, the sub-question is determined to be a redundant sub-question and is deleted;

[0035] Based on the richness value of the sub-question after excluding redundancy, it is determined whether the set richness threshold is reached, and if not, the sub-question is continuously generated, the redundant question is excluded, and the richness value is calculated until the threshold is reached, and the obtained sub-question is used as the optimized sub-question.

[0036] The second aspect, in order to achieve the above object, the application discloses a zero sample visual question and answer system based on Boolean prompt enhancement, comprising:

[0037] The adaptive judgment module is used for obtaining an original question, inputting the original question into a pre-trained visual language model, outputting an original answer, inputting the original answer into a pre-trained visual language model, and adaptively judging the complexity of the original question according to the confidence of the original answer. If it is a simple question, the original answer is outputted;

[0038] The sub-question generation module is configured to, if the complex question is input into a pre-trained visual language model, output keywords and corresponding image descriptions of the complex question, and generate two types of sub-questions based on the complex question and the keywords and corresponding image descriptions of the complex question, wherein the two types of sub-questions respectively include global information and local information.

[0039] The redundancy richness calculation module is configured to calculate cosine similarity between the two types of sub-questions and the complex question and cosine similarity between the two types of sub-questions, and calculate redundancy values and richness values of the two types of sub-questions based on the cosine similarity between the two types of sub-questions and the complex question and the cosine similarity between the two types of sub-questions.

[0040] The sub-question screening module is configured to optimize the two types of sub-questions based on the redundancy values and the richness values, obtain optimized sub-questions, input the optimized sub-questions into the pre-trained visual language model, enhance understanding of the complex question and attention to local visual information, and improve accuracy of zero-shot visual question answering.

[0041] In combination with the second aspect, in some implementations of the second aspect, the system further includes that the pre-trained visual language model of the adaptive judgment module is as follows:

[0042] a i =PVIM(p(q i ,v i ),θ)

[0043] In the formula, PVLM represents a visual language model network Blip-2 or MiniGPT-4, θ represents pre-trained weight parameters, p represents a prompt input into the visual language model, q i represents an i-th question input into the model, v i represents an i-th image input into the model, and a i represents output corresponding to the prompt p, the i-th question and the image; and a mechanism for the adaptive judgment module to adaptively judge complexity of an original question according to confidence of an original answer is as follows:

[0044] bool i =CE(p d (q i ),a i )

[0045] In the formula, CE represents a visual language model confidence network, p d represents a prompt input into the visual language model to make the model output a Boolean type reply, q i represents an i-th question input into the model, and a i represents an answer of the corresponding question q i .i the recognition degree of the answer a of the question q represented by the visual language model, in the form of "Yes" or "No"; i i the recognition degree of the answer a of the question q represented by the visual language model, in the form of "Yes" or "No";

[0046] if the answer is "Yes", the original question is determined as a simple question, and the original answer is outputted;

[0047] if the answer is "No", the original question is determined as a complex question;

[0048] the keyword extraction, image description generation and two types of sub-question generation processes:

[0049] inputting the original complex question into the pre-trained visual language model, setting a keyword extraction prompt, and making the pre-trained visual language model extract keywords about the complex question;

[0050] inputting the corresponding image of the original complex question into the pre-trained visual language model, setting an image description generation prompt, and making the pre-trained visual language model generate a description about the image;

[0051] inputting the original complex question and its corresponding image into the pre-trained visual language model, setting a question asking prompt, and making the pre-trained visual language model generate a question about the global information of the complex question according to the content of the complex question and its corresponding image, as a first type of sub-question;

[0052] inputting the keywords and the corresponding image into the pre-trained visual language model, setting a keyword asking prompt, and making the pre-trained visual language model generate a sub-question about the local information of the question according to the keyword and the image content, as a sub-question for extracting the local information of the question;

[0053] inputting the image description into the pre-trained visual language model, setting an image description asking prompt, and making the pre-trained visual language model generate a sub-question about the local information of the image according to the image description content, as a sub-question for extracting the local information of the image;

[0054] merging the sub-question for extracting the local information of the question and the sub-question for extracting the local information of the image, as a second type of sub-question;

[0055] the calculation process of the redundancy value re of the redundancy richness calculation module:

[0056]

[0057] wherein, N1 represents the number of keywords; e i represents the cosine similarity between the i-th keyword and the original question; s j represents the cosine similarity between the j-th sub-question and the original question; ​

[0058] The redundancy richness calculation module calculates the redundancy richness value r as follows:

[0059]

[0060] In the formula, s i represents the cosine similarity between the i-th sub-question and the original question; s j represents the cosine similarity between the j-th sub-question and the original question; u ij represents the cosine similarity between the i-th sub-question and the j-th sub-question; n represents the number of sub-questions;

[0061] The cosine similarity calculation process is as follows:

[0062]

[0063] In the formula, A i and B i are the i-th components of vectors A and B, respectively;

[0064] The sub-question screening module optimizes the two types of sub-questions based on the redundancy value and the richness value, and the process of obtaining the optimized sub-questions is as follows:

[0065] Based on the redundancy value of each sub-question, it is determined whether the set redundancy threshold is exceeded, and if the threshold is exceeded, the sub-question is determined to be a redundant sub-question and is deleted.

[0066] Based on the richness value of the sub-questions after excluding redundancy, it is determined whether the set richness threshold is reached, and if not, the sub-questions are continuously generated, the redundant questions are excluded, and the richness value is calculated until the threshold is reached, and the obtained sub-questions are used as the optimized sub-questions.

[0067] In another aspect of the present application, in order to achieve the above-mentioned purpose, a terminal device is disclosed, which comprises a memory, a processor, and a computer program stored in the memory and capable of running on the processor, the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the zero-shot visual question answering method based on Boolean prompt enhancement as described above is adopted.

[0068] In still another aspect of the present application, in order to achieve the above-mentioned purpose, a computer readable storage medium is disclosed, which stores a computer program, and when the computer program is loaded and executed by a processor, the zero-shot visual question answering method based on Boolean prompt enhancement as described above is adopted.

[0069] The present application has the following beneficial effects:

[0070] This invention effectively addresses common problems in existing zero-shot visual question answering methods, namely, ignoring textual details and associations in complex questions and neglecting some important details in images. This improves the accuracy of zero-shot visual question answering and makes visual question answering systems more versatile and generalizable. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, those skilled in the art can derive other drawings based on these drawings without inventive effort.

[0072] Figure 1 It is a schematic flow chart of the method of the present invention;

[0073] Figure 2 It is a structural block diagram of the present invention;

[0074] Figure 3 Schematic diagram of the system structure of the present invention;

[0075] Figure 4 2 is a schematic diagram for comparing the verification effects of the present invention. DETAILED DESCRIPTION

[0076] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0077] Example 1:

[0078] like Figure 1 As shown, the zero-shot visual question answering method based on Boolean prompt enhancement includes the following steps:

[0079] S101: Obtain an original question, input the original question into a pre-trained visual language model, output an original answer, input the original answer into the pre-trained visual language model, adaptively judge the complexity of the original question based on the confidence of the original answer, and output the original answer if it is a simple question;

[0080] The pre-trained visual language models are as follows:

[0081] a i =PVLM(p(q i ,v i ),θ)

[0082] Where PVLM represents the visual language model network Blip-2 or MiniGPT-4; θ represents the pre-trained weight parameter; p represents the prompt input to the visual language model; q i represents the i-th question of the input model; v i Represents the i-th picture of the input model; a i Represents the output of the i-th question and image corresponding to the prompt p; the process of adaptively judging the complexity of the original question based on the confidence of the original answer is as follows:

[0083] bool i =CE(p d (q i ),a i )

[0084] Where CE represents the confidence network of the visual language model; p d represents the prompt input to the visual language model that causes the model to output a Boolean response; q i represents the i-th question of the input model; a i Represents the corresponding question q i Answer: bool i Represents the visual language model for question q i The answer is a i The degree of approval, in the form of "Yes" or "No";

[0085] Specifically, for the input question q i , using a pre-trained visual language model (such as BLIP-2 or MiniGPT-4) and a prompt p that enables the pre-trained visual language model to complete the visual question answering task, and let the pre-trained visual language model generate q i The answer is a i ;

[0086] q i and a i As input, the visual language model confidence network is input and the pre-trained visual language model is used to determine whether it agrees with a i As q i Answer hint p d , so that the pre-trained visual language model generates a Boolean answer again i ;

[0087] The obtained bool i If "Yes", then q i For simple questions, the model can answer correctly without further operation and outputs the answer a i ;

[0088] The obtained bool i If the answer is "No", determine q i It is a complex problem, and the model is difficult to answer.

[0089] S102: If it is a complex problem, input the complex problem into the pre-trained visual language model, output the keywords of the complex problem and the corresponding image description, and generate two types of sub-questions based on the complex problem and the keywords and corresponding image description of the complex problem, wherein the two types of sub-questions respectively contain global information and local information;

[0090] The keyword extraction, image description generation, and two types of sub-question generation processes are as follows:

[0091] Input the original complex problem into the pre-trained visual language model, set the keyword extraction prompt, and make the pre-trained visual language model extract the keywords of the complex problem;

[0092] Input the corresponding image of the original complex problem into the pre-trained visual language model, set the image description generation prompt, and make the pre-trained visual language model generate the description of the image;

[0093] Input the original complex problem and its corresponding image into the pre-trained visual language model, set the question prompt, and make the pre-trained visual language model generate a question about the global information of the problem according to the content of the complex problem and its corresponding image, as the first type of sub-question;

[0094] Input the keywords and corresponding image into the pre-trained visual language model, set the keyword question prompt, and make the pre-trained visual language model generate a sub-question about the local information of the problem according to the keyword and image content, as the sub-question for extracting the local information of the problem;

[0095] Input the image description into the pre-trained visual language model, set the image description question prompt, and make the pre-trained visual language model generate a sub-question about the local information of the image according to the image description content, as the sub-question for extracting the local information of the image;

[0096] Merge the sub-question for extracting the local information of the problem and the sub-question for extracting the local information of the image, as the second type of sub-question;

[0097] Based on the screened complex question q i , use the pre-trained visual language model to extract the text keyword prompt p k , make the pre-trained visual language model extract the keywords w i of the question q j , and form the set of all keywords Ψ i of q i ;

[0098] based on the complex question q i corresponding image v i , using the prompt p that makes the pre-trained visual language model generate an image description c to make the pre-trained visual language model generate an image v i description cap i .

[0099] based on the extracted keywords Ψ i and image description cap i , using the prompt p that makes the pre-trained visual language model generate a question b to generate a sub-question Θ containing local information of the original question l ;

[0100] based on the original question q i and image v i , using the prompt p that makes the pre-trained visual language model generate a question b to generate a sub-question Θ containing global information of the original question g , merge the sub-questions Θ l and Θ g to obtain the sub-question Θ of the original question q i .

[0101] S103: Calculate the cosine similarity between the two types of sub-questions and the complex question, and the cosine similarity between the two types of sub-questions, and calculate the redundancy value and richness value of the two types of sub-questions based on the cosine similarity between the two types of sub-questions and the complex question, and the cosine similarity between the two types of sub-questions;

[0102] Based on the cosine similarity, calculate the similarity between the generated sub-question Θ and the original question, and the similarity between the sub-questions, and construct the redundancy value;

[0103] The redundancy value is:

[0104]

[0105] In the formula, N1 represents the number of keywords; e i represents the cosine similarity between the i-th keyword and the original question; s j represents the cosine similarity between the j-th sub-question and the original question.

[0106] Based on the cosine similarity, consider the similarity between the sub-questions, calculate the difference value between the generated sub-question Θ and the original question, and construct the richness value;

[0107] The richness value is:

[0108]

[0109] In the formula, s i represents the cosine similarity of the i-th sub-question and the original question; s j represents the cosine similarity of the j-th sub-question and the original question; u ij represents the cosine similarity of the i-th sub-question and the j-th sub-question.

[0110] The cosine similarity is calculated as:

[0111]

[0112] In the formula, A i and B i are the i-th components of vectors A and B, respectively.

[0113] S104: Based on the redundancy value and the richness value, the two types of sub-questions are optimized to obtain optimized sub-questions, and the optimized sub-questions are input into the pre-trained visual language model to enhance the understanding of complex questions and the attention to local visual information, and improve the accuracy of zero-shot visual question answering.

[0114] Based on the redundancy value and the richness value, the two types of sub-questions are optimized to obtain optimized sub-questions, and the optimized sub-questions are input into the pre-trained visual language model to enhance the understanding of complex questions and the attention to local visual information, and improve the accuracy of zero-shot visual question answering.

[0115] Based on the redundancy value of each sub-question, it is determined whether it exceeds the set redundancy threshold value. If the threshold value is exceeded, the sub-question is determined to be a redundant sub-question and is deleted.

[0116] Based on the richness value of the sub-question after excluding redundancy, it is determined whether it reaches the set richness threshold value. If it does not reach the threshold value, the sub-question is continuously generated, the redundant question is excluded, and the richness value is calculated until the threshold value is reached. The obtained sub-question is used as the optimized sub-question.

[0117] In order to verify the effectiveness of the present application, as shown in the verification example one and the verification example two, the visual effects of the present application method and the prior art method are compared. Figure 4

[0118] In the verification stage, the sample picture one is various food, the question is "What kind of fruit is cut in half and darker than the other?", the question and the image are input into the model of the prior art method, and the wrong answer "Orange" is obtained; the question and the image are input into the new model of the present application method, and the correct answer "Grapes" is obtained. The present application verifies the correct understanding of complex questions and improves the accuracy of visual question answering.

[0119] ​In the verification stage, the sample picture two is the situation on the road, and the question is "What type of animal is on a leash on the sidewalk?", the question and the image are input into the model of the existing method, and an incorrect answer "Horse" is obtained; the question and the image are input into the new model of the method, and a correct answer "Dog" is obtained. It is verified that the method pays attention to important details of the question in the image, and improves the accuracy of visual question answering.

[0120] In the second aspect, as shown in the embodiment two, in order to achieve the above object, the application discloses a zero-shot visual question answering system based on Boolean prompt enhancement, which comprises: Figure 3

[0121] The adaptive judgment module 11 is configured to acquire the original question, input the original question into the pre-trained visual language model, output the original answer, input the original answer into the pre-trained visual language model, and adaptively judge the complexity of the original question according to the confidence of the original answer. If it is a simple question, the original answer is output.

[0122] The sub-question generation module 12 is configured to input the complex question into the pre-trained visual language model if it is a complex question, and output the keywords of the complex question and the corresponding image description. Two types of sub-questions are generated based on the complex question and the keywords and the corresponding image description of the complex question, wherein the two types of sub-questions respectively contain global information and local information.

[0123] The redundancy richness calculation module 13 is configured to calculate the cosine similarity between the two types of sub-questions and the complex question and the cosine similarity between the two types of sub-questions, and calculate the redundancy value and the richness value of the two types of sub-questions based on the cosine similarity between the two types of sub-questions and the complex question and the cosine similarity between the two types of sub-questions.

[0124] The sub-question screening module 14 is configured to optimize the two types of sub-questions based on the redundancy value and the richness value, obtain the optimized sub-questions, input the optimized sub-questions into the pre-trained visual language model, enhance the understanding of the complex question and the attention to the local visual information, and improve the accuracy of the zero-shot visual question answering.

[0125] In combination with the second aspect, in some implementations of the second aspect, the system further comprises: the pre-trained visual language model of the adaptive judgment module is as follows:

[0126] a i =PVLM(p(q i ,v i ),θ)

[0127] ​In the formula, PVLM represents the visual language model network Blip-2 or MiniGPT-4; represents the pre-training weight parameter; p represents the prompt input into the visual language model; q i represents the i th question input into the model; v i represents the i th picture input into the model; a i represents the output corresponding to the prompt p and the i th question and picture; the mechanism for the adaptive judgment module to adaptively judge the complexity of the original question according to the confidence of the original answer is as follows:

[0128] bool i =CE(p d (q i ),a i )

[0129] In the formula, CE represents the visual language model confidence network; p d represents the prompt input into the visual language model to make the model output a boolean type reply; q i represents the i th question input into the model; a i represents the answer to the corresponding question q i ; bool i represents the degree of recognition of the visual language model for the answer a i to the question q i , in the form of “Yes” or “No”;

[0130] If the answer is “Yes”, the original question is judged to be a simple question, and the original answer is output;

[0131] If the answer is “No”, the original question is judged to be a complex question;

[0132] The keyword extraction, image description generation, and two types of sub-question generation processes are as follows:

[0133] The original complex question is input into the pre-trained visual language model, and a keyword extraction prompt is set to make the pre-trained visual language model extract keywords related to the complex question;

[0134] The picture corresponding to the original complex question is input into the pre-trained visual language model, and a picture description generation prompt is set to make the pre-trained visual language model generate a description of the picture;

[0135] The original complex question and the picture corresponding thereto are input into the pre-trained visual language model, and a question asking prompt is set to make the pre-trained visual language model generate a question about the global information of the complex question and the picture corresponding thereto as a first type of sub-question;

[0136] The keyword and the corresponding picture are input into the pre-trained visual language model, a question prompt is set for the keyword, so that the pre-trained visual language model generates a sub-question about local information of the question according to the keyword and the picture content, as the sub-question for extracting local information of the question;

[0137] The image description is input into the pre-trained visual language model, a question prompt is set for the image description, so that the pre-trained visual language model generates a sub-question about local information of the picture according to the image description content, as the sub-question for extracting local information of the picture;

[0138] The sub-question for extracting local information of the question and the sub-question for extracting local information of the picture are merged as the second type of sub-question;

[0139] The calculation process of the redundancy value re of the redundancy richness calculation module is as follows:

[0140]

[0141] In the formula, N1 represents the number of keywords; e i represents the cosine similarity between the ith keyword and the original question; s j represents the cosine similarity between the jth sub-question and the original question;

[0142] The calculation process of the richness value r of the redundancy richness calculation module is as follows:

[0143]

[0144] In the formula, s i represents the cosine similarity between the ith sub-question and the original question; s j represents the cosine similarity between the jth sub-question and the original question; u ij represents the cosine similarity between the ith sub-question and the jth sub-question; n represents the number of sub-questions;

[0145] The calculation process of the cosine similarity is as follows:

[0146]

[0147] In the formula, A i and B i are the ith components of vectors A and B, respectively;

[0148] The process of the sub-question screening module for optimizing the two types of sub-questions based on the redundancy value and the richness value is as follows:

[0149] Based on the redundancy value of each sub-question, it is determined whether it exceeds the set redundancy threshold value. If the threshold value is exceeded, the sub-question is determined to be a redundant sub-question and is deleted;

[0150] Based on the richness value of the sub-problems after excluding the redundancy, it is determined whether the set richness threshold is reached, and if not, the sub-problems are continuously generated, the redundant problems are excluded, and the richness values are calculated until the threshold is reached, and the derived sub-problems are used as the optimized sub-problems.

[0151] Based on the same inventive concept, the present application further provides a computer device, which comprises one or more processors and a memory for storing one or more computer programs; the program comprises program instructions, and the processor is configured to execute the program instructions stored in the memory. The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal and are configured to implement one or more instructions, and are specifically configured to load and execute one or more instructions in the computer storage medium to implement the above method.

[0152] It needs to be further explained that, based on the same inventive concept, the present application further provides a computer storage medium, which stores a computer program, and the computer program is executed by the processor to implement the above method. The storage medium can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of the computer readable storage medium include: electrical connections with one or more conductive wires, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, device or component.

[0153] In the description of the specification, the description of the terms "one embodiment", "an example", "a specific example" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present disclosure. In the specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in an appropriate manner.

[0154] The basic principles, main features and advantages of the present disclosure are shown and described above. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions in the specification are only to illustrate the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, various changes and improvements of the present disclosure can be made, which all fall within the scope of the claimed present disclosure.

Claims

1. A zero-shot visual question answering method based on Boolean prompt enhancement, characterized by: The method comprises the following steps: Obtain the original question, input the original question into the pre-trained visual language model, output the original answer, input the original answer into the pre-trained visual language model, and adaptively judge the complexity of the original question based on the confidence of the original answer. If it is a simple question, output the original answer; If the question is complex, the complex question is input into a pre-trained visual language model, and the output is the keywords of the complex question and the corresponding image description. Based on the complex question and the keywords of the complex question and the corresponding image description, two types of sub-questions are generated, wherein the two types of sub-questions contain global information and local information respectively; Calculate the cosine similarity between the two types of sub-problems and the complex problem, as well as the cosine similarity between the two types of sub-problems. Based on the cosine similarity between the two types of sub-problems and the complex problem, as well as the cosine similarity between the two types of sub-problems, calculate the redundancy value and richness value of the two types of sub-problems. The two types of sub-problems are optimized based on the redundancy value and richness value to obtain the optimized sub-problems. The optimized sub-problems are input into the pre-trained visual language model to enhance the understanding of complex problems and the attention to local visual information, thereby improving the accuracy of zero-shot visual question answering.

2. The zero-shot visual question answering method based on Boolean prompt enhancement according to claim 1, characterized in that: The pre-trained visual language model is as follows: a i =PVLM(p(q i ,v i ),i) Where PVLM stands for the visual language model network Blip-2 or MiniGPT-4; θ represents the pre-trained weight parameter; p represents the prompt input to the visual language model; q i represents the i-th question of the input model; v i Represents the i-th picture of the input model; a i represents the output for the i-th question and image corresponding to the prompt p.

3. The zero-shot visual question answering method based on Boolean prompt enhancement according to claim 1, characterized in that: The mechanism for adaptively judging the complexity of the original question based on the confidence of the original answer is as follows: boolean i =CE(p d (q i ),a i ) Where CE represents the confidence network of the visual language model; p d represents the prompt input to the visual language model that causes the model to output a Boolean response; q i represents the i-th question of the input model; a i Represents the corresponding question q i Answer: bool i Represents the visual language model for question q i The answer is a i The degree of approval, in the form of Yes or No; If the answer is Yes, the original question is judged to be a simple question and the original answer is output; The answer is No, and the original question is judged to be a complex question.

4. The zero-shot visual question answering method based on Boolean prompt enhancement according to claim 1, characterized in that: The process of extracting keywords, generating image descriptions, and generating two types of sub-problems: Input the original complex question into the pre-trained visual language model and set a keyword prompt to extract the question so that the pre-trained visual language model can extract keywords related to the complex question. Input the image corresponding to the original complex question into the pre-trained visual language model, set the prompt to generate image description, and let the pre-trained visual language model generate a description of the image; Input the original complex question and its corresponding image into the pre-trained visual language model, set a question prompt for the question, so that the pre-trained visual language model generates questions about the global information of the question based on the complex question and its corresponding image content as the first type of sub-questions; Input keywords and corresponding images into a pre-trained visual language model, set a question prompt for the keyword, so that the pre-trained visual language model generates sub-questions about local information of the question based on the keywords and image content, as the sub-questions for extracting local information of the question; Input the image description into the pre-trained visual language model and set a question prompt for the image description so that the pre-trained visual language model generates a sub-question about the local information of the image based on the image description content, which is used as the sub-question for extracting the local information of the image; Combine the sub-problem of extracting local information of the problem and the sub-problem of extracting local information of the image as the second type of sub-problem.

5. The zero-shot visual question answering method based on Boolean prompt enhancement according to claim 1, characterized in that: The calculation process of the redundancy value re and the richness value r is: Redundancy value re: Where N1 represents the number of keywords; e i represents the cosine similarity between the ith keyword and the original question; s j Represents the cosine similarity between the jth sub-problem and the original problem; Richness value r: Where s i represents the cosine similarity between the ith sub-problem and the original problem; s j represents the cosine similarity between the jth sub-problem and the original problem; u ij represents the cosine similarity between the i-th subproblem and the j-th subproblem; n represents the number of subproblems; Cosine similarity calculation: Where A i and B i are the i-th components of vectors A and B respectively.

6. The zero-shot visual question answering method based on Boolean prompt enhancement according to claim 1, characterized in that: The process of optimizing the two types of sub-problems based on the redundancy value and the richness value to obtain the optimized sub-problems is as follows: Based on the redundancy value of each sub-problem, determine whether it exceeds the set redundancy threshold. If it exceeds the threshold, the sub-problem is determined to be redundant and deleted; Based on the richness value of the sub-problem after eliminating redundancy, determine whether it reaches the set richness threshold. If not, continue to generate sub-problems, eliminate redundant problems and calculate the richness value until the threshold is reached. Use the obtained sub-problem as the optimized sub-problem.

7. A zero-shot visual question answering system based on Boolean prompt enhancement, which adopts the zero-shot visual question answering method based on Boolean prompt enhancement according to any one of claims 1 to 6, characterized in that: include: The adaptive judgment module is used to obtain the original question, input the original question into the pre-trained visual language model, output the original answer, input the original answer into the pre-trained visual language model, and adaptively judge the complexity of the original question based on the confidence level of the original answer. If the question is simple, the original answer is output; A sub-question generation module is configured to input a complex question into a pre-trained visual language model, output keywords for the complex question and a corresponding image description, and generate two types of sub-questions based on the complex question and the keywords and corresponding image description, wherein the two types of sub-questions contain global information and local information respectively; A redundancy and richness calculation module is used to calculate the cosine similarity between the two types of sub-problems and the complex problem, as well as the cosine similarity between the two types of sub-problems, and calculate the redundancy value and richness value of the two types of sub-problems based on the cosine similarity between the two types of sub-problems and the complex problem, as well as the cosine similarity between the two types of sub-problems; The sub-question screening module is used to optimize two types of sub-questions based on redundancy and richness values ​​to obtain optimized sub-questions. The optimized sub-questions are then input into a pre-trained visual language model to enhance the understanding of complex problems and the focus on local visual information, thereby improving the accuracy of zero-shot visual question answering.

8. The zero-shot visual question answering system based on Boolean prompt enhancement according to claim 7, characterized in that: The visual language model pre-trained by the adaptive judgment module is as follows: a i =PVIM(p(q i ,v i ),θ) Where PVLM stands for the visual language model network Blip-2 or MiniGPT-4; θ represents the pre-trained weight parameter; p represents the prompt input to the visual language model; q i represents the i-th question of the input model; v i Represents the i-th picture of the input model; a i represents the output for the i-th question and image corresponding to the prompt p; The mechanism by which the adaptive judgment module adaptively judges the complexity of the original question based on the confidence of the original answer is as follows: boolean i =CE(p d (q i ),a i ) Where CE represents the confidence network of the visual language model; p d represents the prompt input to the visual language model that causes the model to output a Boolean response; q i represents the i-th question of the input model; a i Represents the corresponding question q i Answer: bool i Represents the visual language model for question q i The answer is a i The degree of approval, in the form of Yes or No; If the answer is Yes, the original question is judged to be a simple question and the original answer is output; The answer is No, and the original question is judged to be a complex question; The process of extracting keywords, generating image descriptions, and generating two types of sub-problems: Input the original complex question into the pre-trained visual language model and set a keyword prompt to extract the question so that the pre-trained visual language model can extract keywords related to the complex question. Input the image corresponding to the original complex question into the pre-trained visual language model, set the prompt to generate image description, and let the pre-trained visual language model generate a description of the image; Input the original complex question and its corresponding image into the pre-trained visual language model, set a question prompt for the question, so that the pre-trained visual language model generates questions about the global information of the question based on the complex question and its corresponding image content as the first type of sub-questions; Input keywords and corresponding images into a pre-trained visual language model, set a question prompt for the keyword, so that the pre-trained visual language model generates sub-questions about local information of the question based on the keywords and image content, as the sub-questions for extracting local information of the question; Input the image description into the pre-trained visual language model and set a question prompt for the image description so that the pre-trained visual language model generates a sub-question about the local information of the image based on the image description content, which is used as the sub-question for extracting the local information of the image; Combine the sub-problem of extracting local information of the problem and the sub-problem of extracting local information of the image as the second type of sub-problem; The calculation process of the redundancy value re of the redundancy richness calculation module is as follows: Where N1 represents the number of keywords; e i represents the cosine similarity between the ith keyword and the original question; s j Represents the cosine similarity between the jth sub-problem and the original problem; The calculation process of the redundancy richness calculation module richness value r: Where s i represents the cosine similarity between the ith sub-problem and the original problem; s j represents the cosine similarity between the jth sub-problem and the original problem; u ij represents the cosine similarity between the i-th subproblem and the j-th subproblem; n represents the number of subproblems; Cosine similarity calculation process: Where A i and B i are the i-th components of vectors A and B respectively; The sub-problem screening module optimizes the two types of sub-problems based on the redundancy value and the richness value, and the process of obtaining the optimized sub-problems is as follows: Based on the redundancy value of each sub-problem, determine whether it exceeds the set redundancy threshold. If it exceeds the threshold, the sub-problem is determined to be redundant and deleted; Based on the richness value of the sub-problem after eliminating redundancy, determine whether it reaches the set richness threshold. If not, continue to generate sub-problems, eliminate redundant problems and calculate the richness value until the threshold is reached. Use the obtained sub-problem as the optimized sub-problem.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, the zero-shot visual question answering method based on Boolean prompt enhancement according to any one of claims 1 to 6 is adopted.

10. A computer-readable storage medium storing a computer program, wherein: When the computer program is loaded and executed by a processor, the zero-shot visual question answering method based on Boolean prompt enhancement according to any one of claims 1 to 6 is adopted.

Images