Zero-sample visual question and answer method and system based on Boolean prompt enhancement

Abstract

The invention discloses a zero-sample visual question answering method and system based on Boolean prompt enhancement, and relates to the technical field of zero-sample visual question answering, and the method comprises the steps: adaptively judging the complexity of an input question based on a pre-training visual language model, outputting an answer to a simple question, and selecting a question with a complex model; based on the selected complex problem, extracting a keyword, and generating a description of an image corresponding to the problem; generating a sub-problem for capturing global information based on the original problem, and generating a sub-problem containing local information from a problem keyword and image description; constructing a redundancy value and a richness value of the sub-problem based on the cosine similarity between the generated sub-problem and the original problem; the sub-questions with high redundancy are deleted by using the redundancy value, the richness of the content of the sub-questions is ensured by using the richness value, the understanding of the model on complex questions and the attention on local visual information are enhanced by using the obtained sub-questions as prompts, and the accuracy of zero-sample visual questions and answers is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of zero-sample visual question answering, and in particular to a zero-sample visual question answering method and system based on Boolean prompt enhancement. Background Art

[0002] With the rapid development of convolutional neural networks, the accuracy of visual question answering has been greatly improved. However, most of the current visual question answering methods are trained based on a large amount of labeled data, which covers specific question types and answer categories. In practical applications, the data that various industries need to process will always have new question types and answer categories that are not included in the training data. This visual question answering model trained based on labeled data can only handle question-answer pairs known in the training phase. When encountering new questions that have never appeared in the training set, the model often performs poorly. This limitation severely restricts the application of visual question answering technology in a wider and more complex real-world environment. To overcome this challenge, researchers began to introduce zero-shot technology into the visual question answering task, allowing the model to try to answer questions that have never been seen before based on existing knowledge and model structure without training for new question types and answer categories, thereby making the visual question answering system more versatile and generalizable.

[0003] Recently, many zero-shot visual question answering methods have tried to use pre-trained visual language models to convert questions into input templates pre-defined by the model to fully utilize the knowledge learned by the pre-trained visual language model to achieve zero-shot visual question answering. However, these methods have two shortcomings: First, when encountering complex questions, existing methods often ignore the text details and associations in the question, hindering the comprehensive understanding of the question; second, existing methods do not pay attention to the local information of the image, resulting in ignoring some details in the image that are important to the question when the image content is particularly complex or requires detailed observation. Therefore, the key to the zero-shot visual question answering task lies in how to deal with complex problems and how to enhance attention to local information in the image. Summary of the invention

[0004] In order to solve the deficiencies mentioned in the above background technology, the purpose of the present invention is to provide a zero-sample visual question answering method and system based on Boolean prompt enhancement.

[0005] In a first aspect, the objective of the present invention can be achieved by the following technical solution: a zero-sample visual question answering method based on Boolean prompt enhancement, the method comprising the following steps:

[0006] Obtain the original question, input the original question into the pre-trained visual language model, output the original answer, input the original answer into the pre-trained visual language model, adaptively judge the complexity of the original question based on the confidence of the original answer, and output the original answer if it is a simple question;

[0007] If it is a complex question, the complex question is input into a pre-trained visual language model, and the keywords of the complex question and the corresponding image description are outputted, and two types of sub-questions are generated based on the complex question and the keywords of the complex question and the corresponding image description, wherein the two types of sub-questions contain global information and local information respectively;

[0008] Calculate the cosine similarity between the two types of sub-problems and the complex problem, and the cosine similarity between the two types of sub-problems. Based on the cosine similarity between the two types of sub-problems and the complex problem, and the cosine similarity between the two types of sub-problems, calculate the redundancy value and richness value of the two types of sub-problems.

[0009] The two types of sub-problems are optimized based on the redundancy value and the richness value to obtain the optimized sub-problems, which are then input into a pre-trained visual language model to enhance the understanding of complex problems and the attention to local visual information, thereby improving the accuracy of zero-sample visual question answering.

[0010] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the pre-trained visual language model is as follows:

[0011] a i =PVLM(p(q i ,v i ),θ)

[0012] Where PVLM represents the visual language model network Blip-2 or MiniGPT-4; θ represents the pre-trained weight parameter; p represents the prompt input to the visual language model; q i represents the i-th question of the input model; v i represents the i-th picture of the input model; a i Represents the output of the i-th question and image corresponding to the prompt p. In combination with the first aspect, in certain implementations of the first aspect, the method further includes: the mechanism of adaptively judging the complexity of the original question based on the confidence of the original answer is as follows:

[0013] bool i =CE(p d (q i ),a i )

[0014] Where CE represents the confidence network of the visual language model; p d represents the prompt input to the visual language model that causes the model to output a Boolean response; q i represents the i-th question of the input model; a i Represents the corresponding question q iAnswer: bool i Represents the visual language model for question q i The answer is a i The degree of approval, in the form of "Yes" or "No";

[0015] The answer is "Yes", the original question is judged to be a simple question, and the original answer is output;

[0016] The answer is "No", and the original question is judged to be a complex question.

[0017] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the keyword extraction, image description generation, and two types of sub-question generation processes:

[0018] Input the original complex question into the pre-trained visual language model, set a prompt for extracting the question keywords, and enable the pre-trained visual language model to extract keywords about the complex question;

[0019] Input the image corresponding to the original complex question into the pre-trained visual language model, set the prompt for generating image description, and make the pre-trained visual language model generate a description of the image;

[0020] Input the original complex question and its corresponding image into the pre-trained visual language model, set a prompt for the question, so that the pre-trained visual language model generates questions about the global information of the question according to the complex question and its corresponding image content as the first type of sub-questions;

[0021] Input keywords and corresponding pictures into the pre-trained visual language model, set a question prompt for the keywords, so that the pre-trained visual language model generates sub-questions about local information of the question according to the keywords and the picture content, as the sub-questions for extracting local information of the question;

[0022] Input the image description into the pre-trained visual language model, set a question prompt for the image description, so that the pre-trained visual language model generates a sub-question about the local information of the image according to the image description content, as the sub-question of extracting the local information of the image;

[0023] The sub-problem of extracting local information of the problem and the sub-problem of extracting local information of the image are combined as the second type of sub-problem.

[0024] In combination with the first aspect, in some implementations of the first aspect, the method further includes: a calculation process of the redundancy value re and the richness value r:

[0025] Redundancy value re:

[0026]

[0027] In the formula, N1 represents the number of keywords; e i represents the cosine similarity between the ith keyword and the original question; s j Represents the cosine similarity between the jth sub-problem and the original problem; the richness value r:

[0028]

[0029] In the formula, s i represents the cosine similarity between the ith sub-problem and the original problem; s j represents the cosine similarity between the jth sub-problem and the original problem; u ij represents the cosine similarity between the ith subproblem and the jth subproblem; n represents the number of subproblems;

[0030] Cosine similarity calculation:

[0031]

[0032] In the formula, A i and B i are the i-th components of vectors A and B respectively.

[0033] In combination with the first aspect, in some implementations of the first aspect, the method further includes: the process of optimizing the two types of sub-problems based on the redundancy value and the richness value to obtain the optimized sub-problems is as follows:

[0034] Based on the redundancy value of each sub-problem, determine whether it exceeds the set redundancy threshold. If it exceeds the threshold, the sub-problem is determined to be a redundant sub-problem and deleted;

[0035] Based on the richness value of the sub-problem after eliminating redundancy, determine whether it reaches the set richness threshold. If not, continue to generate sub-problems, eliminate redundant problems and calculate the richness value until the threshold is reached, and use the obtained sub-problem as the optimized sub-problem.

[0036] In a second aspect, in order to achieve the above-mentioned object, the present invention discloses a zero-sample visual question answering system based on Boolean prompt enhancement, comprising:

[0037] An adaptive judgment module is used to obtain the original question, input the original question into a pre-trained visual language model, output the original answer, input the original answer into the pre-trained visual language model, adaptively judge the complexity of the original question based on the confidence of the original answer, and output the original answer if it is a simple question;

[0038] A sub-question generation module is used for inputting the complex question into a pre-trained visual language model if the question is complex, outputting keywords of the complex question and corresponding image descriptions, and generating two types of sub-questions based on the complex question and the keywords of the complex question and corresponding image descriptions, wherein the two types of sub-questions contain global information and local information respectively;

[0039] A redundancy and richness calculation module is used to calculate the cosine similarity between the two types of sub-problems and the complex problem, and the cosine similarity between the two types of sub-problems, and calculate the redundancy value and richness value of the two types of sub-problems based on the cosine similarity between the two types of sub-problems and the complex problem, and the cosine similarity between the two types of sub-problems;

[0040] The sub-question screening module is used to optimize the two types of sub-questions based on the redundancy value and the richness value to obtain the optimized sub-questions, and input the optimized sub-questions into the pre-trained visual language model to enhance the understanding of complex problems and the attention to local visual information, thereby improving the accuracy of zero-sample visual question answering.

[0041] In conjunction with the second aspect, in some implementations of the second aspect, the system further includes: the visual language model pre-trained by the adaptive judgment module is as follows:

[0042] a i =PVIM(p(q i ,v i ),θ)

[0043] Where PVLM represents the visual language model network Blip-2 or MiniGPT-4; θ represents the pre-trained weight parameter; p represents the prompt input to the visual language model; q i represents the i-th question of the input model; v i represents the i-th picture of the input model; a i Represents the output of the i-th question and image corresponding to the prompt p; the mechanism by which the adaptive judgment module adaptively judges the complexity of the original question based on the confidence of the original answer is as follows:

[0044] bool i =CE(p d (q i ),a i )

[0045] Where CE represents the confidence network of the visual language model; p d represents the prompt input to the visual language model that causes the model to output a Boolean response; q i represents the i-th question of the input model; a i Represents the corresponding question q i Answer: booli Represents the visual language model for question q i The answer is a i The degree of approval, in the form of "Yes" or "No";

[0046] The answer is "Yes", the original question is judged to be a simple question, and the original answer is output;

[0047] The answer is "No", and the original question is considered a complex question;

[0048] The keyword extraction, image description generation and two types of sub-question generation process:

[0049] Input the original complex question into the pre-trained visual language model, set a prompt for extracting the question keywords, and enable the pre-trained visual language model to extract keywords about the complex question;

[0050] Input the image corresponding to the original complex question into the pre-trained visual language model, set the prompt for generating image description, and make the pre-trained visual language model generate a description of the image;

[0051] Input the original complex question and its corresponding image into the pre-trained visual language model, set a prompt for the question, so that the pre-trained visual language model generates questions about the global information of the question according to the complex question and its corresponding image content as the first type of sub-questions;

[0052] Input keywords and corresponding pictures into the pre-trained visual language model, set a question prompt for the keywords, so that the pre-trained visual language model generates sub-questions about local information of the question according to the keywords and the picture content, as the sub-questions for extracting local information of the question;

[0053] Input the image description into the pre-trained visual language model, set a question prompt for the image description, so that the pre-trained visual language model generates a sub-question about the local information of the image according to the image description content, as the sub-question of extracting the local information of the image;

[0054] Combine the sub-problem of extracting local information of the problem and the sub-problem of extracting local information of the image as the second type of sub-problem;

[0055] The calculation process of the redundancy value re of the redundancy richness calculation module is as follows:

[0056]

[0057] In the formula, N1 represents the number of keywords; e i represents the cosine similarity between the ith keyword and the original question; s j Represents the cosine similarity between the jth sub-problem and the original problem;

[0058] The calculation process of the redundancy richness calculation module richness value r is:

[0059]

[0060] In the formula, s i represents the cosine similarity between the ith sub-problem and the original problem; s j represents the cosine similarity between the jth sub-problem and the original problem; u ij represents the cosine similarity between the ith subproblem and the jth subproblem; n represents the number of subproblems;

[0061] Cosine similarity calculation process:

[0062]

[0063] In the formula, A i and B i are the i-th components of vectors A and B respectively;

[0064] The sub-problem screening module optimizes the two types of sub-problems based on the redundancy value and the richness value, and the process of obtaining the optimized sub-problems is as follows:

[0065] Based on the redundancy value of each sub-problem, determine whether it exceeds the set redundancy threshold. If it exceeds the threshold, the sub-problem is determined to be a redundant sub-problem and deleted;

[0066] Based on the richness value of the sub-problem after eliminating redundancy, determine whether it reaches the set richness threshold. If not, continue to generate sub-problems, eliminate redundant problems and calculate the richness value until the threshold is reached, and use the obtained sub-problem as the optimized sub-problem.

[0067] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a terminal device is disclosed, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the memory stores a computer program capable of running on the processor, and when the processor loads and executes the computer program, the zero-sample visual question answering method based on Boolean prompt enhancement as described above is adopted.

[0068] In another aspect of the present invention, in order to achieve the above-mentioned purpose, a computer-readable storage medium is disclosed, in which a computer program is stored. When the computer program is loaded and executed by a processor, the zero-sample visual question answering method based on Boolean prompt enhancement as described above is adopted.

[0069] Beneficial effects of the present invention:

[0070] The present invention can well solve the common problems in existing zero-shot visual question answering methods, namely, ignoring text details and associations in complex questions and ignoring some details in the image that are important to the question, thereby improving the accuracy of zero-shot visual question answering. It can also make the visual question answering system more universal and generalizable. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0072] Figure 1 It is a schematic flow chart of the method of the present invention;

[0073] Figure 2 It is a structural block diagram of the present invention;

[0074] Figure 3 It is a schematic diagram of the system structure of the present invention;

[0075] Figure 4 It is a schematic diagram for comparing the verification effects of the present invention. DETAILED DESCRIPTION

[0076] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] Embodiment 1:

[0078] like Figure 1 As shown, a zero-shot visual question answering method based on Boolean prompt enhancement includes the following steps:

[0079] S101: obtaining an original question, inputting the original question into a pre-trained visual language model, outputting an original answer, inputting the original answer into a pre-trained visual language model, adaptively judging the complexity of the original question based on the confidence of the original answer, and outputting the original answer if it is a simple question;

[0080] The pre-trained visual language models are as follows:

[0081] a i =PVLM(p(q i ,v i ),θ)

[0082] Where PVLM represents the visual language model network Blip-2 or MiniGPT-4; θ represents the pre-trained weight parameter; p represents the prompt input to the visual language model; q i represents the i-th question of the input model; v i represents the i-th picture of the input model; a i Represents the output of the i-th question and image corresponding to the prompt p; the process of adaptively judging the complexity of the original question based on the confidence of the original answer is as follows:

[0083] bool i =CE(p d (q i ),a i )

[0084] Where CE represents the confidence network of the visual language model; p d represents the prompt input to the visual language model that causes the model to output a Boolean response; q i represents the i-th question of the input model; a i Represents the corresponding question q i Answer: bool i Represents the visual language model for question q i The answer is a i The degree of approval, in the form of "Yes" or "No";

[0085] Specifically, for the input question q i , using a pre-trained visual language model (such as BLIP-2 or MiniGPT-4) and a prompt p that enables the pre-trained visual language model to complete the visual question answering task, and let the pre-trained visual language model generate q i The answer is a i ;

[0086] Q i and a i As input, the visual language model confidence network is input, and the pre-trained visual language model is used to determine whether to agree with a i As q i Answer hints d , so that the pre-trained visual language model generates a Boolean answer again i ;

[0087] The bool obtained i If it is "Yes", then q i For simple questions, the model can answer correctly without any subsequent operations, and outputs the answer a i ;

[0088] The bool obtained i If it is "No", then q i For complex questions, it is difficult for models to answer.

[0089] S102: If it is a complex question, the complex question is input into a pre-trained visual language model, and the keywords of the complex question and the corresponding image description are outputted, and two types of sub-questions are generated based on the complex question and the keywords of the complex question and the corresponding image description, wherein the two types of sub-questions contain global information and local information respectively;

[0090] The keyword extraction, image description generation and two types of sub-question generation process:

[0091] Input the original complex question into the pre-trained visual language model, set a prompt for extracting the question keywords, and enable the pre-trained visual language model to extract keywords about the complex question;

[0092] Input the image corresponding to the original complex question into the pre-trained visual language model, set the prompt for generating image description, and make the pre-trained visual language model generate a description of the image;

[0093] Input the original complex question and its corresponding image into the pre-trained visual language model, set a prompt for the question, so that the pre-trained visual language model generates questions about the global information of the question according to the complex question and its corresponding image content as the first type of sub-questions;

[0094] Input keywords and corresponding pictures into the pre-trained visual language model, set a question prompt for the keywords, so that the pre-trained visual language model generates sub-questions about local information of the question according to the keywords and the picture content, as the sub-questions for extracting local information of the question;

[0095] Input the image description into the pre-trained visual language model, set a question prompt for the image description, so that the pre-trained visual language model generates a sub-question about the local information of the image according to the image description content, as the sub-question of extracting the local information of the image;

[0096] Combine the sub-problem of extracting local information of the problem and the sub-problem of extracting local information of the image as the second type of sub-problem;

[0097] Based on the complex problems q i , using the pre-trained visual language model to extract text keyword prompts p k , so that the pre-trained visual language model extracts the question q i Keyword w j , forming q i All keywords of i ;

[0098] Based on complex problems i The corresponding image v i , using the hint p that enables the pre-trained visual language model to generate image descriptions c , so that the pre-trained visual language model generates image v i Description of cap i .

[0099] Based on the extracted keywords i and image description cap i , using the prompt p that enables the pre-trained visual language model to generate questions b , generate a subproblem Θ containing local information of the original problem l ;

[0100] Based on the original question q i and image v i , using the prompt p that enables the pre-trained visual language model to generate questions b , generate a subproblem Θ containing global information of the original problem g , and the subproblem Θ l and θ g Merge to get the original question q i Subproblem Θ of .

[0101] S103: Calculate the cosine similarity between the two types of sub-problems and the complex problem and the cosine similarity between the two types of sub-problems, and calculate the redundancy value and richness value of the two types of sub-problems based on the cosine similarity between the two types of sub-problems and the complex problem and the cosine similarity between the two types of sub-problems;

[0102] Based on the cosine similarity, calculate the similarity between the generated subproblem Θ and the original problem and the similarity between the subproblems to construct the redundancy value;

[0103] The redundancy values ​​are:

[0104]

[0105] In the formula, N1 represents the number of keywords; e i represents the cosine similarity between the ith keyword and the original question; s j Represents the cosine similarity between the jth sub-problem and the original problem.

[0106] Based on cosine similarity, consider the similarity between sub-problems, calculate the difference between the generated sub-problem Θ and the original problem, and construct the richness value;

[0107] The richness values ​​are:

[0108]

[0109] In the formula, s i represents the cosine similarity between the ith sub-problem and the original problem; s j represents the cosine similarity between the jth sub-problem and the original problem; u ij Represents the cosine similarity between the ith subproblem and the jth subproblem.

[0110] The cosine similarity is calculated as:

[0111]

[0112] In the formula, A i and B i are the i-th components of vectors A and B respectively.

[0113] S104: Optimize the two types of sub-problems based on the redundancy value and the richness value to obtain optimized sub-problems, and input the optimized sub-problems into a pre-trained visual language model to enhance the understanding of complex problems and the focus on local visual information, thereby improving the accuracy of zero-sample visual question answering.

[0114] The two types of sub-problems are optimized based on the redundancy value and the richness value. The process of obtaining the optimized sub-problems is as follows:

[0115] Based on the redundancy value of each sub-problem, determine whether it exceeds the set redundancy threshold. If it exceeds the threshold, the sub-problem is determined to be a redundant sub-problem and deleted;

[0116] Based on the richness value of the sub-problem after eliminating redundancy, determine whether it reaches the set richness threshold. If not, continue to generate sub-problems, eliminate redundant problems and calculate the richness value until the threshold is reached, and use the obtained sub-problem as the optimized sub-problem.

[0117] In order to verify the effectiveness of the present invention, Figure 4 As shown, the visualization effects of the method of the present invention and the existing method are compared on verification sample 1 and verification sample 2 respectively.

[0118] In the verification phase, the sample images showed various foods, and the question was "What kind of fruit is cut in half and darker than the other?". The question and image were input into the model of the existing method, and the wrong answer "Orange" was obtained; the question and image were input into the new model of the method of the present invention, and the correct answer "Grapes" was obtained. This verifies the correct understanding of complex problems by the present invention, and improves the accuracy of visual question answering.

[0119] In the verification phase, the second sample image shows a scene on the road, and the question is "What type of animal is on a leash on the sidewalk?". The question and image are input into the model of the existing method, and the wrong answer "Horse" is obtained; the question and image are input into the new model of the method of the present invention, and the correct answer "Dog" is obtained. This verifies that the present invention pays attention to important details of the question in the image, and improves the accuracy of visual question answering.

[0120] Embodiment 2: In the second aspect, as Figure 3 As shown, in order to achieve the above purpose, the present invention discloses a zero-sample visual question answering system based on Boolean prompt enhancement, comprising:

[0121] The adaptive judgment module 11 is used to obtain the original question, input the original question into the pre-trained visual language model, output the original answer, input the original answer into the pre-trained visual language model, adaptively judge the complexity of the original question based on the confidence of the original answer, and output the original answer if it is a simple question;

[0122] The sub-question generation module 12 is used for inputting the complex question into a pre-trained visual language model if the question is complex, outputting keywords of the complex question and corresponding image descriptions, and generating two types of sub-questions based on the complex question and the keywords of the complex question and corresponding image descriptions, wherein the two types of sub-questions contain global information and local information respectively;

[0123] A redundancy and richness calculation module 13 is used to calculate the cosine similarity between the two types of sub-problems and the complex problem and the cosine similarity between the two types of sub-problems, and calculate the redundancy value and richness value of the two types of sub-problems based on the cosine similarity between the two types of sub-problems and the complex problem and the cosine similarity between the two types of sub-problems;

[0124] The sub-question screening module 14 is used to optimize the two types of sub-questions based on the redundancy value and the richness value to obtain optimized sub-questions, and input the optimized sub-questions into a pre-trained visual language model to enhance the understanding of complex problems and the attention to local visual information, thereby improving the accuracy of zero-sample visual question answering.

[0125] In conjunction with the second aspect, in some implementations of the second aspect, the system further includes: the visual language model pre-trained by the adaptive judgment module is as follows:

[0126] a i =PVLM(p(q i ,v i ),θ)

[0127] Where PVLM represents the visual language model network Blip-2 or MiniGPT-4; θ represents the pre-trained weight parameter; p represents the prompt input to the visual language model; q i represents the i-th question of the input model; v i represents the i-th picture of the input model; a i Represents the output of the i-th question and image corresponding to the prompt p; the mechanism by which the adaptive judgment module adaptively judges the complexity of the original question based on the confidence of the original answer is as follows:

[0128] bool i =CE(p d (q i ),a i )

[0129] Where CE represents the confidence network of the visual language model; p d represents the prompt input to the visual language model that causes the model to output a Boolean response; q i represents the i-th question of the input model; a i Represents the corresponding question q i Answer: bool i Represents the visual language model for question q i The answer is a i The degree of approval, in the form of "Yes" or "No";

[0130] The answer is "Yes", the original question is judged to be a simple question, and the original answer is output;

[0131] The answer is "No", and the original question is considered a complex question;

[0132] The keyword extraction, image description generation and two types of sub-question generation process:

[0133] Input the original complex question into the pre-trained visual language model, set a prompt for extracting the question keywords, and enable the pre-trained visual language model to extract keywords about the complex question;

[0134] Input the image corresponding to the original complex question into the pre-trained visual language model, set the prompt for generating image description, and make the pre-trained visual language model generate a description of the image;

[0135] Input the original complex question and its corresponding image into the pre-trained visual language model, set a prompt for the question, so that the pre-trained visual language model generates questions about the global information of the question according to the complex question and its corresponding image content as the first type of sub-questions;

[0136] Input keywords and corresponding pictures into the pre-trained visual language model, set a question prompt for the keywords, so that the pre-trained visual language model generates sub-questions about local information of the question according to the keywords and the picture content, as the sub-questions for extracting local information of the question;

[0137] Input the image description into the pre-trained visual language model, set a question prompt for the image description, so that the pre-trained visual language model generates a sub-question about the local information of the image according to the image description content, as the sub-question of extracting the local information of the image;

[0138] Combine the sub-problem of extracting local information of the problem and the sub-problem of extracting local information of the image as the second type of sub-problem;

[0139] The calculation process of the redundancy value re of the redundancy richness calculation module is as follows:

[0140]

[0141] In the formula, N1 represents the number of keywords; e i represents the cosine similarity between the ith keyword and the original question; s j Represents the cosine similarity between the jth sub-problem and the original problem;

[0142] The calculation process of the redundancy richness calculation module richness value r is:

[0143]

[0144] In the formula, s i represents the cosine similarity between the ith sub-problem and the original problem; s j represents the cosine similarity between the jth sub-problem and the original problem; u ij represents the cosine similarity between the ith subproblem and the jth subproblem; n represents the number of subproblems;

[0145] Cosine similarity calculation process:

[0146]

[0147] In the formula, A i and B i are the i-th components of vectors A and B respectively;

[0148] The sub-problem screening module optimizes the two types of sub-problems based on the redundancy value and the richness value, and the process of obtaining the optimized sub-problems is as follows:

[0149] Based on the redundancy value of each sub-problem, determine whether it exceeds the set redundancy threshold. If it exceeds the threshold, the sub-problem is determined to be a redundant sub-problem and deleted;

[0150] Based on the richness value of the sub-problem after eliminating redundancy, determine whether it reaches the set richness threshold. If not, continue to generate sub-problems, eliminate redundant problems and calculate the richness value until the threshold is reached, and use the obtained sub-problem as the optimized sub-problem.

[0151] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors, and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0152] It needs to be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium, on which a computer program is stored, and the computer program is executed by a processor to execute the above method. The storage medium can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electrical, magnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0153] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present disclosure. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0154] The above shows and describes the basic principles, main features and advantages of the present disclosure. Those skilled in the art should understand that the present disclosure is not limited by the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present disclosure. Without departing from the spirit and scope of the present disclosure, the present disclosure may have various changes and improvements, and these changes and improvements fall within the scope of the present disclosure to be protected.

Claims

1. A zero-shot visual question answering method based on Boolean cue enhancement, characterized in that: The method comprises the following steps: Obtain the original question, input the original question into the pre-trained visual language model, output the original answer, input the original answer into the pre-trained visual language model, adaptively judge the complexity of the original question based on the confidence of the original answer, and output the original answer if it is a simple question; If it is a complex question, the complex question is input into a pre-trained visual language model, and the keywords of the complex question and the corresponding image description are outputted, and two types of sub-questions are generated based on the complex question and the keywords of the complex question and the corresponding image description, wherein the two types of sub-questions contain global information and local information respectively; Calculate the cosine similarity between the two types of sub-problems and the complex problem, and the cosine similarity between the two types of sub-problems. Based on the cosine similarity between the two types of sub-problems and the complex problem, and the cosine similarity between the two types of sub-problems, calculate the redundancy value and richness value of the two types of sub-problems. The two types of sub-problems are optimized based on the redundancy value and the richness value to obtain the optimized sub-problems, which are then input into a pre-trained visual language model to enhance the understanding of complex problems and the attention to local visual information, thereby improving the accuracy of zero-sample visual question answering.

2. The zero-shot visual question answering method based on Boolean prompt enhancement according to claim 1, characterized in that: The pre-trained visual language model is as follows: a i =PVLM(p(q i ,v i ),i) Where PVLM stands for the visual language model network Blip-2 or MiniGPT-4; θ represents the pre-trained weight parameter; p represents the prompt input to the visual language model; q i represents the i-th question of the input model; v i represents the i-th picture of the input model; a i represents the output for the i-th question and image corresponding to the prompt p.

3. The zero-shot visual question answering method based on Boolean prompt enhancement according to claim 1, characterized in that: The mechanism for adaptively judging the complexity of the original question based on the confidence of the original answer is as follows: boolean i =CE(p d (q i ),a i ) Where CE represents the confidence network of the visual language model; p d represents the prompt input to the visual language model that causes the model to output a Boolean response; q i represents the i-th question of the input model; a i Represents the corresponding question q i Answer: bool i Represents the visual language model for question q i The answer is a i The degree of approval, in the form of Yes or No; The answer is Yes, the original question is judged to be a simple question, and the original answer is output; The answer is No, and the original question is judged to be a complex question.

4. The zero-shot visual question answering method based on Boolean prompt enhancement according to claim 1, characterized in that: The process of extracting keywords, generating image descriptions, and generating two types of sub-problems: Input the original complex question into the pre-trained visual language model, set a prompt for extracting the question keywords, and enable the pre-trained visual language model to extract keywords about the complex question; Input the image corresponding to the original complex question into the pre-trained visual language model, set the prompt for generating image description, and make the pre-trained visual language model generate a description of the image; Input the original complex question and its corresponding image into the pre-trained visual language model, set a question prompt for the question, so that the pre-trained visual language model generates questions about the global information of the question according to the complex question and its corresponding image content as the first type of sub-questions; Input keywords and corresponding pictures into the pre-trained visual language model, set a question prompt for the keywords, so that the pre-trained visual language model generates sub-questions about local information of the question according to the keywords and the picture content, as the sub-questions for extracting local information of the question; Input the image description into the pre-trained visual language model, set a question prompt for the image description, so that the pre-trained visual language model generates a sub-question about the local information of the image according to the image description content, as the sub-question of extracting the local information of the image; The sub-problem of extracting local information of the problem and the sub-problem of extracting local information of the image are combined as the second type of sub-problem.

5. The zero-shot visual question answering method based on Boolean prompt enhancement according to claim 1, characterized in that: The calculation process of the redundancy value re and the richness value r is: Redundancy value re: In the formula, N1 represents the number of keywords; e i represents the cosine similarity between the ith keyword and the original question; s j Represents the cosine similarity between the jth sub-problem and the original problem; Richness value r: In the formula, s i represents the cosine similarity between the ith sub-problem and the original problem; s j represents the cosine similarity between the jth sub-problem and the original problem; u ij represents the cosine similarity between the ith subproblem and the jth subproblem; n represents the number of subproblems; Cosine similarity calculation: In the formula, A i and B i are the i-th components of vectors A and B respectively.

6. The zero-shot visual question answering method based on Boolean prompt enhancement according to claim 1, characterized in that: The process of optimizing the two types of sub-problems based on the redundancy value and the richness value to obtain the optimized sub-problems is as follows: Based on the redundancy value of each sub-problem, determine whether it exceeds the set redundancy threshold. If it exceeds the threshold, the sub-problem is determined to be a redundant sub-problem and deleted; Based on the richness value of the sub-problem after eliminating redundancy, determine whether it reaches the set richness threshold. If not, continue to generate sub-problems, eliminate redundant problems and calculate the richness value until the threshold is reached, and use the obtained sub-problem as the optimized sub-problem.

7. A zero-shot visual question answering system based on Boolean prompt enhancement, which adopts the zero-shot visual question answering method based on Boolean prompt enhancement according to any one of claims 1 to 6, characterized in that: include: An adaptive judgment module is used to obtain the original question, input the original question into a pre-trained visual language model, output the original answer, input the original answer into the pre-trained visual language model, adaptively judge the complexity of the original question based on the confidence of the original answer, and output the original answer if it is a simple question; A sub-question generation module is used for inputting the complex question into a pre-trained visual language model if the question is complex, outputting keywords of the complex question and corresponding image descriptions, and generating two types of sub-questions based on the complex question and the keywords of the complex question and corresponding image descriptions, wherein the two types of sub-questions contain global information and local information respectively; A redundancy and richness calculation module is used to calculate the cosine similarity between the two types of sub-problems and the complex problem, and the cosine similarity between the two types of sub-problems, and calculate the redundancy value and richness value of the two types of sub-problems based on the cosine similarity between the two types of sub-problems and the complex problem, and the cosine similarity between the two types of sub-problems; The sub-question screening module is used to optimize the two types of sub-questions based on the redundancy value and the richness value to obtain the optimized sub-questions, and input the optimized sub-questions into the pre-trained visual language model to enhance the understanding of complex problems and the attention to local visual information, thereby improving the accuracy of zero-sample visual question answering.

8. The zero-shot visual question answering system based on Boolean prompt enhancement according to claim 7, characterized in that: The visual language model pre-trained by the adaptive judgment module is as follows: a i =PVIM(p(q i ,v i ),θ) Where PVLM stands for the visual language model network Blip-2 or MiniGPT-4; θ represents the pre-trained weight parameter; p represents the prompt input to the visual language model; q i represents the i-th question of the input model; v i represents the i-th picture of the input model; a i represents the output for the i-th question and image corresponding to the prompt p; The mechanism by which the adaptive judgment module adaptively judges the complexity of the original question based on the confidence of the original answer is as follows: boolean i =CE(p d (q i ),a i ) Where CE represents the confidence network of the visual language model; p d represents the prompt input to the visual language model that causes the model to output a Boolean response; q i represents the i-th question of the input model; a i Represents the corresponding question q i Answer: bool i Represents the visual language model for question q i The answer is a i The degree of approval, in the form of Yes or No; The answer is Yes, the original question is judged to be a simple question, and the original answer is output; The answer is No, and the original question is considered a complex question; The process of extracting keywords, generating image descriptions, and generating two types of sub-problems: Input the original complex question into the pre-trained visual language model, set a prompt for extracting the question keywords, and enable the pre-trained visual language model to extract keywords about the complex question; Input the image corresponding to the original complex question into the pre-trained visual language model, set the prompt for generating image description, and make the pre-trained visual language model generate a description of the image; Input the original complex question and its corresponding image into the pre-trained visual language model, set a question prompt for the question, so that the pre-trained visual language model generates questions about the global information of the question according to the complex question and its corresponding image content as the first type of sub-questions; Input keywords and corresponding pictures into the pre-trained visual language model, set a question prompt for the keywords, so that the pre-trained visual language model generates sub-questions about local information of the question according to the keywords and the picture content, as the sub-questions for extracting local information of the question; Input the image description into the pre-trained visual language model, set a question prompt for the image description, so that the pre-trained visual language model generates a sub-question about the local information of the image according to the image description content, as the sub-question of extracting the local information of the image; Combine the sub-problem of extracting local information of the problem and the sub-problem of extracting local information of the image as the second type of sub-problem; The calculation process of the redundancy value re of the redundancy richness calculation module is as follows: In the formula, N1 represents the number of keywords; e i represents the cosine similarity between the ith keyword and the original question; s j Represents the cosine similarity between the jth sub-problem and the original problem; The calculation process of the redundancy richness calculation module richness value r is: In the formula, s i represents the cosine similarity between the ith sub-problem and the original problem; s j represents the cosine similarity between the jth sub-problem and the original problem; u ij represents the cosine similarity between the ith subproblem and the jth subproblem; n represents the number of subproblems; Cosine similarity calculation process: In the formula, A i and B i are the i-th components of vectors A and B respectively; The sub-problem screening module optimizes the two types of sub-problems based on the redundancy value and the richness value, and the process of obtaining the optimized sub-problems is as follows: Based on the redundancy value of each sub-problem, determine whether it exceeds the set redundancy threshold. If it exceeds the threshold, the sub-problem is determined to be a redundant sub-problem and deleted; Based on the richness value of the sub-problem after eliminating redundancy, determine whether it reaches the set richness threshold. If not, continue to generate sub-problems, eliminate redundant problems and calculate the richness value until the threshold is reached, and use the obtained sub-problem as the optimized sub-problem.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that: The memory stores a computer program that can be run on the processor. When the processor loads and executes the computer program, the zero-sample visual question answering method based on Boolean prompt enhancement according to any one of claims 1 to 6 is adopted.

10. A computer-readable storage medium having a computer program stored therein, characterized in that: When the computer program is loaded and executed by a processor, the zero-shot visual question answering method based on Boolean prompt enhancement according to any one of claims 1 to 6 is adopted.

Images