Knowledge-based visual question answering method and device based on size model cooperation, and medium

Abstract

The knowledge type visual question answering method and device based on size model cooperation, and medium belong to the technical field of visual question answering, solve the problem of how to improve the accuracy of visual question answering technology facing complex problems;The image description extraction step generates an image description and a key object label set highly associated with natural language questions, the entity enhancement processing step extracts key entities and generates clarification questions and entity examples for resolving semantic ambiguity, the candidate answer generation step generates several semantically complementary answers under different reasoning dimensions based on a multi-round generation strategy, the model built-in scoring mechanism is used to screen and optimize the candidate answers, and the context example retrieval step retrieves the most relevant examples from the current input;Finally, the large language model receives the unified input prompt formed by splicing the output results of the above steps, and generates answers in a self-recurrent manner, effectively improving the accuracy of visual question answering technology facing complex problems.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of visual question answering, and relates to a knowledge-based visual question answering method based on size model cooperation, equipment and medium. BACKGROUND

[0002] Visual question answering is one of the core tasks in the field of multi-modal artificial intelligence, aiming to enable machines to understand image content and answer related natural language questions. The prior art such as the invention patent CN119357430A discloses a visual question answering processing method, which can determine the problem appeal of the user to the product through analysis of the dialogue information, and then according to the problem appeal and the knowledge graph of the product, a series of frames most relevant to the problem appeal can be extracted from a long product problem video, and then a short video with a shorter time length is generated using the above frames, and the short video is displayed synchronously with the problem appeal. However, with the development of deep learning, the existing visual question answering method has the following defects when dealing with complex problems:

[0003] (1) Insufficient alignment of vision and semantics: the image description generated by the existing method is usually a general description, lacking specific description for specific problems, resulting in missing of key information or interference of redundant information, affecting the precise understanding and reasoning ability of the model to the problem.

[0004] (2) Fuzzy entity semantics: for complex problems involving fine-grained or specific domain entities, existing models have difficulty in distinguishing entities with similar appearance or similar semantics, resulting in ambiguous or inaccurate answers, reducing the reliability of the model.

[0005] (3) Insufficient context support: existing methods mostly ignore the example-driven reasoning mechanism, relying only on pure text context retrieval, ignoring the multi-modal characteristics of the visual question answering task, making it difficult for the model to effectively utilize prior knowledge when facing complex problems requiring multi-step reasoning, resulting in limited generalization ability.

[0006] Therefore, it is particularly important to propose a visual question answering solution that can effectively integrate visual and semantic information, resolve entity ambiguity, and provide rich multi-modal context support to improve the accuracy and robustness of visual question answering technology in the face of complex problems. SUMMARY

[0007] The technical problem to be solved by the application is how to improve the accuracy of visual question answering technology in the face of complex problems.

[0008] The application solves the above technical problems by the following technical solutions:

[0009] The knowledge-based visual question answering method based on size model cooperation comprises the following steps:

[0010] S1, receiving a user inputted image to be analyzed and a natural language question related to the image content, and performing standardization processing;

[0011] S2, fusing the image and the question after standardization processing to generate an image description associated with the question, identifying key objects in the image to generate a key object label set, and outputting a complete image description;

[0012] S3, extracting key entities in the natural language question, constructing a clarification question in combination with the key object label set, and generating entity examples based on a pre-trained visual-linguistic model with the clarification question and the image after standardization processing as inputs;

[0013] S4, generating a set of original candidate answers based on a multi-round generation strategy, and comprehensively evaluating the set of original candidate answers using a model built-in scoring mechanism to optimize the set of candidate answers;

[0014] S5, extracting visual features and text features of the image and the question based on a pre-trained multi-modal encoder, calculating comprehensive similarity scores of each sample in the training data set, and filtering out the most relevant image-question pairs according to the similarity to form a set of context examples;

[0015] S6, based on the output results of steps S2-S5, constructing a unified input prompt for the large language model and passing it to the large language model to generate an answer.

[0016] Further, the S2 comprises the following steps:

[0017] S21, based on the image to be analyzed, a pre-trained visual model is used to generate visual features highly consistent with the natural language question, specifically:

[0018] First, the image is divided into multiple regional features, and the question is parsed into corresponding semantic vectors;

[0019] Then, the importance of each region is dynamically adjusted based on semantic relevance, specifically:

[0020] The regional features of the image and the natural language question are respectively encoded into comparable semantic representations, and then the degree of association between each regional feature and the question semantics is calculated, and the regional features are sorted and weighted according to the degree of association;

[0021] S22, taking the visual features after semantic reinforcement processing as input, using a language generation model to output a question-oriented image description, the question-oriented image description specifically being an image description most relevant to the question semantics, including key objects, attributes and scene information;

[0022] S23, a set of key object labels is extracted by a visual-semantic alignment method, a complete image description is output by fusing the problem-oriented image description and the set of key object labels, and specifically,

[0023] First, an object detection model is used to process the image to obtain a set of candidate objects, including object categories, positions, and confidence levels.

[0024] Next, after obtaining the problem-oriented semantic attention region, Grad-CAM is used to visualize the internal feature response of the visual model to highlight the image region most relevant to the current problem as the semantic attention region.

[0025] Then, the semantic consistency of the semantic attention region and the candidate object is determined to determine whether the candidate object is a key object. Specifically, when the semantic relevance between the category, position, and attribute of the candidate object and the semantic attention region is not less than the corresponding threshold, respectively, the candidate object is determined to be a key object. Otherwise, the candidate object is determined to be a non-key object or a background object.

[0026] Finally, a set of key object labels is extracted, including object names and core attributes, to truly reflect the image entities required by the problem. The complete image description is output by fusing the problem-oriented image description and the set of key object labels. , the following logic is used:

[0027]

[0028] wherein, represents the image, represents the initial problem, represents the problem-oriented image description, and represents the set of key object labels.

[0029] Further, the semantic relevance between the category, position, and attribute of the candidate object and the semantic attention region is not less than the corresponding threshold, respectively, specifically including the following contents:

[0030] S231, the semantic similarity between the category semantic of the candidate object and the visual semantic represented by the semantic attention region in the cross-modal embedding space is not less than a first threshold;

[0031] S232, the spatial overlap between the position of the candidate object and the semantic attention region is not less than a second threshold;

[0032] S233, the consistency score of the core attribute semantic of the candidate object and the high response attribute feature corresponding to the semantic attention region is not less than a third threshold.

[0033] Further, the S3 includes the following steps:

[0034] S31, performing semantic analysis on the cleaned natural language question by using a pre-trained language model, and extracting key entities in the question, including but not limited to objects, attributes, scene components, and obtaining a key entity set , wherein, represents the mth core semantic unit that may affect the inference result;

[0035] S32, constructing a clarification question based on the key entity set extracted in S31 and the key object label set extracted in S23, specifically, inputting the original natural language question and the recognized key entities into a pre-trained language model in a fixed format, and the model automatically generates an additional question directly related to the key entities as a clarification question according to the above prompt;

[0036] S33, inputting the clarification question and the input image into a pre-trained visual-language model OFA to generate an entity example with clear semantic direction, including the classification, key attributes, functions or scene association information of the entity; Specifically, the entity example is represented by the following logic :

[0037]

[0038] , wherein, represents the clarification question, and OFA represents a general multi-modal pre-trained visual-language model.

[0039] Further, the S4 includes the following steps:

[0040] S41, using a pre-trained visual-language model OFA to input the input image I and the natural language question Q, and generating a multi-dimensional original candidate answer set based on a multi-round generation strategy;

[0041] S42, using a built-in scoring mechanism of the model to comprehensively evaluate the original candidate answer set to optimize the candidate answer set .

[0042] Further, the S42 specifically includes the following content:

[0043] First, calculate the semantic matching score, specifically, input the natural language question Q and the mth candidate answer into the same pre-trained language encoder to obtain the corresponding semantic vector representations and , calculate the semantic correlation degree of the two by using a similarity function, and then map the semantic correlation degree to the interval [0, 1] through linear normalization to obtain the semantic matching score :

[0044] ​

[0045] wherein, denotes a linear normalization operation on the semantic relevance degree output by the similarity function;

[0046] Secondly, the image consistency score is calculated, specifically, the input image I and the first candidate answer are jointly input into the pre-trained visual-language model, the image-text matching score output by the model is normalized to obtain the image consistency score :

[0047]

[0048] wherein, denotes a linear normalization operation on the image-text matching score output by the pre-trained visual-language model;

[0049] Then, the language quality score is calculated, specifically, only the candidate answer is taken as the input text, the average word token probability calculated by the pre-trained language model is taken as the fluency index, and the language quality score reflecting the syntactic structure and readability is obtained by normalizing it to the interval [0, 1] :

[0050]

[0051] wherein, denotes a linear normalization operation on the fluency index;

[0052] Finally, the above three scores are linearly combined according to the preset weight to form the comprehensive score of the candidate answer :

[0053]

[0054] wherein, , , respectively denote non-negative weight coefficients, and satisfy ;

[0055] The score threshold is set, only the answers with scores exceeding the score threshold are retained, and the retained answers are uniformly formatted to form the candidate answer set ; wherein, denotes the retained candidate answer.

[0056] Further, the S5 comprises the following steps:

[0057] S51, using a pre-trained multi-modal encoder to extract visual features of the input image , text features of the input natural language question , and text features of the candidate questions sampled from the pre-set training data set , using the following logic representation:

[0058]

[0059] wherein, and represent visual and text encoders, respectively;

[0060] S52, for each sample of the input and the training data set, respectively calculating visual similarity and text similarity , calculating a comprehensive similarity score , using the following logic representation:

[0061]

[0062] wherein, and represent weight coefficients of visual similarity and text similarity, respectively;

[0063] S53, according to the comprehensive similarity score , sorting the samples of the training data set, selecting the top-K most relevant samples as context examples, converting the context examples into a structured format suitable for large language model input, and summarizing them into a context example set, using the following logic representation:

[0064]

[0065] wherein, represents the context example after formatting processing, is the concise text description of the i-th example image, is the i-th example question, is the answer corresponding to the i-th example.

[0066] Further, the S6 comprises the following steps:

[0067] S61, based on the output results of steps S2-S5, forming a unified input prompt P suitable for the large language model according to a fixed information structure; specifically, the output result of step S2 is a complete image description , the output result of step S3 is an entity example , the output result of step S4 is a candidate answer set , and the output result of step S5 is a context example set ;

[0068] S62, with unified input prompt P as input, the large language model generates answers word by word by autoregressive method, and the prediction rule uses the following logic expression:

[0069]

[0070] Wherein, is the word unit generated at step t, is the word unit generated before step t, is the instruction prompt.

[0071] An electronic device comprising a memory for storing a program supporting the processor to execute the above-mentioned knowledge-based visual question answering method based on the size model cooperation, and a processor configured to execute the program stored in the memory.

[0072] A storage medium, a computer program is stored on the storage medium, when the computer program is run by the processor, the steps of the above-mentioned knowledge-based visual question answering method based on the size model cooperation are executed.

[0073] The advantages of the present application are:

[0074] The present application proposes a knowledge-based visual question answering method based on the cooperation of the size model, generates an image description and a key object label set highly related to the natural language question in the image description extraction step, significantly enhances the alignment effect of visual information and semantic information, solves the problem of insufficient alignment between visual content and question semantics in the existing method, and guides the large language model to focus more accurately on the area related to the question in the image, thereby significantly improving the reasoning performance;

[0075] In the entity enhancement processing step, the key entity is extracted and the clarifying question and entity example for resolving semantic ambiguity are generated, which can effectively supplement the context knowledge and reduce the semantic ambiguity at the entity level, so that the large language model obtains more stable and accurate entity representation;

[0076] In the candidate answer generation step, a plurality of semantically complementary answers are generated under different reasoning dimensions based on the multi-round generation strategy, a multi-dimensional original candidate answer set is constructed, and the optimized candidate answer set is further selected as the reference answer space for subsequent large language model reasoning based on the built-in scoring mechanism of the model, which can significantly reduce the reasoning range and improve the output stability;

[0077] In the context example retrieval step, the most relevant examples are retrieved with the current input, which provides rich multi-modal context information for the model, not only enhances the adaptability of the model to new problems, but also significantly improves its generalization performance, so that it can more stably generate high-quality answers when facing complex reasoning tasks.

[0078] Finally, the large language model receives the unified input prompt formed by splicing the output results of the above steps to infer and generate answers in a self-recurrent manner, and the application constructs a large-small model collaborative framework based on a pre-trained visual-language model and a large language model, realizes accurate understanding of image content and effective reasoning based on external knowledge by combining visual information and natural language processing, thereby generating accurate answers and completing the whole process of question and answer reasoning from input to output, effectively improving the accuracy of complex problems faced by visual question and answer technology, and having better robustness and generalization ability in diversified and knowledge-intensive visual question and answer scenes, and having good application prospects. BRIEF DESCRIPTION OF DRAWINGS

[0079] Figure 1 is a flowchart of a knowledge-based visual question and answer method based on large-small model collaboration of the embodiment one of the application;

[0080] Figure 2 is a performance comparison diagram of the embodiment one of the application and the prior art in terms of knowledge resources and accuracy based on OK-VQA data set;

[0081] Figure 3 is a performance comparison diagram of the embodiment one of the application and the prior art in terms of knowledge resources and accuracy based on A-OKVQA data set;

[0082] Figure 4 is a comparison diagram of the influence of each step of the embodiment one of the application on OK-VQA and A-OKVQA data sets. DETAILED DESCRIPTION

[0083] To make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely below in conjunction with the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, rather than all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the application.

[0084] The technical scheme of the application will be further described below in conjunction with the drawings in the specification and specific embodiments:

[0085] Embodiment one

[0086] The application uses a pre-trained visual-language model as a small model to process subdivided tasks and optimizes the inference ability of a large language model to form a large-small model collaborative framework, generates accurate answers for user input, and completes the whole process of question and answer reasoning from input to output. Figure 1As shown, specifically, the application discloses a knowledge-based visual question answering method based on size model cooperation, comprising the following steps:

[0087] S1, receiving input and preprocessing: receiving the image to be analyzed input by the user and the natural language question related to the image content, and performing standardization processing.

[0088] In this embodiment, the system first receives the image to be analyzed input by the user, the image format is RGB channel format, and the resolution is uniformly adjusted to 224*224 pixels; at the same time, the system receives the natural language question related to the image content, the natural language question is used as a knowledge-based question, which needs to be combined with the image content and external knowledge for reasoning in the future, and the knowledge-based question in this embodiment is, for example, "what kind of school does the building style in the picture belong to", "who is the author of this painting" and the like.

[0089] In this embodiment, the standardization processing includes the following contents:

[0090] The input image to be analyzed is subjected to format normalization processing, specifically, removing image noise and unifying image size to standard resolution, so as to ensure the consistency and high quality of image data;

[0091] The input natural language question is subjected to text cleaning, specifically, filtering special characters, unifying text case and correcting grammatical errors, so as to ensure that the subsequent module can stably and efficiently process the input information.

[0092] S2, image information extraction: fusing the image after standardization processing and the question to generate image description of the associated question, identifying the key objects in the image to generate a key object label set, and outputting complete image description. The S2 specifically includes the following contents:

[0093] S21, based on the image to be analyzed, a pre-trained visual model is used to generate visual features highly consistent with the natural language question, specifically:

[0094] Firstly, the image is divided into multiple regional features, and the question is analyzed into a corresponding semantic vector.

[0095] Subsequently, the importance of each region is dynamically adjusted based on the semantic correlation, specifically:

[0096] The region features of the image and the natural language question are respectively encoded into comparable semantic representations, the degree of association between each region feature and the question semantics is calculated, the region features are sorted and weighted according to the degree of association, if the degree of association between the question semantics and the region features is high, the region highly related to the question is focused on by assigning a high weight value, and otherwise the irrelevant region is weakened; wherein the degree of association can be any one of semantic similarity, attention score or cross-modal matching score.

[0097] In the embodiment, compared with directly extracting the overall description of the image or based on a fixed attention mode in the traditional visual question answering, the application can automatically filter the regions irrelevant to the question or with low degree of association by introducing a screening mechanism based on semantic correlation, thereby avoiding semantic deviation caused by dispersion of visual information and enabling visual attention to automatically focus on the image part most relevant to the meaning of the question. For example, when the question asks about the "architectural style", the model can preferentially focus on the architectural details in the image while ignoring the background information irrelevant to the question.

[0098] The process ensures that the finally generated visual features have clear question orientation and can effectively reflect the key visual content pointed to by the question, so that the visual representation obtained based on this mechanism is more focused and more interpretable in content, not only improves the alignment accuracy of vision and language, but also provides more reliable basic information for subsequent image description generation and object label extraction, so that the entire question and answer link can establish stable semantic association in the initial stage and improve the understanding ability of the system for complex knowledge-type questions.

[0099] In the embodiment, the pre-trained visual model uses a pre-trained multi-modal encoder (BLIP) as an example that can be implemented by the application, which has the ability to adapt to the visual question answering task.

[0100] S22, using the visual features processed by the semantic reinforcement as input, a language generation model is used to output a question-oriented image description, the question-oriented image description is specifically an image description most relevant to the question semantics, including key object, attribute and scene information.

[0101] In the embodiment, the language generation model uses a LLaMA2(13b) large language model, which receives the question semantics and the visual features simultaneously during the generation process, so that the decoder of the language generation model with conditional control ability can preferentially refer to the image content most relevant to the question in word selection and sentence structure.

[0102] In this embodiment, compared with the traditional visual question answering which only focuses on the global scene and does not distinguish the problem requirements, the application is subject to the problem semantic constraints when generating image description, and the decoder can only extract objects, attributes and scene information from the visual area with high semantic correlation, thereby avoiding generating redundant background or irrelevant content to the problem, and making the generated result more in line with the problem requirements.

[0103] The process makes the output image description naturally revolve around the problem in content, and does not output image description irrelevant to the task, nor generalizes the image description, for example, automatically emphasizes the character attributes when asking for character information, focuses on the relevant structure when asking for building features, and outputs the description as "There is a person in the picture wearing ancient costumes, and the background is a landscape" for the question "Who is the author of this painting?", instead of the generalized description "a picture". The final image description has clear problem orientation and high information density, providing more accurate input basis for subsequent small model function modules and large model reasoning stage.

[0104] S23, a set of key object labels is extracted by a visual-semantic alignment method, and a complete image description is output by fusing the problem-oriented image description and the set of key object labels, which specifically includes the following contents:

[0105] Firstly, an object detection model is used to process the image to obtain a candidate object set, including object category, position and attribute; wherein the object detection model can be any one of Faster R-CNN, Mask R-CNN or RetinaNet.

[0106] Then, after obtaining the problem-oriented semantic attention area, Grad-CAM is used to visualize the internal feature response of the visual model to highlight the image area most relevant to the current problem as the semantic attention area. Grad-CAM technology will generate a note heat map reflecting the relevance of the problem according to the gradient information generated by the model when processing the image and the question, so that the area that the model really pays attention to in the image is presented in the form of obvious highlights. Through threshold screening and region segmentation of the note heat map, multiple semantic attention areas can be obtained, which are used to represent the visual clues most relied on by the model when answering the current question.

[0107] Then, the semantic attention area and the candidate object are determined for semantic consistency to determine whether the candidate object is a key object. Specifically, according to the high response visual semantic features highlighted by the semantic attention area, the category semantics and attribute semantics of the candidate object are matched and analyzed: when the semantic correlation between the category, position and attribute of the candidate object and the semantic attention area is all not lower than the corresponding threshold, the candidate object is determined as a key object; otherwise, the candidate object is determined as a non-key object or a background object.

[0108] Further, the semantic relevance between the category, position and attribute of the candidate object and the semantic attention region is not less than a corresponding threshold, specifically including the following contents:

[0109] S231, the semantic similarity between the category semantic of the candidate object and the visual semantic represented by the semantic attention region in the cross-modal embedding space is not less than a first threshold; wherein the semantic similarity can be obtained by using any one of cosine similarity, dot product similarity or matching probability function;

[0110] S232, the spatial overlap degree between the position of the candidate object and the semantic attention region is not less than a second threshold;

[0111] S233, the consistency score between the core attribute semantic of the candidate object and the high response attribute feature corresponding to the semantic attention region is not less than a third threshold.

[0112] Finally, a key object label set is extracted, including object name and core attribute, to truly reflect the image entities required by the question; the question-oriented image description and the key object label set are fused to output a complete image description , which is represented by the following logic:

[0113]

[0114] wherein, represents the image, represents the initial question, represents the question-oriented image description, and represents the key object label set, for example, “person (wearing ancient costume)” and “landscape (background)” are identified to form ={ (name + attribute), (name + attribute),…, (name + attribute)} in a standardized format, so it can be seen that the image information output by step S2 still maintains the original expression form.

[0115] The embodiment not only considers the explicit recognition result of the traditional detector, but also introduces the semantic attention region as a screening basis, so that the key object label set and the question semantic remain consistency, thereby improving the overall visual-semantic alignment quality. The embodiment can effectively screen out the key objects that are truly related to the question semantic, and the names and core attributes of these objects are included in the key object label set, while the irrelevant or lowly associated objects are excluded. The final key object set can truly reflect the entities in the image that are most closely related to the question semantic, and provide accurate basic information for subsequent functional modules.

[0116] S3, entity enhancement processing: extracting key entities in the natural language question, constructing a clarification question in combination with the key object label set; taking the clarification question and the standardized processed image as input, generating an entity example based on the pre-trained visual-linguistic model. The S3 specifically includes the following contents:

[0117] S31, using a pre-trained language model to perform semantic analysis on the natural language question after text cleaning, extracting key entities in the question, including but not limited to objects, attributes, scene components, and obtaining a key entity set wherein, represents the mth core semantic unit that may affect the inference result. For example, from the question "What is the architectural style in the picture?", the key entities "building", "style" and "genre" are extracted.

[0118] In this embodiment, the pre-trained language model can be a visual-linguistic model OFA. Unlike the traditional keyword matching method, this embodiment determines the semantic role of the entity in the sentence through context dependency, thereby explicitly determining the inquiry object and the modification content of the question at the structural level, so that the subsequent disambiguation function can focus on the semantic core that really affects the inference.

[0119] S32, based on the key entity set extracted in S31 and the key object label set extracted in S23, a clarification question is constructed to refine the entity semantics and resolve ambiguity, avoiding model misjudgment.

[0120] In this embodiment, the process of constructing a clarification question is as follows:

[0121] Through the prompt (Prompt) method, the visual-linguistic model OFA is guided to automatically generate a clarification question for refining the entity semantics from the original question itself. Specifically, first, the original natural language question and the identified key entities are input to the pre-trained language model in a fixed format, such as: "Please generate a clarification question for further refining the meaning of the key entity based on the following question: Original question: ××; Key entity: ××". The model automatically generates an additional question related to the key entity and capable of helping the model further distinguish semantics as a clarification question based on the above prompt. Through the above lightweight clarification question generation method based on the prompt, a clarification question closely related to the context of the current natural language question can be quickly obtained without relying on complex rules or attribute analysis logic.

[0122] For example, when the entity is "architectural style", the system will prioritize the appearance structure, roof shape, color characteristics, etc. as the basis for asking questions, thereby constructing clarification questions such as "Is the roof structure of the building pointed or flat?" and "Does the main color of the facade belong to warm or cool color?" When the entity is "person's costume", the system generates questions such as "Is the costume pattern a traditional element or a modern pattern" based on texture, pattern or color, further refining the semantic characteristics of the entity and providing more accurate semantic information. The clarification question generation mechanism proposed in this embodiment directly corresponds the clarification question to the real observable features of the image, thereby forming a semantic guidance question with clear direction, significantly improving the technical effect of ambiguity resolution.

[0123] S33, input the clarification question and the input image into the pre-trained visual-linguistic model OFA to generate an entity example with clear semantic direction, including the classification, key attributes, functions or scene association information of the entity; specifically, the entity example individual is represented by the following logic :

[0124]

[0125] wherein, represents the clarification question, and OFA (One-For-All) represents a general multi-modal pre-trained visual-linguistic model. The example generation process guides the model to extract representative description information from the image content according to the attribute dimension pointed by the clarification question, so that the generated result can naturally reflect the differences between entity categories.

[0126] For example, for the entity "European architecture", the system can generate examples such as "with pointed roof, brick wall structure, commonly found in traditional European streets"; for "traditional costume", examples such as "high-saturation embroidery patterns, commonly used in national costumes for festivals" can be generated. Through the generation of multi-dimensional examples, the present application can effectively supplement the context knowledge at the entity level and reduce semantic ambiguity, so that the model obtains a more stable and accurate entity representation, laying a clear semantic foundation for the subsequent candidate answer generation and reasoning stage.

[0127] As Figure 4As shown, when the knowledge-based visual question answering lacks the entity enhancement module processing, the visual question answering accuracy performance on the OK-VQA dataset decreases by 0.89% after removing the entity enhancement processing, and decreases by 0.84% on the A-OKVQA dataset, where A-OKVQA and OK-VQA are important datasets in the field of visual question answering (VQA) proposed in 2019 and 2022 respectively. Although the decrease is small, this result shows that in the scenario where multiple semantically similar objects appear in the image, the reliability of reasoning can be significantly enhanced by constructing a clarification question to eliminate entity ambiguity, further confirming the important role of the entity enhancement module in improving the reasoning accuracy and robustness of the model.

[0128] S4, candidate answer generation processing: generating an original candidate answer set based on a multi-round generation strategy, and comprehensively evaluating the original candidate answer set by using a model built-in scoring mechanism to optimize the candidate answer set The S4 specifically includes the following contents:

[0129] S41, using a pre-trained visual-linguistic model OFA, taking the input image I and the natural language question Q as inputs, generating a multi-dimensional original candidate answer set based on a multi-round generation strategy.

[0130] In this embodiment, the input image I and the natural language question Q are jointly taken as the input of the model decoder, and the decoding process is divided into several rounds, each round generates a candidate answer with different decoding parameters, to ensure that the same question can generate several semantically complementary answers under different reasoning dimensions, and to construct a multi-dimensional original candidate answer set.

[0131] Specifically, the decoding process includes an attribute round, a causal round and a common sense round. The attribute round generates a candidate answer by focusing on the visible attributes in the image according to the attribute judgment prompt, the causal round generates a candidate answer by explaining the short sentence according to the causal reasoning prompt, and the common sense round generates a candidate answer by combining the image content with the common sense library information according to the knowledge retrieval prompt; the decoding parameters include temperature, top-k and repetition penalty value.

[0132] In this embodiment, the outputs of all rounds are collected into the original candidate answer set, which is represented by the following logic:

[0133]

[0134] Wherein, represents the original candidate answer set.

[0135] S42, using a model built-in scoring mechanism to comprehensively evaluate the original candidate answer set, and optimizing the candidate answer set , specifically including the following contents:

[0136] First, a semantic matching score is calculated, specifically, the natural language question Q and the i-th candidate answer A are input into the same pre-trained language encoder to obtain corresponding semantic vector representations and respectively, the semantic relevance degree of the two is calculated using a similarity function, and then the semantic relevance degree is mapped to the interval [0, 1] through linear normalization to obtain the semantic matching score:

[0137]

[0138] , wherein represents the linear normalization operation on the semantic relevance degree output by the similarity function. In the embodiment, the similarity function can be cosine similarity.

[0139] Second, an image consistency score is calculated, specifically, the input image I and the i-th candidate answer A are jointly input into the pre-trained vision-language model, the image-text matching score output by the model is normalized to obtain the image consistency score:

[0140]

[0141] , wherein represents the linear normalization operation on the image-text matching score output by the pre-trained vision-language model.

[0142] Then, a language quality score is calculated, specifically, only the candidate answer A is taken as the input text, the average word token probability of the pre-trained language model is calculated as a fluency index, and the fluency index is normalized to the interval [0, 1] to obtain the language quality score reflecting the syntax structure and readability:

[0143]

[0144] , wherein represents the linear normalization operation on the fluency index.

[0145] Finally, the three scores are linearly combined according to the preset weight to form the comprehensive score of the candidate answer:

[0146]

[0147] , , ​​​​​​​​​​​​​​​respectively, and satisfy The embodiment further sets a score threshold , only keeps answers with scores exceeding the score threshold, and uniformly formats the kept answers to form a candidate answer collection ; wherein, represents the kth kept candidate answer, and in the embodiment, k can be between 5 and 10, and the score threshold can be between 0.6 and 0.8.

[0148] Further, the embodiment can also remove highly similar answers through semantic similarity judgment to ensure that the output collection maintains diversity without redundancy, so as to filter out irrelevant, incorrect or semantically deviated content. The filtered and optimized candidate answer collection can be used as a reference answer space for subsequent large language model reasoning, which can significantly reduce the reasoning range and improve the output stability.

[0149] Figure 4 As shown in the figure, when the knowledge-based visual question answering lacks candidate answer generation processing, the performance will be significantly reduced, and the visual question answering accuracy performance is reduced by 2.78% on the OK-VQA dataset and 2.38% on the A-OKVQA dataset. This shows that providing a structured candidate answer space for large language models (LLMs) plays a key role in stabilizing the reasoning process and preventing irrelevant or hallucination outputs. Compared with entity enhancement, candidate answer generation has a stronger direct impact on the final prediction result, which further emphasizes its necessity in narrowing the reasoning range.

[0150] S5, context example retrieval module processing: based on the pre-trained multi-modal encoder, the visual features and text features of the image and the question are extracted, the comprehensive similarity scores of each sample in the training dataset are calculated, the most relevant image-question pairs are sorted according to the similarity, and the context example set is summarized . The S5 specifically includes the following contents:

[0151] S51, a pre-trained multi-modal encoder (BLIP) is used to extract visual features of the input image, text features of the input natural language question , and text features of the candidate question sampled from the preset training dataset

[0152]

[0153] wherein, and represent the visual and text encoders respectively, and in the embodiment, the preset training dataset is specifically the A-OKVQA dataset and the OKVQA dataset.​

[0154] Further, the visual features of the sample images and the text features of the sample questions in the preset training data set are extracted, wherein each sample contains a triple of image, question and answer.

[0155] S52, for each sample in the input and the training data set, the visual similarity and the text similarity are calculated respectively. The comprehensive similarity score is calculated by the following logic expression:

[0156]

[0157] wherein, and represent the weight coefficients of the visual similarity and the text similarity respectively, which can be dynamically adjusted according to the actual situation, and in the embodiment, the visual similarity and the text similarity are calculated by the cosine similarity algorithm, and the visual and text similarities are aggregated by the average value.

[0158] S53, the samples in the training data set are sorted according to the comprehensive similarity score , the top-K most relevant samples are selected as the context examples, the context examples are converted into a structured format suitable for the input of the large language model, and are summarized into a context example set, and the following logic expression is used:

[0159]

[0160] wherein, represents the context example after the formatting processing, is the concise text description of the i-th example image, is the i-th example question, is the i-th example corresponding answer, which is used to provide a multi-modal reference basis for the large model inference; in the embodiment, the top-K most relevant samples can be the top 5-15 samples in the ranking as the most relevant samples, which can preferentially cover the problem core reasoning types such as entity recognition, attribute judgment and knowledge association.

[0161] In the embodiment, step S5 gives different weights to the visual similarity and the text similarity, and combines both to participate in the example retrieval process, thereby improving the comprehensiveness and accuracy of the retrieval. Based on all training samples are sorted, and then the top-K examples with the highest relevance are selected; the embodiment not only makes full use of multi-modal information, but also optimizes the example selection process with efficient algorithms, and provides more representative and valuable data support for subsequent analysis or application.

[0162] S6, constructing a unified input prompt P adapted to the large language model based on the output results of steps S2-S5, and delivering the unified input prompt P to the large language model, and the large language model generating an answer through a self-recurrence manner. The S6 specifically includes the following contents:

[0163] S61, forming the unified input prompt P adapted to the large language model based on the output results of steps S2-S5 according to a fixed information structure; specifically, the output result of step S2 (the image information extraction module) is a complete image description , which is used to provide an accurate visual semantic basis for the large language model; the output result of step S3 (the entity enhancement module) is an entity example , which is used to eliminate entity ambiguity; the output result of step S4 (the candidate answer generation module) is a candidate answer set , which is used to limit the semantic range where the answer may be located; and the output result of step S5 (the context example retrieval module) is a context example set , which is used to provide examples similar to the current task reasoning mode, and provide a mode reference for model reasoning.

[0164] In the embodiment, the system concatenates the above contents and the task instruction in a fixed order, so that the prompt information is unfolded layer by layer from visual content, key object, example reasoning, candidate answer to final task target, forming a structured prompt with semantic levels, ensuring that the outputs of the four modules form joint constraints at the input end, rather than simple concatenation, thereby achieving multi-module collaboration in semantics and logic.

[0165] S62, taking the unified input prompt P as input, the large language model generates an answer word by word through a self-recurrence manner, and the prediction rule thereof is represented by the following logic:

[0166]

[0167] wherein, is a word unit generated at step t, is a word unit generated before step t, is an instruction prompt, which is used to prompt the large language model to generate a final answer according to the data such as annotations, entity information and candidate answers. According to the above formula, the output results of the different functional modules of steps S2-S5 are embedded in the unified input prompt P in different forms, thereby imposing multi-dimensional constraints on the generation process of the word unit ; wherein the complete image description contains visual description and label information, so that the large language model focuses on the content that actually exists in the image; the example set provides a reference for the reasoning mode, so that the model can reuse similar reasoning logic; and the candidate answer set limits the semantic range of the final answer, so that the output is closer to the candidate space of real and reliable candidates.

[0168] In the present embodiment, the large language model adopts LLaMA2 (13b), the parameters of the large language model remain frozen, and the inference process generates answers in an autoregressive manner.

[0169] Further, to improve the stability of the large language model in generating answers, the present embodiment further adopts a multi-query integration strategy to set different decoding parameters under the same prompt P, including temperature, top-k, and repetition penalty value, to generate multiple inference sequences in parallel.

[0170] Further, the final answer is output in the form of a natural language sequence and is terminated when the end symbol is generated. The system can explicitly associate the answer with the key object in the image or the prompt information when needed to enhance the result interpretability. The present embodiment combines image semantics, object labels, candidate answers, and example reasoning through the construction of a unified input prompt and the multi-query integration strategy, so that the four modules form a complementary and collaborative mechanism in the inference stage, thereby obtaining a stable, reasonable, and interpretable final answer.

[0171] Further, the knowledge-based visual question answering method based on large and small model collaboration proposed in the present embodiment is verified.

[0172] As Figure 2 shown, in the comparative experiment on the OK-VQA dataset, the question-answering accuracy of the generation method based on GPT-3 (such as the PICa-Full model, the Prophet model, and the FIIG (Ensemble) model) is between 48.0% and 61.3%. Although the Prophet model and the FIIG model improve the performance through heuristic prompts and integration strategies, they rely on pure text prompts and are difficult to fully capture fine-grained visual semantic information, thus cannot completely solve knowledge-intensive reasoning tasks; in contrast, the multi-modal pre-training model (such as the SmOLa model, the VIGC model, the PaLI model, and the VPD model) performs better, with an accuracy of between 62.4% and 64.6%. This result shows that multi-modal pre-training and instruction adjustment can effectively connect the visual and language modalities, thereby improving the performance of the model in the visual question answering task.

[0173] And the model proposed in the present embodiment has an accuracy of 62.57% (as Figure 2 Ours (Base) in the present embodiment), which has surpassed most models based on GPT-3 and is close to the performance of the PaLI model; after introducing the entity enhancement and candidate answer generation modules, the accuracy of the complete model is further improved to 66.72% (as Figure 2Ours (Full) outperformed the strongest baseline model, VPD, by as much as 2.12% in accuracy. This result demonstrates that modular design can effectively reduce entity ambiguity, enrich contextual reasoning, and generate more accurate and knowledge-driven answers.

[0174] like Figure 3 As shown, the A-OKVQA dataset is larger in scale and covers a wider range of knowledge-intensive questions compared to the OK-VQA dataset. In experiments on the A-OKVQA dataset, the GPT-3-based model exhibited certain performance limitations. Specifically, the Prophet model achieved a question-answering accuracy of 58.2%, while the FIIG (Ensemble) model achieved 59.8%, representing only a slight improvement in accuracy compared to their results on the OK-VQA dataset. This indicates that when facing the more complex and knowledge-intensive A-OKVQA dataset, relying solely on heuristic hints is insufficient for effective performance optimization and generalization, further highlighting the limitation of heuristic hints in effectively generalizing to datasets with higher reasoning complexity.

[0175] Meanwhile, multimodal pre-trained models achieved more competitive results. For example, the InstructBLIP (Vicuna-7B) model achieved an accuracy of 64.0%, the VIGC model improved slightly to 64.1%, the VPD model achieved an accuracy of 62.7%, and the SmOLa model led the baseline models with an accuracy of 65.3%. These results further confirm the advantages of multimodal instruction tuning and ensemble reasoning mechanisms in handling complex, knowledge-intensive visual question answering tasks.

[0176] like Figure 3 As shown, the proposed model exhibits superior performance in both configurations. The accuracy of the base model reaches 66.32%, surpassing all baseline models. After introducing entity examples and candidate answers, the accuracy of the complete model further improves to 69.51%, exceeding the strongest baseline by as much as 4.21%. The improved consistency across different datasets highlights two major advantages of this invention: first, entity augmentation effectively resolves semantic ambiguity in complex problems; second, candidate answer guidance stabilizes the reasoning process and reduces error propagation. These findings validate the robustness and generalization ability of the proposed multi-module framework in diverse, knowledge-intensive visual question-answering scenarios, indicating its broad application prospects.

[0177] Example 2

[0178] An apparatus includes a memory and a processor, the memory being used to store a program that supports the processor in executing the knowledge-based visual question-answering method based on size-model collaboration as described in Embodiment 1, the processor being configured to execute the program stored in the memory.

[0179] Embodiment three

[0180] A storage medium, a computer program is stored on the storage medium, the computer program is run by the processor to execute the steps of the knowledge-based visual question answering method based on the size model cooperation in embodiment one.

[0181] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit it; although the foregoing embodiments of the present application are described in detail, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A knowledge-based visual question answering method based on size model cooperation, characterized in that, Comprising the following steps: S1, receiving a user inputted image to be analyzed and a natural language question related to the image content, and performing standardization processing; S2, fuse the image after the standardization processing with the question, generate the image description of the associated question, identify the key object in the image to generate a key object label set, fuse the image description and the key object label set, and output a complete image description ; wherein, denotes an image, denotes an initial question, denotes a question-oriented image description, and denotes a key object label set; S3, extracting key entities in the natural language question, combining the key object label set to construct a clarification question; taking the clarification question and the standardized processed image as input, generating entity examples based on the pre-trained visual-linguistic model; wherein the entity examples are individual , represent the clarification question, and OFA represents a general multi-modal pre-trained visual-linguistic model; S4, generating a set of original candidate answers based on a multi-round generation strategy, comprehensively evaluating the set of original candidate answers by using a built-in scoring mechanism of the model, and optimizing the set of candidate answers ; S5, based on the pre-trained multi-modal encoder, extracting visual features and text features of the image and the question, calculating a comprehensive similarity score of each sample in the training data set, screening out the most relevant image-question pair according to the similarity, and collecting the context example set ; S6, based on the output results of steps S2-S5, constructing a unified input prompt suitable for the large language model and passing it to the large language model to generate an answer, wherein the unified input prompt P is specifically spliced in a fixed order based on the output results of steps S2-S5 and the task instruction; the output results of steps S2-S5 are complete image descriptions in turn , entity examples , candidate answer sets , and context example sets .

2. The size model based collaborative knowledge visual question answering method according to claim 1, wherein, The S2 comprises the following steps: S21, based on the image to be analyzed, a pre-trained visual model is used to generate visual features highly consistent with the natural language question, specifically: First, the image is divided into multiple regional features, and the question is parsed into a corresponding semantic vector; Subsequently, the importance of each region is dynamically adjusted based on semantic relevance, specifically: The regional features of the image and the natural language question are respectively encoded into comparable semantic representations, and the correlation degree between each regional feature and the question semantics is calculated, and the regional features are sorted and weighted according to the correlation degree; S22, taking the visual features after semantic reinforcement processing as input, using a language generation model to output a question-oriented image description, the question-oriented image description specifically includes key object, attribute and scene information which is most relevant to the question semantics; S23, a key object label set is extracted through visual-semantic alignment, and a complete image description is output by fusing the question-oriented image description and the key object label set, specifically: First, an object detection model is used to process the image to obtain a candidate object set, including object category, location and confidence; Next, after obtaining the semantic attention region of the question, Grad-CAM is used to visualize the internal feature response of the visual model to highlight the image region most relevant to the current question as the semantic attention region; Then, the semantic consistency between the semantic attention region and the candidate object is determined to determine whether the candidate object is a key object; Specifically, when the semantic relevance between the category, location and attribute of the candidate object and the semantic attention region is all not lower than the corresponding threshold, the candidate object is determined to be a key object; Otherwise, it is determined that the candidate object is a non-key object or a background object; Finally, the key object label set is extracted, including the object name and core attributes, to truly reflect the image entities that need to be concerned by the question; the question-oriented image description and the key object label set are fused to output a complete image description , using the following logic expression: wherein, represents an image, represents an initial question, represents a question-oriented image description, and represents a set of key object labels.

3. The size model based collaborative knowledge visual question answering method according to claim 2, wherein, The semantic relevance between the category, location and attribute of the candidate object and the semantic attention region is all not lower than the corresponding threshold, specifically including the following contents: S231, the semantic similarity between the category semantics of the candidate object and the visual semantics represented by the semantic attention region in the cross-modal embedding space is not lower than a first threshold; S232, the spatial overlap between the location of the candidate object and the semantic attention region is not lower than a second threshold; S233, the consistency score between the core attribute semantics of the candidate object and the high response attribute feature corresponding to the semantic attention region is not lower than a third threshold.

4. The size model based collaborative knowledge visual question answering method according to claim 2, wherein, The S3 comprises the following steps: S31, using a pre-trained language model to perform semantic analysis on the natural language question after text cleaning, extracting key entities in the question, including but not limited to objects, attributes, scene components, and obtaining a key entity set wherein, represents the mth core semantic unit that may affect the inference result. S32, based on the key entity set extracted in S31 and the key object label set extracted in S23, a clarification question is constructed, specifically the original natural language question and the identified key entity are input into a pre-trained language model in a fixed format, and the model generates an additional question directly related to the key entity as a clarification question based on the above prompt; S33, input the clarification question and the input image into the pre-trained visual-linguistic model OFA jointly to generate an entity example with clear semantic direction, including the classification, key attributes, functions or scene association information of the entity; specifically, the entity example individual is represented by the following logic : wherein, denotes the clarification question, and OFA denotes a general multimodal pre-trained visual-linguistic model.

5. The size model based collaborative knowledge visual question answering method according to claim 4, wherein, The S4 comprises the following steps: S41, using a pre-trained visual-language model OFA, taking the input image I and the natural language question Q as input, generating a multi-dimensional original candidate answer set based on a multi-round generation strategy; S42, a model built-in scoring mechanism is used to comprehensively evaluate the original candidate answer set, and the candidate answer set is optimized .

6. The size model based collaborative knowledge visual question answering method according to claim 5, wherein, The S42 specifically includes the following content: First, a semantic matching score is calculated, specifically, the natural language question Q and the first candidate answer are respectively input into the same pre-trained language encoder to obtain corresponding semantic vector representations and , the semantic correlation degree of the two is calculated using a similarity function, and the semantic correlation degree is mapped to the [0, 1] interval through linear normalization to obtain the semantic matching score : wherein, represents a linear normalization operation on the semantic relevance degree output by the similarity function; Secondly, calculate the image consistency score, specifically, input the image I and the first candidate answer into the pre-trained visual-language model, output the image-text matching score from the model, and normalize the score to obtain the image consistency score : wherein, denotes a linear normalization operation on the image-text matching scores output by the pre-trained visual-linguistic model; Then, the language quality score is calculated, specifically only with the candidate answer As the input text, the average wordpiece probability is calculated by using the pre-trained language model as the fluency index, and is normalized to the interval [0, 1] to obtain the language quality score reflecting the syntactic structure and readability : wherein, represents a linear normalization operation on the fluency index; Finally, the above three scores are linearly combined according to preset weights to form a comprehensive score of the candidate answer : wherein, , , respectively represent non-negative weight coefficients, and satisfy ; Setting a score threshold , only retaining answers with scores exceeding the score threshold, and uniformly formatting the retained answers to form a candidate answer collection ; wherein, represents the th retained candidate answer.

7. The size model based collaborative knowledge visual question answering method according to claim 5, wherein, The S5 includes the following steps: S51, a pre-trained multi-modal encoder is used to extract visual features of the input image , text features of the input natural language question , and text features of the candidate questions sampled from the pre-set training data set , using the following logic representation: wherein, and denote visual and textual encoders, respectively; S52, for each sample of the input and the training dataset, respectively, compute visual similarity with text similarity , compute the integrated similarity score using the following logic representation: wherein, with respectively represent the weight coefficients of visual similarity and text similarity; S53, according to the comprehensive similarity score ranking the samples of the training dataset, selecting top-K most relevant samples as context examples, converting the context examples into a structured format that fits the large language model input, aggregating into a context example set, using the following logic representation: wherein, represents an example of the context after the formatting processing, is a short text description of the i-th example image, is the i-th example question, is the i-th example corresponding answer.

8. The size model based collaborative knowledge visual question answering method according to claim 1, wherein, The S6 includes the following steps: S61, based on the output results of steps S2-S5, forming a unified input prompt P suitable for the large language model according to a fixed information structure; specifically, the output result of step S2 is a complete image description , the output result of step S3 is an entity example , the output result of step S4 is a candidate answer set , and the output result of step S5 is a context example set . S62, taking the unified input prompt P as input, the large language model generates answers word by word in a self-recurrent manner, and the prediction rule uses the following logic expression: wherein, is a word token generated for step t, is a word token generated prior to step t, is an instruction prompt.

9. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the knowledge-based visual question answering method based on the size model cooperation according to any one of claims 1-8, and the processor is configured to execute the program stored in the memory.

10. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to execute the steps of the knowledge-based visual question answering method based on the size model cooperation according to any one of claims 1-8.

Images