Intellectual visual question and answer method and device based on big and small model collaboration and medium

Abstract

The invention discloses a knowledge-based visual question-answering method and device based on big and small model collaboration and a medium, belongs to the technical field of visual question-answering, and solves the problem of how to improve the accuracy of complex questions faced by a visual question-answering technology. In the image description extraction step, an image description and key object label set which is highly associated with a natural language problem is generated, and in the entity enhancement processing step, key entities are extracted and a clarity problem and an entity example which are used for resolving semantic ambiguity are generated. In the candidate answer generation step, a plurality of semantically complementary answers are generated under different reasoning dimensions on the basis of a multi-round generation strategy, optimized candidate answers are screened on the basis of a model built-in scoring mechanism, in the context example retrieval step, examples most related to current input are retrieved, and finally, the candidate answers are obtained. And the large language model receives a unified input prompt formed by splicing output results in the steps, and inference is performed in an autoregression mode to generate answers, so that the accuracy of the visual question-answering technology facing complex questions is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of visual question answering technology, and relates to a knowledge-based visual question answering method, device and medium based on the collaboration of large and small models. Background Technology

[0002] Visual question answering is one of the core tasks in the field of multimodal artificial intelligence, aiming to enable machines to understand image content and answer related natural language questions. Existing technologies, such as the invention patent CN119357430A, disclose a visual question answering processing method. By analyzing dialogue information, it can determine the user's question about a product. Then, based on the question and the product's knowledge graph, it can extract a series of frames most relevant to the question from a longer product question video. These frames are then used to generate a shorter video, which is displayed synchronously with the question. However, with the development of deep learning, existing visual question answering methods have the following shortcomings when handling complex problems: (1) Insufficient alignment between vision and semantics: The image descriptions generated by existing methods are usually general descriptions, lacking specific descriptions for specific problems, resulting in omission of key information or interference from redundant information, which affects the model's accurate understanding and reasoning ability of the problem.

[0003] (2) Entity semantic ambiguity: For complex problems involving fine-grained or domain-specific entities, existing models have difficulty distinguishing entities that are similar in appearance or semantics, resulting in ambiguity or inaccuracy in the generated answers, which reduces the reliability of the model.

[0004] (3) Insufficient context support: Most existing methods ignore the example-driven reasoning mechanism and rely only on plain text context retrieval. They ignore the multimodal characteristics of visual question answering tasks, which makes it difficult for the model to effectively utilize prior knowledge when facing complex problems that require multi-step reasoning, resulting in limited generalization ability.

[0005] Therefore, it is particularly important to propose a visual question answering solution that can effectively integrate visual and semantic information, resolve entity ambiguity, and provide rich multimodal contextual support in order to improve the accuracy and robustness of visual question answering technology in the face of complex problems. Summary of the Invention

[0006] The technical problem to be solved by this invention is how to improve the accuracy of visual question answering technology in the face of complex questions.

[0007] The present invention solves the above-mentioned technical problems through the following technical solutions: A knowledge-based visual question answering method based on size-model collaboration includes the following steps: S1, receive the image to be analyzed and the natural language questions related to the content of the image input by the user, and perform standardization processing; S2 merges the standardized image with the question to generate an image description related to the question, identifies key objects in the image to generate a set of key object labels, and outputs a complete image description. S3 extracts key entities from natural language questions and constructs clarification questions by combining key object label sets; using the clarification questions and standardized images as input, it generates entity examples based on a pre-trained vision-language model. S4. Generate an original set of candidate answers based on a multi-round generation strategy, and use the model's built-in scoring mechanism to comprehensively evaluate the original set of candidate answers and optimize the set of candidate answers. S5: Based on the pre-trained multimodal encoder, visual and textual features of images and questions are extracted. The comprehensive similarity score of each sample in the training dataset is calculated. The most relevant image-question pairs are selected based on similarity and summarized into a context example set. S6. Based on the output results of steps S2 to S5, construct a unified input prompt that adapts to the large language model and pass it to the large language model to generate the answer.

[0008] Further, S2 includes the following steps: S21, based on the image to be analyzed, uses a pre-trained visual model to generate visual features that highly fit the natural language problem, specifically: First, the image is divided into multiple region features, and the problem is parsed into corresponding semantic vectors; Subsequently, the importance of each region is dynamically adjusted based on semantic relevance, specifically as follows: The image's regional features and natural language questions are encoded into comparable semantic representations. Then, the degree of association between each regional feature and the question's semantics is calculated. Based on the degree of association, the regional features are ranked and weights are assigned in order. S22, taking the semantically enhanced visual features as input, and using a language generation model to output a problem-oriented image description, the problem-oriented image description is specifically an image description most relevant to the semantics of the problem, including key objects, attributes and scene information; S23, extracts a set of key object labels using a visual-semantic alignment method, merges the problem-oriented image description with the set of key object labels, and outputs a complete image description, specifically: First, the image is processed using an object detection model to obtain a set of candidate objects, including object category, location, and confidence level. Next, after obtaining the problem-oriented semantic interest region, Grad-CAM is used to visualize the internal feature response of the visual model to highlight the image region most relevant to the current problem as the semantic interest region; Then, the semantic consistency between the semantic interest region and the candidate object is determined to determine whether the candidate object is a key object. Specifically, when the semantic relevance between the category, position and attribute of the candidate object and the semantic interest region is not lower than the corresponding threshold, the candidate object is determined to be a key object; otherwise, the candidate object is determined to be a non-key object or a background object. Finally, a set of key object labels, including object names and core attributes, is extracted to accurately reflect the image entities that the problem requires attention to. The problem-oriented image description and the set of key object labels are then fused to output a complete image description. This can be represented using the following logic:

[0009] in, Represents an image. This represents the initial question. This represents a problem-oriented image description, while This represents a set of key object tags.

[0010] Furthermore, the semantic relevance between the category, location, and attribute of the candidate object and the semantic interest region is not lower than the corresponding threshold, specifically including the following: S231, the semantic similarity between the category semantics of the candidate object and the visual semantics represented by the semantic interest region in the cross-modal embedding space is not lower than the first threshold; S232, The spatial overlap between the location of the candidate object and the semantic interest region is not less than the second threshold; S233, the consistency score between the core attribute semantics of the candidate object and the high-response attribute features corresponding to the semantic interest region is not lower than the third threshold.

[0011] Further, S3 includes the following steps: S31, employs a pre-trained language model to perform semantic parsing on the natural language problem after text cleaning, extracting key entities from the problem, including but not limited to objects, attributes, and scene components, to obtain a set of key entities. ,in, This represents the m-th core semantic unit that may affect the reasoning result; S32, based on the key entity set extracted in S31 and the key object label set extracted in S23, construct a clarification question. Specifically, input the original natural language question and the identified key entities into the pre-trained language model in a fixed format. Based on the above prompts, the model automatically generates an additional question directly related to the key entity as a clarification question. S33, jointly input the clarification question and the input image into the pre-trained visual-language model OFA to generate entity examples with clear semantic orientation, including entity classification, key attributes, and functional or scene-related information; specifically, the following logic is used to represent individual entity examples. :

[0012] in, To clarify the question, OFA stands for General Multimodal Pretrained Vision-Language Model.

[0013] Further, S4 includes the following steps: S41 uses a pre-trained visual-language model OFA, taking the input image I and the natural language question Q as input, and generates a multi-dimensional set of original candidate answers based on a multi-round generation strategy; S42 employs a built-in scoring mechanism to comprehensively evaluate the original candidate answer set and optimize it. .

[0014] Furthermore, S42 specifically includes the following: First, calculate the semantic matching score, specifically by matching the natural language question Q with the first... One candidate answer Each input is fed into the same pre-trained language encoder to obtain the corresponding semantic vector representation. and The semantic relevance between the two elements is calculated using a similarity function, and then mapped to the [0,1] interval using linear normalization to obtain the semantic matching score. :

[0015] in, This indicates a linear normalization operation performed on the semantic relevance of the similarity function output; Secondly, the image consistency score is calculated, specifically by comparing the input image I with the first image I. One candidate answer The image consistency score is obtained by jointly inputting a pre-trained vision-language model, outputting an image-text matching score, and then normalizing the score. :

[0016] in, This indicates that the image-text matching score output by the pre-trained vision-language model is linearly normalized. Then, the language quality score is calculated, specifically based on only candidate answers. As input text, the average word probability is calculated using a pre-trained language model as a fluency metric and normalized to the [0,1] interval to obtain a language quality score reflecting syntactic structure and readability. :

[0017] in, This indicates that the fluency metric has been linearly normalized. Finally, the scores from the three categories are linearly combined according to preset weights to form a comprehensive score for the candidate answers. :

[0018] in, , , Let represent the non-negative weight coefficients respectively, and satisfy . ; Set a rating threshold Only answers with scores exceeding a certain threshold are retained, and these retained answers are then formatted in a standardized way to form a collection of candidate answers. ;in, Indicates the first One of the remaining candidate answers.

[0019] Further, S5 includes the following steps: S51 employs a pre-trained multimodal encoder to extract visual features from the input image. Text features of input natural language problems and text features of candidate questions sampled from a pre-defined training dataset. This can be represented using the following logic:

[0020] in, and These represent the visual and text encoders, respectively. S52, For each sample in the input and training datasets, calculate the visual similarity separately. Text similarity Calculate the overall similarity score This can be represented using the following logic:

[0021] in, and These represent the weighting coefficients for visual similarity and textual similarity, respectively. S53, based on comprehensive similarity score The samples in the training dataset are sorted, and the Top-K most relevant samples are selected as context examples. These context examples are then converted into a structured format suitable for the input of a large language model and summarized into a context example set, represented using the following logic:

[0022] in, This represents an example of a formatted context. A concise text description for the i-th example image. For the i-th example problem, This is the answer corresponding to the i-th example.

[0023] Further, S6 includes the following steps: S61, based on the output results of steps S2 to S5, form a unified input prompt P adapted to the large language model according to a fixed information structure; specifically, the output result of step S2 is a complete image description. Step S3 outputs an entity example. Step S4 outputs a set of candidate answers. Step S5 outputs a set of context examples. ; S62, taking the unified input prompt P as input, the large language model generates the answer word by word through autoregression, and its prediction rule is represented by the following logic:

[0024] in, For the lexical units generated in step t, For the words generated before step t, This is a prompt for the instruction.

[0025] An electronic device includes a memory and a processor, the memory being used to store a program that supports the processor in executing the above-described knowledge-based visual question answering method based on size model collaboration, and the processor being configured to execute the program stored in the memory.

[0026] A storage medium storing a computer program, which, when executed by a processor, performs the steps of the above-described knowledge-based visual question answering method based on size-model collaboration.

[0027] The advantages of this invention are: This invention proposes a knowledge-based visual question answering method based on large and small model collaboration. In the image description extraction step, it generates an image description and key object label set that are highly related to the natural language question, which significantly enhances the alignment effect between visual information and semantic information. It solves the problem of insufficient alignment between visual content and question semantics in existing methods, and guides the large language model to more accurately focus on the question-related areas in the image, thereby significantly improving inference performance. In the entity augmentation process, by extracting key entities and generating clarifying questions and entity examples to resolve semantic ambiguity, contextual knowledge can be effectively supplemented and semantic ambiguity reduced at the entity level, enabling large language models to obtain more stable and accurate entity representations. In the candidate answer generation step, a number of semantically complementary answers are generated under different reasoning dimensions based on a multi-round generation strategy, and a multi-dimensional original candidate answer set is constructed. The optimized candidate answer set is further filtered based on the built-in scoring mechanism of the model as a reference answer space for subsequent large language model reasoning, which can significantly reduce the reasoning range and improve the output stability. In the context example retrieval step, the model is provided with rich multimodal context information by retrieving the most relevant examples to the current input. This not only enhances the model's adaptability to new problems, but also significantly improves its generalization performance, enabling it to generate high-quality answers more stably when facing complex reasoning tasks. Finally, the large language model receives the unified input prompt formed by concatenating the output results of the above steps, and generates the answer through autoregressive reasoning. This invention constructs a collaborative framework of large and small models based on a pre-trained visual-language model and a large language model. By combining visual information and natural language processing, it achieves accurate understanding of image content and effective reasoning based on external knowledge, thereby generating accurate answers and completing the entire question-answering reasoning process from input to output. This effectively improves the accuracy of visual question answering technology in the face of complex problems. It has superior robustness and generalization ability in diverse and knowledge-intensive visual question answering scenarios and has good application prospects. Attached Figure Description

[0028] Figure 1 This is a flowchart illustrating the knowledge-based visual question answering method based on size model collaboration according to Embodiment 1 of the present invention. Figure 2 This is a schematic diagram comparing the performance of this embodiment of the present invention with existing methods using the OK-VQA dataset in terms of knowledge resources and accuracy. Figure 3 This is a schematic diagram comparing the performance of this embodiment of the present invention with existing methods using the A-OKVQA dataset in terms of knowledge resources and accuracy. Figure 4This is a schematic diagram comparing the impact of each step in Embodiment 1 of the present invention on the OK-VQA and A-OKVQA datasets. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments: Example 1 This invention uses a pre-trained visual-language model as a small model to handle subdivided tasks and comprehensively optimizes the reasoning capabilities of a large language model to form a collaborative framework of small and large models. This framework generates accurate answers from user input, completing the entire question-answering reasoning process from input to output. Figure 1 As shown, specifically, this invention discloses a knowledge-based visual question answering method based on size model collaboration, including the following steps: S1, Receive Input and Preprocess: Receive the image to be analyzed and natural language questions related to the image content from the user, and perform standardization processing.

[0031] In this embodiment, the system first receives the image to be analyzed input by the user. The image format is RGB channel format, and the resolution is uniformly adjusted to 224×224 pixels. At the same time, the system receives natural language questions related to the content of the image. These natural language questions are knowledge-based questions, which need to be combined with the image content and external knowledge for reasoning. In this embodiment, knowledge-based questions include "What school of art does the building in the picture belong to?" and "Who is the author of this painting?"

[0032] In this embodiment, the standardization process includes the following: The input image to be analyzed is normalized, specifically by removing image noise and standardizing the image size to a standard resolution, in order to ensure the consistency and high quality of the image data. Text cleaning is performed on the input natural language problem, specifically filtering special characters, standardizing text case, and correcting grammatical errors, to ensure that subsequent modules can process input information stably and efficiently.

[0033] S2, Image Information Extraction: The standardized image is fused with the question to generate an image description related to the question. Key objects in the image are identified to generate a set of key object labels, and a complete image description is output. Specifically, S2 includes the following: S21, based on the image to be analyzed, uses a pre-trained visual model to generate visual features that highly fit the natural language problem, specifically: First, the image is divided into multiple region features, and the problem is parsed into corresponding semantic vectors.

[0034] Subsequently, the importance of each region is dynamically adjusted based on semantic relevance, specifically as follows: The image's regional features and natural language questions are encoded into comparable semantic representations. The correlation between each regional feature and the question's semantics is then calculated. The regional features are ranked according to the correlation and weights are assigned accordingly. If the correlation between the question's semantics and the regional features is high, higher weight values ​​are assigned to give the highly relevant regions more attention, while irrelevant regions are downgraded. The correlation can be any of semantic similarity, attention score, or cross-modal matching score.

[0035] In this embodiment, compared to traditional visual question answering methods that directly extract the overall description of an image or rely on fixed attention patterns, this invention introduces a semantic relevance-based filtering mechanism. This mechanism automatically filters out areas that are irrelevant to the question or have low relevance, thereby avoiding semantic bias caused by scattered visual information. Visual attention is automatically focused on the image portion most relevant to the question's meaning. For example, when the question asks for "architectural style," the model can prioritize architectural details in the image while ignoring background information irrelevant to the question.

[0036] This process ensures that the final generated visual features are clearly question-oriented and can effectively reflect the key visual content that the question points to. This makes the visual representations obtained based on this mechanism more focused and interpretable in terms of content. It not only improves the alignment accuracy between vision and language, but also provides more reliable basic information for subsequent image description generation and object label extraction. This enables the entire question-answering chain to establish stable semantic associations in the initial stage and improves the system's ability to understand complex knowledge-based questions.

[0037] In this embodiment, the pre-trained visual model uses a pre-trained multimodal encoder (BLIP). As an example of the implementation of this invention, the model has the ability to adapt to visual question answering tasks.

[0038] S22, taking the semantically enhanced visual features as input, and using a language generation model to output a problem-oriented image description, the problem-oriented image description is specifically an image description most relevant to the semantics of the problem, including key objects, attributes and scene information.

[0039] In this embodiment, the language generation model adopts the LLaMA2(13b) large language model, which simultaneously receives the semantic and visual features of the question during the generation process. This enables the decoder of the language generation model with conditional control capabilities to prioritize the image content most relevant to the question in terms of word selection and sentence structure.

[0040] In this embodiment, compared to traditional visual question answering which only focuses on the global scene and does not distinguish the question requirements, the present invention is constrained by the semantics of the question when generating image descriptions. The decoder can only extract object, attribute and scene information from visual areas with high semantic relevance, thereby avoiding the generation of redundant background or content unrelated to the question, making the generated results more in line with the question requirements.

[0041] This process ensures that the output image descriptions naturally revolve around the question, avoiding task-irrelevant descriptions and generalizations. For example, when asking about a person, it automatically emphasizes their attributes; when asking about architectural features, it focuses on relevant structures. For the question "Who is the artist of this painting?", the output description should be "There is a person in ancient clothing in the painting, and the background is a landscape," rather than the generalized description "a painting." The resulting image descriptions have a clear question orientation and high information density, providing more accurate input for subsequent small model modules and the large model's inference stage.

[0042] S23, extracts a set of key object labels using a visual-semantic alignment method, merges the problem-oriented image description with the set of key object labels, and outputs a complete image description, which specifically includes the following: First, the image is processed using an object detection model to obtain a set of candidate objects, including object category, location, and attributes; wherein, the object detection model can be any one of Faster R-CNN, Mask R-CNN, or RetinaNet.

[0043] Next, after obtaining the question-oriented semantic attention regions, Grad-CAM is used to visualize the internal feature responses of the visual model, highlighting the image regions most relevant to the current question as semantic attention regions. Grad-CAM generates an attention heatmap reflecting the relevance of the question based on the gradient information generated by the model when processing the image and the question, making the regions the model truly focuses on in the image clearly highlighted. By thresholding and segmenting the attention heatmap, multiple semantic attention regions can be obtained, representing the visual cues the model relies on most when answering the current question.

[0044] Then, the semantic consistency between the semantic interest region and the candidate object is determined to identify whether the candidate object is a key object. Specifically, based on the high-response visual semantic features highlighted by the semantic interest region, the category semantics and attribute semantics of the candidate object are matched and analyzed: when the semantic relevance between the category, position, and attribute of the candidate object and the semantic interest region is not lower than the corresponding threshold, the candidate object is determined to be a key object; otherwise, the candidate object is determined to be a non-key object or a background object.

[0045] Furthermore, the semantic relevance between the category, location, and attribute of the candidate object and the semantic interest region is not lower than the corresponding threshold, specifically including the following: S231, the semantic similarity between the category semantics of the candidate object and the visual semantics represented by the semantic interest region in the cross-modal embedding space is not lower than a first threshold; wherein, the semantic similarity can be obtained by any one of cosine similarity, dot product similarity or matching probability function; S232, The spatial overlap between the location of the candidate object and the semantic interest region is not less than the second threshold; S233, the consistency score between the core attribute semantics of the candidate object and the high-response attribute features corresponding to the semantic interest region is not lower than the third threshold.

[0046] Finally, a set of key object labels, including object names and core attributes, is extracted to accurately reflect the image entities that the problem requires attention to. The problem-oriented image description and the set of key object labels are then fused to output a complete image description. This can be represented using the following logic:

[0047] in, Represents an image. This represents the initial question. This represents a problem-oriented image description, while This represents a set of key object labels, such as identifying "people (wearing ancient clothing)" and "landscape (background)", forming... ={ (Name + Attributes) (name + attributes),..., The standardized format of (name + attribute)} shows that the image information output in step S2 still retains its original expression form.

[0048] This embodiment not only considers the explicit recognition results of traditional detectors but also introduces semantically relevant regions as a filtering criterion, ensuring consistency between the key object label set and the question semantics, thereby improving the overall visual-semantic alignment quality. This embodiment effectively filters out key objects truly related to the question semantics and includes the names and core attributes of these objects in the key object label set, while irrelevant or weakly related objects are excluded. The final key object set accurately reflects the entities in the image most closely related to the question semantics, providing accurate foundational information for subsequent functional modules.

[0049] S3, Entity Augmentation Processing: Extract key entities from the natural language problem and construct a clarification question by combining it with a set of key object labels; using the clarification question and the standardized image as input, generate entity examples based on a pre-trained vision-language model. Specifically, S3 includes the following: S31, employs a pre-trained language model to perform semantic parsing on the natural language problem after text cleaning, extracting key entities from the problem, including but not limited to objects, attributes, and scene components, to obtain a set of key entities. ,in, This represents the m-th core semantic unit that may affect the reasoning result. For example, from the question "What school of thought does the architectural style in the picture belong to?", the key entities "architecture", "style", and "school of thought" are extracted.

[0050] In this embodiment, the pre-trained language model can be an OFA (Optical Visual-Language Model). Unlike traditional keyword-based matching, this embodiment determines the semantic role of an entity in a sentence through contextual dependencies, thereby clarifying the question's target and modifiers at the structural level. This allows subsequent disambiguation functions to focus on the semantic core that truly influences reasoning.

[0051] S32, based on the key entity set extracted in S31 and the key object label set extracted in S23, constructs a clarification question to refine entity semantics and resolve ambiguities, thus avoiding model misjudgment.

[0052] In this embodiment, the process of constructing the clarification question is as follows: By using prompts, the Optical-Language Model (OFA) is guided to automatically generate clarifying questions to refine entity semantics, starting from the original question itself. Specifically, the original natural language question and the identified key entities are first input into the pre-trained language model in a fixed format, such as: "Please generate a clarifying question to further clarify the meaning of the key entity based on the following question: Original question: ××; Key entity: ××". Based on the prompts, the model automatically generates an additional question directly related to the key entity, helping the model to further distinguish semantics, as the clarifying question. This prompt-based, lightweight method of generating clarifying questions quickly yields clarifying questions closely related to the current natural language question context, without relying on complex rules or attribute analysis logic.

[0053] For example, when the entity is "architectural style," the system prioritizes external structure, roof shape, and color characteristics as the basis for questions, thus constructing clarifying questions such as "Is the building's roof structure pointed or flat?" and "Is the main color of the facade warm or cool?" When the entity is "clothing," the system generates questions such as "Are the clothing patterns traditional or modern patterns?" based on texture, pattern, or color, further refining the semantic features of the entity and providing more accurate semantic information. The clarification question generation mechanism proposed in this embodiment directly corresponds the clarification questions to the real observable features of the image, thereby forming semantically guiding questions with clear direction and significantly improving the technical effect of disambiguation.

[0054] S33, jointly input the clarification question and the input image into the pre-trained visual-language model OFA to generate entity examples with clear semantic orientation, including entity classification, key attributes, and functional or scene-related information; specifically, the following logic is used to represent individual entity examples. :

[0055] in, The clarification question is represented by OFA (One-For-All), which stands for General Multimodal Pre-trained Vision-Language Model. The example generation process guides the model to extract representative descriptive information from the image content based on the attribute dimension pointed to by the clarification question, enabling the generated results to naturally reflect entity category differences.

[0056] For example, for the entity "European architecture," the system can generate examples such as "characterized by pointed roofs and brick and stone exteriors, commonly found in traditional European neighborhoods"; for "traditional clothing," it can generate examples such as "ethnic clothing featuring highly saturated embroidery patterns, often used in festive occasions." By generating multi-dimensional examples, this invention can effectively supplement contextual knowledge and reduce semantic ambiguity at the entity level, enabling the model to obtain more stable and accurate entity representations, and laying a clear semantic foundation for subsequent candidate answer generation and reasoning stages.

[0057] like Figure 4 As shown, when knowledge-based visual question answering lacks entity augmentation, removing entity augmentation results in a 0.89% decrease in accuracy on the OK-VQA dataset and a 0.84% ​​decrease on the A-OKVQA dataset. A-OKVQA and OK-VQA are important datasets in the field of visual question answering (VQA), proposed in 2019 and 2022, respectively. Although the decrease is small, this result demonstrates that in scenarios where multiple semantically similar objects appear in an image, constructing clarifying questions to eliminate entity ambiguity can significantly enhance the reliability of inference, further confirming the important role of entity augmentation in improving the accuracy and robustness of model inference.

[0058] S4, Candidate Answer Generation Processing: An initial candidate answer set is generated based on a multi-round generation strategy. The model's built-in scoring mechanism is then used to comprehensively evaluate and optimize the initial candidate answer set. S4 specifically includes the following: S41 uses a pre-trained vision-language model OFA, taking the input image I and the natural language question Q as input, and generates a multi-dimensional set of original candidate answers based on a multi-round generation strategy.

[0059] In this embodiment, the input image I and the natural language question Q are combined as the input to the model decoder, and the decoding process is divided into several rounds. Each round generates candidate answers with different decoding parameters to ensure that the same question can generate several semantically complementary answers under different reasoning dimensions, thus constructing a multi-dimensional original candidate answer set.

[0060] Specifically, the decoding process includes an attribute wheel, a causal wheel, and a common sense wheel. The attribute wheel uses attribute judgment prompts to require the answer to focus on visible attributes in the image to generate candidate answers. The causal wheel uses causal reasoning prompts to require explanatory short sentences to generate candidate answers. The common sense wheel uses knowledge retrieval prompts to require combining image content with common sense base information to generate candidate answers. The decoding parameters include temperature, top-k, and repetition penalty value.

[0061] In this embodiment, the outputs of all rounds are aggregated into an original set of candidate answers, represented using the following logic:

[0062] in, This represents the original set of candidate answers.

[0063] S42 employs a built-in scoring mechanism to comprehensively evaluate the original candidate answer set and optimize it. Specifically, it includes the following: First, calculate the semantic matching score, specifically by matching the natural language question Q with the first... One candidate answer Each input is fed into the same pre-trained language encoder to obtain the corresponding semantic vector representation. and The semantic relevance between the two elements is calculated using a similarity function, and then mapped to the [0,1] interval using linear normalization to obtain the semantic matching score. :

[0064] in, This indicates that the semantic relevance of the similarity function output is linearly normalized. In this embodiment, the similarity function can be cosine similarity.

[0065] Secondly, the image consistency score is calculated, specifically by comparing the input image I with the first image I. One candidate answer The image consistency score is obtained by jointly inputting a pre-trained vision-language model, outputting an image-text matching score, and then normalizing the score. :

[0066] in, This indicates that the image-text matching score output by the pre-trained vision-language model is linearly normalized.

[0067] Then, the language quality score is calculated, specifically based on only candidate answers. As input text, the average word probability is calculated using a pre-trained language model as a fluency metric and normalized to the [0,1] interval to obtain a language quality score reflecting syntactic structure and readability. :

[0068] in, This indicates that the fluency metric has been linearly normalized.

[0069] Finally, the scores from the three categories are linearly combined according to preset weights to form a comprehensive score for the candidate answers. :

[0070] in, , , Let represent the non-negative weight coefficients respectively, and satisfy . This embodiment further sets a scoring threshold. Only answers with scores exceeding a certain threshold are retained, and these retained answers are then formatted in a standardized way to form a collection of candidate answers. ;in, Indicates the first The remaining candidate answers are k, which can be between 5 and 10 in this embodiment, and the scoring threshold can be between 0.6 and 0.8.

[0071] Furthermore, this embodiment can also deduplicate highly similar answers through semantic similarity judgment, ensuring that the output set remains diverse without redundancy, thereby filtering out irrelevant, incorrectly described, or semantically biased content. The optimized candidate answer set serves as a reference answer space for subsequent large language model inference, which can significantly reduce the inference scope and improve output stability.

[0072] Figure 4 As shown, the lack of candidate answer generation in knowledge-based visual question answering leads to a significant performance drop, with a 2.78% decrease in accuracy on the OK-VQA dataset and a 2.38% decrease on the A-OKVQA dataset. This demonstrates that providing a structured candidate answer space for large language models (LLMs) plays a crucial role in stabilizing the reasoning process and preventing irrelevant or illusory outputs. Compared to entity augmentation, candidate answer generation has a stronger direct impact on the final prediction results, further highlighting its necessity in narrowing the scope of reasoning.

[0073] S5, Context Instance Retrieval Module Processing: Based on a pre-trained multimodal encoder, visual and textual features of images and questions are extracted. The comprehensive similarity score of each sample in the training dataset is calculated, and the most relevant image-question pairs are selected based on similarity ranking, and then aggregated into a context instance set. S5 specifically includes the following: S51 employs a pre-trained multimodal encoder (BLIP) to extract visual features from the input image. Text features of input natural language problems and text features of candidate questions sampled from a pre-defined training dataset. This can be represented using the following logic:

[0074] in, and These represent the visual and text encoders, respectively. In this embodiment, the preset training datasets are specifically the A-OKVQA dataset and the OKVQA dataset.

[0075] Furthermore, visual features of sample images and textual features of sample questions are extracted from all samples in the preset training dataset, where each sample contains a triple consisting of an image, a question, and an answer.

[0076] S52, For each sample in the input and training datasets, calculate the visual similarity separately. Text similarity Calculate the overall similarity score This can be represented using the following logic:

[0077] in, and These represent the weighting coefficients for visual similarity and text similarity, respectively. These coefficients can be dynamically adjusted based on actual circumstances. In this embodiment, visual similarity... Text similarity The similarity was calculated using the cosine similarity algorithm and then aggregated using the average value to determine the visual and textual similarity.

[0078] S53, based on comprehensive similarity score The samples in the training dataset are sorted, and the Top-K most relevant samples are selected as context examples. These context examples are then converted into a structured format suitable for the input of a large language model and summarized into a context example set, represented using the following logic:

[0079] in, This represents an example of a formatted context. A concise text description for the i-th example image. For the i-th example problem, The answer corresponding to the i-th example is used to provide multimodal reference for large model reasoning. In this embodiment, the Top-K most relevant samples can be the top 5 to 15 samples in the ranking, which can preferentially cover the core reasoning types of problems such as entity recognition, attribute judgment, and knowledge association.

[0080] In this embodiment, step S5 improves the comprehensiveness and accuracy of the retrieval by assigning different weights to visual similarity and textual similarity, and by integrating both into the example retrieval process. Based on... All training samples are sorted, and then the Top-K examples with the highest relevance are selected. This embodiment not only makes full use of multimodal information, but also optimizes the example selection process with efficient algorithms, providing more representative and valuable data support for subsequent analysis or application.

[0081] S6, based on the output results of steps S2-S5, constructs a unified input prompt P adapted to the large language model and passes it to the large language model. The large language model generates the answer through an autoregressive approach. S6 specifically includes the following: S61, based on the output results of steps S2~S5, a unified input prompt P adapted to the large language model is formed according to a fixed information structure; specifically, the output result of step S2 (image information extraction module) is a complete image description. This is used to provide an accurate visual semantic foundation for large language models; step S3 (entity enhancement module) outputs entity examples. This is used to eliminate entity ambiguity; step S4 (candidate answer generation module) outputs a set of candidate answers. This is used to limit the semantic scope in which the answer may be located; step S5 (context example retrieval module) outputs a set of context examples. It is used to provide examples similar to the current task reasoning pattern, and to provide pattern reference for model reasoning.

[0082] In this embodiment, the system concatenates the above content with the task instructions in a fixed order, so that the prompt information unfolds layer by layer from visual content, key objects, example reasoning, candidate answers to the final task goal, forming a structured prompt with semantic hierarchy. This ensures that the outputs of the four modules form joint constraints at the input end, rather than being simply connected, thereby achieving multi-module collaboration in semantics and logic.

[0083] S62, taking the unified input prompt P as input, the large language model generates the answer word by word through autoregression, and its prediction rule is represented by the following logic:

[0084] in, For the lexical units generated in step t, For the words generated before step t, This is a prompt, used to guide the large language model in generating the final answer based on annotations, entity information, and candidate answers. According to the above formula, the outputs of different functional modules in steps S2-S5 are embedded in the unified input prompt P in different forms, thereby affecting the word segment... The generation process is subject to multi-dimensional constraints; among which, the complete image description It includes visual descriptions and label information, enabling large language models to focus on the real content in the image; the example set provides a reference for reasoning patterns, allowing the model to reuse similar reasoning logic; and the candidate answer set limits the semantic range of the final answer, making the output closer to a realistic and credible candidate space.

[0085] In this embodiment, the large language model adopts LLaMA2(13b), the parameters of the large language model are kept frozen, and the reasoning process generates answers in an autoregressive manner.

[0086] Furthermore, to improve the stability of the answers generated by the large language model, this embodiment adopts a multi-query integration strategy, setting different decoding parameters under the same prompt P, including temperature, top-k, and repetition penalty value, to generate multiple inference sequences in parallel.

[0087] Furthermore, the final answer is output in the form of a natural language sequence and terminates when the end symbol appears. The system can explicitly associate the answer with key objects or prompts in the image when needed to enhance the interpretability of the result. This embodiment fully integrates image semantics, object labels, candidate answers, and example reasoning by constructing a unified input prompt construction and multi-query integration strategy. This enables the four modules to form a complementary and collaborative mechanism during the reasoning stage, thereby obtaining a stable, reasonable, and interpretable final answer.

[0088] Furthermore, the knowledge-based visual question answering method based on size model collaboration proposed in this embodiment is verified.

[0089] like Figure 2 As shown in the comparative experiments on the OK-VQA dataset, the question-answering accuracy of GPT-3-based generation methods (such as the PICa-Full model, the Prophet model, and the FIIG (Ensemble) model) ranged from 48.0% to 61.3%. Although the Prophet and FIIG models improved performance through heuristic prompts and ensemble strategies, they rely on plain text prompts and struggle to fully capture fine-grained visual semantic information, thus failing to fully address knowledge-intensive reasoning tasks. In contrast, multimodal pre-trained models (such as the SmOLa model, the VIGC model, the PaLI model, and the VPD model) performed better, with accuracy ranging from 62.4% to 64.6%. This result demonstrates that multimodal pre-training and instruction tuning can effectively connect visual and linguistic modalities, thereby improving model performance in visual question-answering tasks.

[0090] The model proposed in this embodiment has an accuracy of 62.57% under basic settings (e.g., Figure 2Ours (Base) has surpassed most GPT-3-based models and is close to the performance of the PaLI model; and after introducing entity augmentation and candidate answer generation modules, the accuracy of the complete model has been further improved to 66.72% (e.g., Figure 2 Ours (Full) outperformed the strongest baseline model, VPD, by as much as 2.12% in accuracy. This result demonstrates that modular design can effectively reduce entity ambiguity, enrich contextual reasoning, and generate more accurate and knowledge-driven answers.

[0091] like Figure 3 As shown, the A-OKVQA dataset is larger in scale and covers a wider range of knowledge-intensive questions compared to the OK-VQA dataset. In experiments on the A-OKVQA dataset, the GPT-3-based model exhibited certain performance limitations. Specifically, the Prophet model achieved a question-answering accuracy of 58.2%, while the FIIG (Ensemble) model achieved 59.8%, representing only a slight improvement in accuracy compared to their results on the OK-VQA dataset. This indicates that when facing the more complex and knowledge-intensive A-OKVQA dataset, relying solely on heuristic hints is insufficient for effective performance optimization and generalization, further highlighting the limitation of heuristic hints in effectively generalizing to datasets with higher reasoning complexity.

[0092] Meanwhile, multimodal pre-trained models achieved more competitive results. For example, the InstructBLIP (Vicuna-7B) model achieved an accuracy of 64.0%, the VIGC model improved slightly to 64.1%, the VPD model achieved an accuracy of 62.7%, and the SmOLa model led the baseline models with an accuracy of 65.3%. These results further confirm the advantages of multimodal instruction tuning and ensemble reasoning mechanisms in handling complex, knowledge-intensive visual question answering tasks.

[0093] like Figure 3 As shown, the proposed model exhibits superior performance in both configurations. The accuracy of the base model reaches 66.32%, surpassing all baseline models. After introducing entity examples and candidate answers, the accuracy of the complete model further improves to 69.51%, exceeding the strongest baseline by as much as 4.21%. The improved consistency across different datasets highlights two major advantages of this invention: first, entity augmentation effectively resolves semantic ambiguity in complex problems; second, candidate answer guidance stabilizes the reasoning process and reduces error propagation. These findings validate the robustness and generalization ability of the proposed multi-module framework in diverse, knowledge-intensive visual question-answering scenarios, indicating its broad application prospects.

[0094] Example 2 An apparatus includes a memory and a processor, the memory being used to store a program that supports the processor in executing the knowledge-based visual question-answering method based on size-model collaboration as described in Embodiment 1, the processor being configured to execute the program stored in the memory.

[0095] Example 3 A storage medium storing a computer program, which, when executed by a processor, performs the steps of the knowledge-based visual question-answering method based on size model collaboration in Embodiment 1.

[0096] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A knowledge-based visual question answering method based on size model cooperation, characterized in that, The method comprises the following steps: S1, receiving a user-inputted image to be analyzed and a natural language question related to the image content, and performing standardization processing; S2, fusing the image after standardization processing and the question to generate an image description associated with the question, identifying key objects in the image to generate a key object label set, and outputting a complete image description; S3, extracting key entities in the natural language question, constructing a clarification question in combination with the key object label set, taking the clarification question and the image after standardization processing as inputs, and generating entity examples based on a pre-trained visual-linguistic model; S4, generating a set of original candidate answers based on a multi-round generation strategy, comprehensively evaluating the set of original candidate answers by using a built-in scoring mechanism of the model, and optimizing the set of candidate answers; S5, extracting visual features and text features of the image and the question based on a pre-trained multi-modal encoder, calculating comprehensive similarity scores of each sample in a training data set, screening out the most relevant image-question pair according to the similarity, and summarizing the image-question pair as a set of context examples; S6, based on the output results of steps S2-S5, constructing a unified input prompt suitable for a large language model and passing the unified input prompt to the large language model to generate an answer.

2. The size model based collaborative knowledge visual question answering method according to claim 1, wherein, The S2 comprises the following steps: S21, based on the image to be analyzed, a pre-trained visual model is used to generate visual features highly consistent with the natural language question, specifically: First, the image is divided into multiple regional features, and the question is parsed into a corresponding semantic vector; Then, the importance of each region is dynamically adjusted based on semantic correlation, specifically: The regional features of the image and the natural language question are respectively encoded into comparable semantic representations, the correlation between each regional feature and the question semantics is calculated, the regional features are sorted and weighted according to the correlation, and the regional features are sorted and weighted according to the correlation; S22, taking the visual features after semantic reinforcement processing as inputs, using a language generation model to output a question-oriented image description, the question-oriented image description specifically being an image description most relevant to the question semantics, including key object, attribute and scene information; S23, a key object label set is extracted through visual-semantic alignment, the question-oriented image description and the key object label set are fused, and a complete image description is output, specifically: First, an object detection model is used to process the image to obtain a candidate object set, including object category, position and confidence; Next, after obtaining the semantic attention region of the question, Grad-CAM is used to visualize the internal feature response of the visual model to highlight the image region most relevant to the current question as the semantic attention region; Then, the semantic consistency of the semantic attention region and the candidate object is determined to determine whether the candidate object is a key object; specifically, when the semantic correlation between the category, position and attribute of the candidate object and the semantic attention region is all not lower than the corresponding threshold, the candidate object is determined to be a key object; otherwise, the candidate object is determined to be a non-key object or a background object; Finally, the key object label set is extracted, including the object name and core attributes, to truly reflect the image entities that need to be concerned by the question; the question-oriented image description and the key object label set are fused to output a complete image description , using the following logic expression: wherein, represents an image, represents an initial question, represents a question-oriented image description, and represents a set of key object labels.

3. The size model based collaborative knowledge visual question answering method according to claim 2, wherein, The semantic correlation between the category, position and attribute of the candidate object and the semantic attention region all being not lower than the corresponding threshold specifically includes the following contents: S231, the semantic similarity between the category semantics of the candidate object and the visual semantics represented by the semantic attention region in the cross-modal embedding space is not less than a first threshold value; S232, the spatial overlap between the position of the candidate object and the semantic attention region is not less than a second threshold value; S233, the consistency score of the core attribute semantics of the candidate object and the high response attribute feature corresponding to the semantic attention region is not less than a third threshold value.

4. The size model based collaborative knowledge visual question answering method according to claim 2, wherein, The S3 comprises the following steps: S31, using a pre-trained language model to perform semantic analysis on the natural language question after text cleaning, extracting key entities in the question, including but not limited to objects, attributes, scene components, and obtaining a key entity set wherein, represents the mth core semantic unit that may affect the inference result. S32, based on the key entity set extracted in S31 and the key object label set extracted in S23, a clarification question is constructed, specifically, the original natural language question and the recognized key entity are input to the pre-trained language model in a fixed format, and the model generates an additional question directly related to the key entity as a clarification question according to the above prompt; S33, input the clarification question and the input image into the pre-trained visual-linguistic model OFA jointly to generate an entity example with clear semantic direction, including the classification, key attributes, functions or scene association information of the entity; specifically, the entity example individual is represented by the following logic : wherein, denotes the clarification question, and OFA denotes a general multimodal pre-trained visual-linguistic model.

5. The size model based collaborative knowledge visual question answering method according to claim 4, wherein, The S4 comprises the following steps: S41, using a pre-trained visual-language model OFA, taking the input image I and the natural language question Q as input, generating a multi-dimensional original candidate answer set based on a multi-round generation strategy; S42, a model built-in scoring mechanism is used to comprehensively evaluate the original candidate answer set, and the candidate answer set is optimized .

6. The size model based collaborative knowledge visual question answering method according to claim 5, wherein, The S42 specifically comprises the following contents: First, calculate the semantic matching score, specifically by matching the natural language question Q with the first... One candidate answer Each input is fed into the same pre-trained language encoder to obtain the corresponding semantic vector representation. and The semantic relevance between the two elements is calculated using a similarity function, and then mapped to the [0,1] interval using linear normalization to obtain the semantic matching score. : wherein, represents a linear normalization operation on the semantic relevance degree output by the similarity function; Secondly, calculate the image consistency score, specifically, input the image I and the first candidate answer joint input pre-training visual-language model, output image-text matching score from the model and normalize the score to obtain the image consistency score : wherein, denotes a linear normalization operation on the image-text matching scores output by the pre-trained visual-linguistic model; Then, the language quality score is calculated, specifically only with the candidate answer As the input text, the average token probability is calculated by using the pre-trained language model as the fluency index, and is normalized to the interval [0, 1] to obtain the language quality score reflecting the syntactic structure and readability : wherein, represents a linear normalization operation on the fluency index; Finally, the above three scores are linearly combined according to preset weights to form a comprehensive score of the candidate answer : wherein, , , are non-negative weight coefficients respectively represented by ; Setting a score threshold , only retaining answers with scores exceeding the score threshold, and uniformly formatting the retained answers to form a candidate answer collection ; wherein, represents the th retained candidate answer.

7. The size model based collaborative knowledge visual question answering method according to claim 5, wherein, The S5 comprises the following steps: S51, a pre-trained multi-modal encoder is used to extract visual features of the input image , text features of the input natural language question , and text features of the candidate questions sampled from the pre-set training data set , using the following logic representation: wherein, and represent visual and textual encoders, respectively; S52, for each sample of the input and the training dataset, respectively, compute visual similarity with text similarity , compute the integrated similarity score using the following logic representation: wherein, with respectively represent the weight coefficients of visual similarity and text similarity. S53, according to the comprehensive similarity score ranking the samples of the training dataset, selecting top-K most relevant samples as context examples, converting the context examples into a structured format that fits the large language model input, aggregating into a context example set, using the following logic representation: wherein, represents an example of the context after formatting processing, is a short text description of the i-th example image, is the i-th example question, is the i-th example answer.

8. The size model based collaborative knowledge visual question answering method according to claim 1, wherein, The S6 comprises the following steps: S61, based on the output results of steps S2-S5, forming a unified input prompt P suitable for the large language model according to a fixed information structure; specifically, the output result of step S2 is a complete image description , the output result of step S3 is an entity example , the output result of step S4 is a candidate answer set , and the output result of step S5 is a context example set . S62, taking the unified input prompt P as input, the large language model generates answers word by word through autoregressive method, and its prediction rule uses the following logic expression: wherein, a wordpiece generated for step t, a wordpiece generated prior to step t, is an instruction prompt.

9. An electronic device comprising a memory and a processor, characterized in that The memory is used to store a program supporting the processor to execute the knowledge-based visual question answering method based on the cooperation of large and small models according to any one of claims 1 to 8, and the processor is configured to execute the program stored in the memory.

10. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to perform the steps of the knowledge-based visual question answering method based on the cooperation of large and small models according to any one of claims 1 to 8.

Images