Visual language reasoning method and system fusing context learning and chain reasoning

Abstract

The invention belongs to the technical field of artificial intelligence, and particularly relates to a visual language reasoning method and system fusing context learning and chain reasoning, the method is divided into two stages to carry out multi-modal reasoning, in the first stage, an ROI region of a target image is recognized in combination with a context example and a target problem; and in the second stage, combining the global feature and the ROI region of the target image, the global feature and the ROI feature of the context example and the target question, and predicting to obtain a final text answer. According to the method, local feature extraction and fusion of local features and global features are realized, and stage visual language reasoning is performed in combination with a context example, so that the logical reasoning ability and robustness of a visual language task can be improved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and specifically relates to a visual language reasoning method and system that integrates contextual learning and chain reasoning. Background Art

[0002] Visual language models (VLMs) are a class of multimodal AI systems that can learn from both images and text. They can handle a variety of visual language tasks, such as visual question answering, image captioning, and text-to-image search. A key challenge in VLMs is extracting task-relevant local details from high-resolution images. However, their weak ability to extract local details leads to poor reasoning, poor robustness and accuracy of logical reasoning, and difficulty handling complex visual language reasoning tasks. Furthermore, while VLMs perform well in image generation and reasoning tasks, they rely on single-step reasoning, fail to fully consider the iterative fusion of global and local information, and lack effective utilization of contextual examples. This significantly limits their application in tasks requiring sophisticated visual understanding and dynamic reasoning, which typically require reasoning paths that match human cognitive processes.

[0003] Chained reasoning (CoT) and in-context learning (ICL) have shown significant potential in multimodal reasoning tasks. CoT enhances the explainability of reasoning by decomposing the task into a step-by-step logical path, while ICL dynamically integrates contextual information to improve the adaptability of reasoning. However, most existing studies regard CoT and ICL as independent mechanisms and fail to fully exploit their complementary advantages. Summary of the Invention

[0004] The purpose of the present invention is to address the above-mentioned problems existing in the prior art and to provide a visual language reasoning method and system that integrates contextual learning and chain reasoning, which can efficiently fuse local visual information with global visual information and make full use of contextual example information, thereby significantly improving the logical reasoning ability and robustness of visual language tasks.

[0005] To achieve the above objectives, the technical solutions of the present invention are as follows:

[0006] In a first aspect, the present invention provides a visual language reasoning method that integrates contextual learning and chain reasoning. The visual language reasoning method includes the following two-stage visual language reasoning:

[0007] The first stage is to construct a visual chain reasoning dataset, in which the sample images of the visual chain reasoning dataset are embedded with questions, answers and ROI areas corresponding to the questions; sample images related to the questions and the target questions are selected from the visual chain reasoning dataset to form a context example image set; the input target image and the global image of the context example image set are encoded using a visual encoder; the visual feature vector obtained by encoding is fused with the text query information of the first stage through the local Q-Former module to generate a local visual information representation; the local visual information representation is input into the large language model LLM to generate the ROI area of ​​the target image; the text query information of the first stage includes the target question, the prompt of the first stage, the question embedded in the sample image in the context example image set, and the ROI area corresponding to the question;

[0008] In the second stage, the ROI area and global image of the target image are encoded separately using the visual encoder, and the ROI area and global image of the context example image set are encoded separately; the visual feature vector obtained by encoding is fused with the second-stage text query information through the global Q-Former module to generate a comprehensive representation of the fusion of global and local visual features; the comprehensive representation of the fusion of global and local visual features is input into the large language model to output the final text answer; the second-stage text query information includes the target question, the second-stage prompt, the question embedded in the sample image in the context example image set, and the answer and ROI area corresponding to the question.

[0009] The local visual information representation is generated according to the following formula:

[0010] ;

[0011] In the above formula, Represents local visual information; It is a local Q-Former module; is the visual feature vector obtained by encoding the global image of the target image; A visual feature vector obtained by encoding the global image of the sample image in the context example image set; Query information for the first stage text; are the parameters in the local Q-Former module; for and Connection operation between;

[0012] Generate the ROI area of ​​the target image according to the following formula:

[0013] ;

[0014] In the above formula, is the ROI area of ​​the target image; For the large language model LLM; It is the first fully connected network; are the parameters of the first fully connected network.

[0015] A comprehensive representation that fuses global and local visual features is generated according to the following formula:

[0016] ;

[0017] In the above formula, A comprehensive representation that integrates global and local visual features; is the global Q-Former module; is the visual feature vector obtained by encoding the ROI area of ​​the target image; is the visual feature vector obtained by encoding the ROI area of ​​the sample image in the context example image set; Query information for the second stage text; are the parameters in the global Q-Former module;

[0018] The final text answer is generated according to the following formula:

[0019] ;

[0020] In the above formula, For the final text answer; is the second fully connected network; are the parameters of the second fully connected network.

[0021] The method of selecting sample images related to the target problem from the visual chain reasoning dataset to form a context example image set is as follows:

[0022] Calculate the semantic similarity between the question embedded in each sample image in the visual chain reasoning dataset and the target question, and select the top K sample images with the highest semantic similarity to form a context example image set. ; The calculation formula of the semantic similarity is:

[0023] ;

[0024] In the above formula, Indicates semantic similarity; represents the problem embedded in the i-th sample image in the visual chain reasoning dataset; Indicates the target problem; represents the L2 norm.

[0025] The parameters are optimized by minimizing the total loss function, which is calculated as follows:

[0026] ;

[0027] ;

[0028] ;

[0029] In the above formula, is the total loss function; 、 are the loss functions of the first and second stages respectively; 、 are the weights of the first and second stages respectively; represents the total number of target images used for training; express and The intersection-over-intersection ratio, 、 Represent the real ROI area and the predicted ROI area of ​​the i-th target image respectively; represents the true label of the i-th target image, indicating the correct text answer; represents the text answer probability distribution output by the large language model in the second stage for the i-th target image.

[0030] In a second aspect, the present invention provides a visual language reasoning system integrating contextual learning and chain reasoning, wherein the visual language reasoning system includes a first reasoning module and a second reasoning module;

[0031] The first reasoning module is used to construct a visual chain reasoning dataset, wherein the sample images of the visual chain reasoning dataset are embedded with questions, answers corresponding to the questions, and ROI regions; sample images related to the questions and the target questions are selected from the visual chain reasoning dataset to form a context example image set; the input target image and the global image of the context example image set are encoded using a visual encoder; the visual feature vector obtained by encoding is fused with the first-stage text query information through a local Q-Former module to generate a local visual information representation; the local visual information representation is input into a large language model (LLM) to generate the ROI region of the target image; the first-stage text query information includes the target question, the first-stage prompt, the question embedded in the sample image in the context example image set, and the ROI region corresponding to the question;

[0032] The second reasoning module is used to use a visual encoder to encode the ROI area and the global image of the target image respectively, and to encode the ROI area and the global image of the context example image set respectively; the visual feature vector obtained by encoding is fused with the second-stage text query information through the global Q-Former module to generate a comprehensive representation of the fusion of global and local visual features; the comprehensive representation of the fusion of global and local visual features is input into the large language model to output the final text answer; the second-stage text query information includes the target question, the second-stage prompt, the question embedded in the sample image in the context example image set, the answer corresponding to the question, and the ROI area.

[0033] The first reasoning module is used to generate a local visual information representation according to the following formula:

[0034] ;

[0035] In the above formula, Represents local visual information; It is a local Q-Former module; is the visual feature vector obtained by encoding the global image of the target image; A visual feature vector obtained by encoding the global image of the sample image in the context example image set; Query information for the first stage text; are the parameters in the local Q-Former module; for and Connection operation between;

[0036] The first reasoning module is further configured to generate an ROI region of the target image according to the following formula:

[0037] ;

[0038] In the above formula, is the ROI area of ​​the target image; For the large language model LLM; It is the first fully connected network; are the parameters of the first fully connected network.

[0039] The second reasoning module is used to generate a comprehensive representation that integrates global and local visual features according to the following formula:

[0040] ;

[0041] In the above formula, A comprehensive representation that integrates global and local visual features; is the global Q-Former module; is the visual feature vector obtained by encoding the ROI area of ​​the target image; is the visual feature vector obtained by encoding the ROI area of ​​the sample image in the context example image set; Query information for the second stage text; are the parameters in the global Q-Former module;

[0042] The second reasoning module is further configured to generate a final text answer according to the following formula:

[0043] ;

[0044] In the above formula, For the final text answer; is the second fully connected network; are the parameters of the second fully connected network.

[0045] The first reasoning module is configured to form a context example image set according to the following steps:

[0046] Calculate the semantic similarity between the question embedded in each sample image in the visual chain reasoning dataset and the target question, and select the top K sample images with the highest semantic similarity to form a context example image set. ; The calculation formula of the semantic similarity is:

[0047] ;

[0048] In the above formula, Indicates semantic similarity; represents the problem embedded in the i-th sample image in the visual chain reasoning dataset; Indicates the target problem; represents the L2 norm.

[0049] The visual language reasoning system further includes a model training module, which is used to optimize parameters by minimizing a total loss function. The calculation formula of the total loss function is:

[0050] ;

[0051] ;

[0052] ;

[0053] In the above formula, is the total loss function; 、 are the loss functions of the first and second stages respectively; 、 are the weights of the first and second stages respectively; represents the total number of target images used for training; express and The intersection-over-union ratio, 、 Represent the real ROI area and the predicted ROI area of ​​the i-th target image respectively; represents the true label of the i-th target image, indicating the correct text answer; represents the text answer probability distribution output by the large language model in the second stage for the i-th target image.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The visual language reasoning method described in the present invention integrates contextual learning into chain reasoning and performs multimodal reasoning in two stages, thus forming a new multimodal reasoning framework; in the first stage, the region of interest (ROI) of the target image is identified by combining context examples and target questions; in the second stage, the global features and ROI features of the target image, the global features and ROI features of the context examples, and the target question are combined to predict the final text answer; the above method realizes local feature extraction and the fusion of local features and global features, and combines context examples for staged visual language reasoning, which can improve the logical reasoning ability and robustness of visual language tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 Flowchart of the visual language reasoning method of the present invention.

[0057] Figure 2 This is a structural block diagram of the visual language reasoning system described in the present invention. DETAILED DESCRIPTION

[0058] The present invention will be further described in detail below with reference to specific embodiments and the accompanying drawings.

[0059] Example 1:

[0060] See also Figure 1 A visual-linguistic reasoning method that integrates contextual learning and chained reasoning (IC2T) is proposed. The method includes the following two stages:

[0061] The first stage is to build a visual chain reasoning dataset, in which the sample images of the visual chain reasoning dataset are embedded with questions, answers and ROI areas corresponding to the questions;

[0062] Select sample images related to the target question from the visual chain reasoning dataset to form a context example image set; specifically, use the DistilBERT model to calculate the semantic similarity between the question embedded in each sample image in the visual chain reasoning dataset and the target question, and select the top K sample images ranked by semantic similarity to form the context example image set , using the context example image set To assist in the extraction of fine-grained information; the semantic similarity is calculated by the cosine similarity function, and the calculation formula is:

[0063] ;

[0064] In the above formula, Indicates semantic similarity; represents the problem embedded in the i-th sample image in the visual chain reasoning dataset; Indicates the target problem; represents the L2 norm;

[0065] A visual encoder (e.g., SigLIP model) is used to encode the input target image and the global image of the context example image set. The visual feature vector obtained by encoding the global image of the target image and the visual feature vector obtained by encoding the global image of the sample image in the context example image set are fused with the text query information in the first stage through the local Q-Former module to generate a local visual information representation. The local visual information representation is generated according to the following formula:

[0066] ;

[0067] In the above formula, Represents local visual information; It is a local Q-Former module; is the visual feature vector obtained by encoding the global image of the target image; is a visual feature vector obtained by encoding the global image of the sample image in the context example image set; The first-stage text query information includes the target question, the target image placeholder, the first-stage prompt, a sample image from the context example image set, the question embedded in the sample image, and the ROI region placeholder corresponding to the question. The first-stage prompt is pre-set and, for example, can be set to: Please provide the bounding box coordinates of the ROI region that will help answer the question. are the parameters in the local Q-Former module; for and Connection operation between;

[0068] The local visual information representation is input into the large language model LLM through the first fully connected network to generate the ROI region of the target image; specifically, the ROI region of the target image is generated according to the following formula:

[0069] ;

[0070] In the above formula, is the ROI area of ​​the target image; For the large language model LLM; It is the first fully connected network; are the parameters of the first fully connected network;

[0071] In the second stage, the visual encoder is used to encode the ROI region and global image of the target image, and the ROI region and global image of the context example image set are also encoded.

[0072] The global Q-Former module combines the visual feature vectors obtained by encoding the ROI region of the target image and the global image, as well as the visual feature vectors obtained by encoding the ROI region of the sample image in the context example image set and the global image, with the second-stage text query information to generate a comprehensive representation of the fusion of global and local visual features. Specifically, the comprehensive representation of the fusion of global and local visual features is generated according to the following formula:

[0073] ;

[0074] In the above formula, A comprehensive representation that integrates global and local visual features; It is the global Q-Former module; is the visual feature vector obtained by encoding the ROI area of ​​the target image; is the visual feature vector obtained by encoding the ROI area of ​​the sample image in the context example image set; The second-stage text query information includes the target question, the target image and its ROI region placeholder, the second-stage prompt, the sample image in the context example image set, the question embedded in the sample image, the ROI region corresponding to the question, and the answer placeholder. The second-stage prompt is also pre-set and can be set as follows: Please answer the question based on the global image and ROI region. are the parameters in the global Q-Former module;

[0075] The comprehensive representation of the fusion of global and local visual features is input into the large language model (LLM) to output the final text answer. Specifically, the final text answer is generated according to the following formula:

[0076] ;

[0077] In the above formula, For the final text answer; is the second fully connected network; are the parameters of the second fully connected network.

[0078] The local Q-Former module and global Q-Former module in this invention are existing modules introduced in the paper "BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models." The Q-Former module generates learnable queries that actively participate in the attention mechanism and interact with the image or text query information to extract task-relevant visual feature vectors. The large language model (LLM) is fine-tuned using LoRA (Low-Rank Adaptation) to better adapt to the complex requirements of multimodal reasoning. Training is performed using the DeepSpeed ​​optimizer, optimizing parameters by minimizing the total loss function. These parameters include those of the local Q-Former module, the first fully connected network, the portion of the large language model related to ROI generation, the global Q-Former module, the second fully connected network, and the portion of the large language model related to answer reasoning. The total loss function is defined as:

[0079] ;

[0080] ;

[0081] ;

[0082] In the above formula, is the total loss function; is the loss function of the first stage, which is used to represent the IoU loss of ROI area prediction; is the loss function of the second stage, which is used to represent the cross entropy loss of the final text answer prediction; 、 are the weights of the first and second stages respectively; represents the total number of target images used for training; express and The intersection-over-union ratio, 、 Represent the real ROI area and the predicted ROI area of ​​the i-th target image respectively; represents the true label of the i-th target image, indicating the correct text answer; represents the text answer probability distribution output by the large language model in the second stage for the i-th target image.

[0083] The method described in the present invention uses contextual examples through a local Q-Former module and a large language model (LLM) in the first stage to determine the ROI area (i.e., the initial bounding box) related to the target question in the target image. In the second stage, the global Q-Former module is used to integrate the visual features of the global image and the ROI area to generate the final text answer. The above method uses contextual examples to assist visual language tasks, so that the model can better adapt to diverse visual language tasks. By efficiently fusing local visual information with global visual information, it can accurately locate the area in the image that is directly related to the task. The two-stage reasoning structure is gradually refined, thereby significantly reducing the risk of noise and error propagation in the reasoning process, and effectively improving the generalization ability and robustness of the model.

[0084] Performance Verification:

[0085] 1. Nine existing visual language models (VLMs) were selected for comparison with the proposed IC2T to verify its effectiveness. These models include LLaVA-1.5, Qwen-VL, InstructBLIP, LLaVA-Phi, MobileVLM, BLIP2, TinyLLaVA, LLaVA-1.5+CoS, and MMICL. LLaVA-1.5+CoS improves regional reasoning performance in complex scenes by introducing chained localization reasoning (CoS). MMICL, based on multimodal contextual learning, leverages multimodal contextual examples to enhance the model's understanding of complex tasks. To verify the versatility of the proposed IC2T across large multimodal language models of varying scales, IC2T / 7B and IC2T / 3.1B were built based on LLaVA-1.5 / 7B and the more lightweight TinyLLaVA / 3.1B, respectively.

[0086] Tests were conducted on VQA tasks and multimodal benchmarks respectively. For the VQA task, four datasets, VQA-v2, GQA, Flickr30k and TextVQA, were used; for the multimodal benchmark test, the MME, MMBench and POPE datasets were used. The above datasets cover basic object recognition, scene understanding and complex visual reasoning tasks, and can be used to evaluate multimodal alignment and reasoning capabilities. By combining the VQA task and the multimodal benchmark dataset, the adaptability of the IC2T proposed in this invention in a variety of scenarios can be comprehensively evaluated. Comprehensive scores were calculated for the MME and MMBench datasets, and the answer accuracy was calculated for other datasets. The test was performed on four NVIDIA A100 40GB GPUs, using the DeepSpeed ​​optimizer to minimize memory usage. The test results are shown in Table 1.

[0087] Table 1 Performance comparison of IC2T proposed in this paper and other visual language models

[0088] Model VQA-v2 GQA Flickr30k TextVQA MME MMBench POPE LLaVA-Phi 71.4 - - 48.6 1335.1 59.8 85.0 MobileVLM - 59.0 - 47.5 1288.9 59.6 84.9 TinyLLaVA 79.9 62.0 - 59.1 1464.9 66.9 86.4 LLaVA-1.5 78.5 62.0 58.1 58.2 1510.7 64.3 85.9 Qwen-VL 79.5 59.3 85.8 63.8 - 38.2 - InstructBLIP 63.7 49.5 82.8 50.7 1212.8 36.0 78.9 BLIP2 41.0 32.3 - 42.7 1293.8 - 85.3 LLaVA-1.5+CoS 80.7 63.7 - 60.9 1501.1 64.4 86.4 MMICL 70.56 - 88.79 - 1303.6 - - IC2T / 3.1B 81.0 62.9 83.6 64.9 1517.2 67.3 86.6 IC2T / 7B 81.3 64.1 89.1 66.2 1531.3 68.1 86.2

[0089] (1) Comparison on VQA tasks

[0090] Table 1 shows the performance comparison of the proposed IC2T and other visual language models on four datasets: VQA-v2, GQA, Flickr30k, and TextVQA. It can be seen that both IC2T / 7B and the lightweight IC2T / 3.1B show significant advantages. On the VQA-v2 dataset, IC2T / 7B outperforms LLaVA-1.5+CoS by approximately 1.28%, with an overall performance improvement of 1.9%. IC2T / 3.1B surpasses most baseline models, highlighting its high efficiency in reasoning tasks. On the Flickr50k dataset, IC27 / 7B demonstrates its excellent ability in fine-grained reasoning tasks, achieving a performance improvement of approximately 0.3% compared to MMCL. At the same time, IC2T / 3.1B scores 8.36, which is slightly lower than MMCL, but still demonstrates its robustness and efficiency as a lightweight model for multimodal tasks. On the GQA dataset, IC2T / 7B surpassed LLaVA-1.5 by 3.2% and LLaVA-1.5+CoS by 0.5%. IC2T / 3.1B also outperformed most baseline models, validating its effectiveness in complex reasoning scenarios. On the TextVQA dataset, IC2T / 7B achieved a 5.3% performance improvement over LLaVA-1.5 and a 3.8% improvement over TinyLLaVA, but was slightly lower than Overall. IC27 / 3.1B also performed well, only slightly behind IC27 / 7B. Overall, IC2T achieved significant performance improvements on multiple VQA datasets through multiple designs in multi-task reasoning and ICL, demonstrating the high performance of IC2T as a lightweight model suitable for a variety of application scenarios.

[0091] (2) Comparison on multimodal benchmarks

[0092] Table 1 shows the performance comparison of the proposed IC2T and other visual language models on four datasets: MME, MMBench, and POPE. It can be seen that IC2T / 7B performs well in most benchmark tests, and the lightweight IC27 / 3.1B also shows superiority. On the MME dataset, IC2T / 7B surpasses LLaVA-1.5+CoS by about 2.0% with a highest score of 153.13; at the same time, IC2T / 3.1B scores 1517.2, surpassing TinyLLaVA-3.1B by 3.5%, demonstrating its robustness in a lightweight configuration. On the MM-Bench dataset, IC2T / 7B surpasses LLaVA-1.5 by 5.9%, highlighting its robustness in handling multimodal tasks. On the POPE dataset, IC27 / 7B performed comparably to LLaVA-1.5+CoS, validating its effectiveness in generative reasoning tasks. IC2T / 3.1B outperformed other lightweight models, particularly on the MM-Bench dataset, surpassing TinyLLaVA-3.1B by 0.6%, further solidifying its superiority in lightweight scenarios. These results demonstrate the proposed IC2T's ability to integrate local and global multimodal information, thereby improving logical reasoning and robustness in visual-language tasks.

[0093] 2. To verify the impact of IC2T's cosine similarity algorithm on logical reasoning ability, we conducted tests on the VQA-v2 and GQA datasets using a random selection strategy for contextual examples and a strategy for removing contextual examples. The results are shown in Table 2.

[0094] Table 2 The impact of context examples selected by the cosine similarity algorithm on logical reasoning ability

[0095] Strategy VQA-v2 dataset GQA dataset Remove context example 78.2 61.3 Random Selection Strategy Selection Context Example 63.7 49.5 Cosine similarity algorithm selects context example 81.0 62.9

[0096] As can be seen from Table 2, when context learning is removed, the scores for complex tasks drop significantly, which means that context examples related to the target task are indispensable. Randomly selecting context examples to participate in IC2T can extract some context information, but due to the low semantic relevance to the target task, the support effect for reasoning tasks is limited. The scores on the VQA-v2 and GQA datasets are only 63.7 and 49.5, which are lower than the cosine similarity strategy; and randomly selecting context examples will introduce noise to interfere with the reasoning process, weakening the model's ability to focus on key information and perform coherent reasoning. The IC2T described in the present invention uses cosine similarity to screen samples to ensure that the selected samples can be aligned with the target task, thereby providing accurate context support. The above results show that high-quality context examples can help the model better understand task requirements, decompose complex problems and generate logical reasoning paths, thereby improving consistency and accuracy.

[0097] 3. To verify the impact of the fusion of local and global features in IC2T on logical reasoning ability, we tested and compared the effects of removing global and local features from IC2T on the VQA-v2 and GQA datasets. The test results are shown in Table 3.

[0098] Table 3 The impact of local feature and global feature fusion on logical reasoning ability

[0099] Strategy VQA-v2 dataset GQA dataset Remove global features 79.0 61.5 Remove local features 78.3 60.7 Fusion of local features and global features 81.0 62.9

[0100] Table 3 shows that model performance degrades when global features are removed (scoring is only 79.0 on the VQA-v2 and 61.5 on the GQA datasets). Removing local features (i.e., ROI regions) also degrades model performance (scoring is only 78.3 on the VQA-v2 and 60.7 on the GQA datasets), demonstrating the importance of both global and local features for scene understanding. These results demonstrate that integrating global and local features is crucial for achieving logical consistency and fine-grained multimodal reasoning.

[0101] Example 2:

[0102] See also Figure 2, a visual language reasoning system that integrates context learning and chain reasoning, the visual language reasoning system includes a first reasoning module, a second reasoning module, and a model training module; the first reasoning module is used to construct a visual chain reasoning dataset, wherein the sample images of the visual chain reasoning dataset are embedded with questions, answers corresponding to the questions, and ROI areas; sample images related to the questions and the target questions are selected from the visual chain reasoning dataset to form a context example image set; the input target image and the global image of the context example image set are encoded using a visual encoder, and the visual feature vector obtained by encoding is fused with the first-stage text query information through a local Q-Former module to generate a local visual information representation; the local visual information representation is input into a large language model (LLM) to generate the ROI area of ​​the target image; the first-stage text query information includes the target question, the first-stage prompt, the question embedded in the sample image in the context example image set, and the ROI area corresponding to the question;

[0103] Specifically, the first reasoning module is used to generate a local visual information representation according to the following formula:

[0104] ;

[0105] In the above formula, Represents local visual information; It is a local Q-Former module; is the visual feature vector obtained by encoding the global image of the target image; A visual feature vector obtained by encoding the global image of the sample image in the context example image set; Query information for the first stage text; are the parameters in the local Q-Former module; for and Connection operation between;

[0106] Specifically, the first reasoning module is further configured to generate an ROI region of the target image according to the following formula:

[0107] ;

[0108] In the above formula, is the ROI area of ​​the target image; For the large language model LLM; It is the first fully connected network; are the parameters of the first fully connected network;

[0109] Specifically, the first reasoning module is further used to form a context example image set according to the following steps: calculating the semantic similarity between the question embedded in each sample image in the visual chain reasoning dataset and the target question, selecting the top K sample images with the highest semantic similarity to form the context example image set ; The calculation formula of the semantic similarity is:

[0110] ;

[0111] In the above formula, Indicates semantic similarity; represents the problem embedded in the i-th sample image in the visual chain reasoning dataset; Indicates the target problem; represents the L2 norm;

[0112] The second reasoning module is used to use a visual encoder to encode the ROI region and the global image of the target image, and to encode the ROI region and the global image of the context example image set, respectively; the visual feature vector obtained by encoding is fused with the second-stage text query information through the global Q-Former module to generate a comprehensive representation of the fusion of global and local visual features; the comprehensive representation of the fusion of global and local visual features is input into the large language model to output the final text answer; the second-stage text query information includes the target question, the second-stage prompt, the question embedded in the sample image in the context example image set, and the answer and ROI region corresponding to the question;

[0113] Specifically, the second reasoning module is used to generate a comprehensive representation that integrates global and local visual features according to the following formula:

[0114] ;

[0115] In the above formula, A comprehensive representation that integrates global and local visual features; is the global Q-Former module; is the visual feature vector obtained by encoding the ROI area of ​​the target image; is the visual feature vector obtained by encoding the ROI area of ​​the sample image in the context example image set; Query information for the second stage text; are the parameters in the global Q-Former module;

[0116] Specifically, the second reasoning module is further configured to generate a final text answer according to the following formula:

[0117] ;

[0118] In the above formula, For the final text answer; is the second fully connected network; are the parameters of the second fully connected network;

[0119] The model training module is used to optimize model parameters by minimizing the total loss function, and the model parameters include: the calculation formula of the total loss function is:

[0120] ;

[0121] ;

[0122] ;

[0123] In the above formula, is the total loss function; 、 are the loss functions of the first and second stages respectively; 、 are the weights of the first and second stages respectively; represents the total number of target images used for training; express and The intersection-over-union ratio, 、 Represent the real ROI area and the predicted ROI area of ​​the i-th target image respectively; represents the true label of the i-th target image, indicating the correct text answer; represents the text answer probability distribution output by the large language model in the second stage for the i-th target image.

[0124] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A visual language reasoning method that integrates contextual learning and chain reasoning, characterized by: The visual language reasoning method includes the following two stages of visual language reasoning: The first stage is to build a visual chain reasoning dataset, in which the sample images of the visual chain reasoning dataset are embedded with questions, answers and ROI areas corresponding to the questions; Sample images related to the question and the target question are selected from the visual chain reasoning dataset to form a context example image set. The input target image and the global image of the context example image set are encoded using a visual encoder. The encoded visual feature vector is fused with the first-stage text query information through a local Q-Former module to generate a local visual information representation. The local visual information representation is input into the large language model (LLM) to generate the ROI region of the target image; the first-stage text query information includes the target question, the first-stage prompt, the question embedded in the sample image in the context example image set, and the ROI region corresponding to the question; In the second stage, the visual encoder is used to encode the ROI region and global image of the target image, and the ROI region and global image of the context example image set are also encoded. The encoded visual feature vector is fused with the second-stage text query information through the global Q-Former module to generate a comprehensive representation that integrates global and local visual features. The comprehensive representation that integrates global and local visual features is input into the large language model to output the final text answer. The second-stage text query information includes the target question, the second-stage prompt, the question embedded in the sample image in the context example image set, and the answer and ROI area corresponding to the question.

2. The visual language reasoning method integrating contextual learning and chain reasoning according to claim 1 is characterized by: The local visual information representation is generated according to the following formula: ； In the above formula, Represents local visual information; It is a local Q-Former module; is the visual feature vector obtained by encoding the global image of the target image; is a visual feature vector obtained by encoding the global image of the sample image in the context example image set; Query information for the first stage text; are the parameters in the local Q-Former module; for and Connection operation between; Generate the ROI area of ​​the target image according to the following formula: ； In the above formula, is the ROI area of ​​the target image; For the large language model LLM; It is the first fully connected network; are the parameters of the first fully connected network.

3. The visual language reasoning method integrating contextual learning and chain reasoning according to claim 2 is characterized by: A comprehensive representation that fuses global and local visual features is generated according to the following formula: ； In the above formula, A comprehensive representation that integrates global and local visual features; is the global Q-Former module; is the visual feature vector obtained by encoding the ROI area of ​​the target image; is the visual feature vector obtained by encoding the ROI area of ​​the sample image in the context example image set; Query information for the second stage text; are the parameters in the global Q-Former module; The final text answer is generated according to the following formula: ； In the above formula, For the final text answer; is the second fully connected network; are the parameters of the second fully connected network.

4. The visual language reasoning method integrating contextual learning and chain reasoning according to any one of claims 1 to 3, characterized in that: The method of selecting sample images related to the target problem from the visual chain reasoning dataset to form a context example image set is as follows: Calculate the semantic similarity between the question embedded in each sample image in the visual chain reasoning dataset and the target question, and select the top K sample images with the highest semantic similarity to form a context example image set. ; The calculation formula of the semantic similarity is: ； In the above formula, Indicates semantic similarity; represents the problem embedded in the i-th sample image in the visual chain reasoning dataset; Indicates the target problem; represents the L2 norm.

5. The visual language reasoning method integrating contextual learning and chain reasoning according to any one of claims 1 to 3, characterized in that: The parameters are optimized by minimizing the total loss function, which is calculated as follows: ； ； ； In the above formula, is the total loss function; 、 are the loss functions of the first and second stages respectively; 、 are the weights of the first and second stages respectively; represents the total number of target images used for training; express and The intersection-over-union ratio, 、 Represent the real ROI area and the predicted ROI area of ​​the i-th target image respectively; represents the true label of the i-th target image, indicating the correct text answer; represents the text answer probability distribution output by the large language model in the second stage for the i-th target image.

6. A visual language reasoning system that integrates contextual learning and chained reasoning, characterized by: The visual language reasoning system includes a first reasoning module and a second reasoning module; The first reasoning module is used to construct a visual chain reasoning dataset, wherein a question, an answer corresponding to the question, and an ROI region are embedded in a sample image of the visual chain reasoning dataset; Sample images related to the question and the target question are selected from the visual chain reasoning dataset to form a context example image set. The input target image and the global image of the context example image set are encoded using a visual encoder. The encoded visual feature vector is fused with the first-stage text query information through a local Q-Former module to generate a local visual information representation. The local visual information representation is input into the large language model (LLM) to generate the ROI region of the target image; the first-stage text query information includes the target question, the first-stage prompt, the question embedded in the sample image in the context example image set, and the ROI region corresponding to the question; The second reasoning module is used to encode the ROI region and the global image of the target image using a visual encoder, and to encode the ROI region and the global image of the context example image set respectively; The encoded visual feature vector is fused with the second-stage text query information through the global Q-Former module to generate a comprehensive representation that integrates global and local visual features. The comprehensive representation that integrates global and local visual features is input into the large language model to output the final text answer. The second-stage text query information includes the target question, the second-stage prompt, the question embedded in the sample image in the context example image set, and the answer and ROI area corresponding to the question.

7. The visual language reasoning system integrating contextual learning and chained reasoning according to claim 6 is characterized by: The first reasoning module is used to generate a local visual information representation according to the following formula: ； In the above formula, Represents local visual information; It is a local Q-Former module; is the visual feature vector obtained by encoding the global image of the target image; A visual feature vector obtained by encoding the global image of the sample image in the context example image set; Query information for the first stage text; are the parameters in the local Q-Former module; for and Connection operation between; The first reasoning module is further configured to generate an ROI region of the target image according to the following formula: ； In the above formula, is the ROI area of ​​the target image; For the large language model LLM; It is the first fully connected network; are the parameters of the first fully connected network.

8. The visual language reasoning system integrating contextual learning and chained reasoning according to claim 7 is characterized by: The second reasoning module is used to generate a comprehensive representation that integrates global and local visual features according to the following formula: ； In the above formula, A comprehensive representation that integrates global and local visual features; is the global Q-Former module; is the visual feature vector obtained by encoding the ROI area of ​​the target image; is the visual feature vector obtained by encoding the ROI area of ​​the sample image in the context example image set; Query information for the second stage text; are the parameters in the global Q-Former module; The second reasoning module is further configured to generate a final text answer according to the following formula: ； In the above formula, For the final text answer; is the second fully connected network; are the parameters of the second fully connected network.

9. The visual language reasoning system integrating contextual learning and chain reasoning according to any one of claims 6 to 8, characterized in that: The first reasoning module is configured to form a context example image set according to the following steps: Calculate the semantic similarity between the question embedded in each sample image in the visual chain reasoning dataset and the target question, and select the top K sample images with the highest semantic similarity to form a context example image set. ; The calculation formula of the semantic similarity is: ； In the above formula, Indicates semantic similarity; represents the problem embedded in the i-th sample image in the visual chain reasoning dataset; Indicates the target problem; represents the L2 norm.

10. The visual language reasoning system integrating contextual learning and chain reasoning according to any one of claims 6 to 8, characterized in that: The visual language reasoning system further includes a model training module, which is used to optimize parameters by minimizing a total loss function. The calculation formula of the total loss function is: ； ； ； In the above formula, is the total loss function; 、 are the loss functions of the first and second stages respectively; 、 are the weights of the first and second stages respectively; represents the total number of target images used for training; express and The intersection-over-union ratio, 、 Represent the real ROI area and the predicted ROI area of ​​the i-th target image respectively; represents the true label of the i-th target image, indicating the correct text answer; represents the text answer probability distribution output by the large language model in the second stage for the i-th target image.

Images