Method and system for improving first-person reasoning ability of visual language large model

Abstract

The invention discloses a method and a system for improving the first-person reasoning ability of a visual language large model, which adopt an active learning and causal intervention strategy, and realize the improvement of the understanding ability of the large model for a first-person image without additional training through the extraction, screening and directional injection of first-person knowledge. According to the method, an active learning-driven knowledge extraction mechanism is adopted, internal and external first-person knowledge is automatically extracted by using a large model, and a high-quality and high-correlation first-person knowledge set is obtained through a multi-dimensional screening strategy. On the basis, a causal intervention module based on knowledge guidance is designed, and by analyzing a reasoning path of a large model, injecting first person knowledge into a key decision node and introducing an answer screening module based on deterministic scoring, directional enhancement of the model reasoning ability is realized. The method can be effectively applied to various scenes such as an intelligent system and a human-computer interaction interface which need to be understood by a first person.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal first-person visual understanding, and specifically relates to a method and system for improving the first-person reasoning ability of a large visual language model, which can improve the large visual language model's ability to understand first-person perspective images. Background Art

[0002] With the rapid development of augmented reality and robotics, tasks centered around the first-person perspective (egocentric tasks) have become a research hotspot in computer vision and artificial intelligence. These tasks typically include object recognition and interaction detection from a first-person perspective, identifying user activities or behaviors from a first-person perspective, determining the user's position in the environment, and planning navigation paths. These first-person perspective-based vision tasks are of great value in application scenarios such as smart wearable devices, human-computer interaction, and service robotics.

[0003] Therefore, improving the ability of large-scale visual language models to understand visual information from a first-person perspective becomes particularly critical. Existing large-scale visual language models are already capable of extracting, understanding, and analyzing visual information from a third-person perspective. Therefore, the core challenge facing this task lies in effectively migrating the model from a third-person perspective to a first-person perspective using limited first-person data, and thereby achieving real-time perception and contextual understanding of dynamic environments.

[0004] Currently, researchers mainly use two strategies to improve the first-person perspective understanding ability of large-scale visual language models: 1) End-to-end pre-training based on first-person perspective datasets. This approach leverages large-scale first-person perspective datasets to pre-train a visual language model end-to-end. This allows the model to learn visual features, object relationships, and behavioral patterns directly from first-person perspective data. After pre-training, the model is transferred to downstream tasks through zero-shot learning, eliminating the need for additional task-specific fine-tuning.

[0005] 2) Introducing a lightweight adapter for fine-tuning. This approach fine-tunes the model by introducing a lightweight adapter module based on the pre-trained model to adapt to the unique characteristics of first-person perspective data. Adapters are typically parameter-efficient modules that can adapt the model's output to the new task without significantly increasing the number of model parameters. During fine-tuning, the main parameters of the pre-trained model remain unchanged, and only the parameters of the adapter module are updated.

[0006] While these methods have improved model performance in first-person perspective tasks to some extent, they still have significant limitations. First, existing research often overly focuses on behavioral analysis, neglecting the in-depth integration and utilization of environmental context. First-person perspective data typically contains rich scene information, such as spatial relationships between objects, dynamically changing environmental features, and user-environment interaction patterns. However, current methods fail to fully exploit this contextual information, limiting the model's understanding and reasoning capabilities in complex scenes. Second, due to the small size and data distribution bias of first-person perspective datasets, the model may learn biased knowledge during training, affecting its generalization ability. Furthermore, datasets may contain annotation bias, causing the model to overfit to the characteristics of a specific dataset during training and failing to transfer well to other scenarios or tasks. This means that even fine-tuned large models still have significant room for improvement in understanding first-person perspective images. Finally, both end-to-end pre-training strategies and methods that introduce lightweight adapters for fine-tuning consume significant computational resources and time costs.

[0007] In practical applications, egocentric tasks require large models to not only recognize objects in images, but also analyze the functions of objects in specific contexts and formulate corresponding action plans. For example, when given a first-person image of a driving scene and asked what the object in the model's hand is used for, the original model often generates a superficial response, such as "the object in the hand can be rotated." However, a more accurate response would be "the object in the hand is used to control the direction of the vehicle." This limitation stems from the model's failure to fully extract or utilize egocentric knowledge related to the scene, resulting in its inability to generate in-depth and accurate responses. This phenomenon shows that the lack of first-person perspective knowledge limits the ability of large visual language models to provide comprehensive and accurate answers in egocentric tasks.

[0008] In summary, building a general framework that improves the ability of large visual language models to understand first-person perspective images without additional training has important theoretical and practical value. This framework not only significantly expands the application scope of large visual language models, enabling them to play a vital role in more scenarios, but also provides more natural, accurate, and intelligent visual understanding capabilities for fields such as human-computer interaction, augmented reality, virtual reality, autonomous driving, medical assistance, and smart homes. Summary of the Invention

[0009] The purpose of this invention is to address the shortcomings of large-scale visual language models in understanding first-person perspective images by constructing a unified framework. This framework utilizes active learning strategies to acquire first-person perspective knowledge and integrates it into the reasoning process of the large-scale model through a causal intervention mechanism, thereby improving its understanding of first-person perspective images.

[0010] The technical solutions of the present invention are as follows: A method for improving the first-person reasoning capability of a large visual language model, comprising the following steps: Obtain visual features of the input image, encode and preprocess the input image, use the visual encoder to encode the image into visual features, and use the Grounding DINO model to select the interaction area; Constructing a first-person knowledge set, the first-person knowledge set including internal knowledge and external knowledge, the internal knowledge consisting of images of appropriate sizes selected from the interaction area and their corresponding description pairs, the description pairs generated by inputting scaled interaction area images into a large visual language model; the external knowledge consisting of three groups of images in the knowledge base whose visual features have the highest similarity with the visual features of the input image; Use causal models to model the reasoning process of the large model and inject first-person knowledge sets into the reasoning process of the large model; ask the same question to the large model multiple times, intervene in the reasoning process of the large model through intervention mechanisms, and collect the answers generated by the large model multiple times; Use the visual text encoder to encode all answers and input images, calculate the certainty scores of all generated answers, and select the answer with the highest score as the final output.

[0011] Specifically, it includes: Step 1: Encode and preprocess the input image, using a visual encoder to encode the image into visual features. Then, use the Grounding DINO model to detect and select the interaction area in the image. Step 2: Using an active learning strategy, the interaction regions extracted in Step 1 are scaled into three images of different sizes. These three scaled images are fed into the visual language model to generate corresponding image descriptions, thereby extracting the implicit internal knowledge of the images. Step 3: Build a knowledge base containing first-person perspective images and their corresponding description text. Use a visual encoder to encode the images in the knowledge base and generate a unified visual feature representation for subsequent retrieval and matching. Step 4: Evaluate the matching degree between the visual features in the knowledge base and the visual features of the input image by calculating the cosine similarity. Select the three image-description pairs with the highest similarity as external knowledge. Step 5: Integrate the internal knowledge extracted in step 2 and adaptively select an image of appropriate size and its corresponding description pair as the internal knowledge according to the position of the hand marker in the interaction area; Step 6: Integrate the external knowledge retrieved in step 4, remove redundant information, and filter out image-description pairs whose similarity with the input image exceeds a preset threshold. Combine the filtered external knowledge with the internal knowledge in step 5 to construct a first-person knowledge set; Step 7: Use the causal model to model the reasoning process of the large model and inject the first-person knowledge set into the reasoning process of the large model; Step 8: Ask the same question to the large model multiple times, intervene in the large model's reasoning process through the intervention mechanism in step 7, and collect the answers generated by the large model multiple times; Step 9: Use the encoder to encode all answers and the input image, calculate the certainty score of each answer, and select the answer with the highest score as the final output.

[0012] The present invention also relates to a system for improving the first-person reasoning capability of a large visual language model, comprising the following units: The acquisition unit is used to obtain the visual features of the input image, encode and preprocess the input image, encode the image into visual features using a visual encoder, and use the Grounding DINO model to select the interaction area; a first-person knowledge set construction unit, configured to construct a first-person knowledge set, the first-person knowledge set comprising internal knowledge and external knowledge, the internal knowledge consisting of images of appropriate sizes selected in the interaction area and their corresponding description pairs, and the external knowledge consisting of three groups of images having the highest similarity between visual features in the knowledge base and visual features of the input image; The reasoning unit is used to model the reasoning process of the large model using a causal model and inject the first-person knowledge set into the reasoning process of the large model. It asks the same question to the large model multiple times, intervenes in the reasoning process of the large model through the intervention mechanism, and collects the answers generated by the large model multiple times. The answer evaluation and screening unit is used to encode all answers and input images using a visual text encoder, calculate the certainty scores of all generated answers, and select the answer with the highest score as the final output.

[0013] The beneficial effects of the present invention are as follows: 1. This paper uses causal modeling to model the reasoning process of large-scale visual language models, effectively guiding the model to more accurately understand first-person images through causal intervention techniques. This method can significantly reduce the deviation in model reasoning, improve the semantic understanding and reasoning ability of image scenes, and show greater robustness in complex scenes. 2. This paper proposes an active learning-based strategy that enables large visual language models to autonomously and efficiently extract first-person knowledge. This method fully utilizes the model's inherent potential, significantly improving the model's understanding of the first-person perspective without relying on external models or additional annotated data, thereby reducing dependence on external resources. 3. For different input images, the present invention designs a dynamic knowledge acquisition and screening mechanism that can automatically select the most relevant images and text descriptions for the current scene and adapt to different first-person knowledge. This mechanism not only improves the flexibility of the method, but also enables its wide application in a variety of practical scenarios and strong adaptability. 4. The first-person image understanding enhancement method proposed in this paper can be seamlessly integrated into existing frameworks without requiring additional training or fine-tuning of existing large visual language models. This feature significantly reduces deployment costs while rapidly improving the model's understanding capabilities, making it highly practical and scalable. 5. Experimental verification shows that this invention can effectively improve the ability of various common visual language models to understand first-person images, including the precise recognition and prediction of the behavior of people in the image, the accurate inference of the location and purpose of objects, and the rational formulation of the next action plan based on the scenario. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments, wherein: Figure 1 Flowchart for the implementation of the present invention; Figure 2 A causal model for the reasoning process of the large visual language model constructed for the present invention; Figure 3 This is a flow chart of the method proposed in the present invention for improving the first-person reasoning capability of a large visual language model; Figure 4 This is a diagram showing the effect of the present invention performing first-person image reasoning on the EgoThink dataset; Figure 5 A comparison chart of first-person understanding effects on the EgoThink-Planning dataset; Figure 6 A comparison chart with two existing methods on the EgoThink dataset and two large visual language models. DETAILED DESCRIPTION

[0015] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. That is, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments.

[0016] like Figure 1 and Figure 3 As shown in the figure, a method based on causal intervention and active learning to improve the first-person comprehension ability of the visual language large model is proposed. First, the visual language large model is used to extract internal and external first-person knowledge according to the input image through active learning method. Then, the causal intervention mechanism is used to inject knowledge into the reasoning process of the large model to guide the large model to generate more accurate answers. Finally, the certainty score is used to screen out the final answer.

[0017] like Figure 2 As shown, the core of this embodiment is to use a causal intervention method to model and integrate it into the reasoning process of a large visual language model, guiding the large model to understand image information from a first-person perspective. This method uses filtered and useful first-person knowledge, including images and descriptions, to break down the causal relationship between the large model's input and answer into two parts, incorporating first-person knowledge for guidance: 1) the causal relationship between the input image and first-person knowledge; 2) the causal relationship between first-person knowledge and the answer. This method significantly reduces bias in the model's reasoning process and, by introducing first-person contextual knowledge, guides the model to provide more reasonable responses in first-person contexts, ultimately improving the model's first-person understanding capabilities.

[0018] A method for improving the first-person understanding of large visual language models based on causal intervention and active learning, which specifically includes the following steps: Step 1: Encode and preprocess the input image, use the visual encoder to encode the image into visual features, and use the Grounding DINO model to select the interaction area; Step 2: Using an active learning strategy, the interaction area obtained in step 1 is scaled into three images of different sizes. The three scaled images are input into the visual language model to generate corresponding image descriptions, thereby extracting the implicit internal knowledge of the image itself. Step 3: Build a knowledge base containing first-person perspective images and their corresponding description texts, and use a visual encoder to encode the images in the knowledge base to generate a unified visual feature representation; Step 4: Calculate the matching degree between the visual features in the knowledge base and the visual features of the input image through cosine similarity, and select the three groups of images with the highest similarity as external knowledge; Step 5: Integrate the internal knowledge extracted in step 2 and adaptively select an image of appropriate size and its corresponding description pair as the internal knowledge according to the position of the hand marker in the interaction area; Step 6: Integrate the external knowledge retrieved in step 4, remove redundant information, and filter out image-description pairs whose similarity with the input image exceeds a preset threshold. Combine the filtered external knowledge with the internal knowledge in step 5 to construct a first-person knowledge set; Step 7: Use the causal model to model the reasoning process of the large model and inject the first-person knowledge set into the reasoning process of the large model; Step 8: Ask the same question to the large model multiple times, intervene in the large model's reasoning process through the intervention mechanism in step 7, and collect the answers generated by the large model multiple times; Step 9: Use the encoder to encode all answers and the input image, calculate the certainty score of each answer, and select the answer with the highest score as the final output.

[0019] Flowchart of the method for improving the first-person reasoning ability of the visual language large model. First, based on the active learning strategy, the visual language large model is used to extract internal and external first-person knowledge from the input image. This step makes full use of the model's own reasoning ability without relying on external resources to ensure the comprehensiveness and accuracy of knowledge. Then, the extracted knowledge is injected into the reasoning process of the large model through the causal intervention mechanism. This mechanism can effectively eliminate bias in reasoning, guide the model to generate accurate answers that are more in line with the first-person perspective, and significantly improve the accuracy of semantic understanding and reasoning. Finally, the generated candidate answers are screened using the certainty score, and the answer with the highest confidence is selected as the final output. It mainly includes steps 1-9: Step 1: Select a test dataset, encode and preprocess the input image, use a visual encoder to encode the image into visual features, and use the Grounding DINO model to select the interaction area; This example uses the EgoThink first-person image dataset for experiments. The EgoThink dataset is used for first-person visual question answering and contains 700 annotated images extracted from the Ego4D dataset. Each image's annotation contains a question-answer pair. This benchmark is divided into six main categories and twelve subcategories, covering activities, objects, locations, prediction, reasoning, and planning.

[0020] Step 2: Using an active learning strategy, the interaction area obtained in step 1 is scaled into three images of different sizes. The three scaled images are input into the visual language model to generate corresponding image descriptions, thereby extracting the implicit internal knowledge of the image itself. The specific contents of this step are as follows: Filter based on whether the interactive area selected in step 1 contains hand information. First, filter based on whether the interactive area selected in step 1 contains hand information: if the image contains hand information, then based on the proportional relationship between the hand area and the overall image, crop the interactive area centered on the hand; if the image does not contain hand information, then crop the lower middle part of the image as the default interactive area. Then, based on the distance from the interactive area to the edge of the image, crop according to three proportional sizes (the size of the interactive area at one-quarter, one-third, and one-half of the distance from the interactive area to the edge of the image), which can be expressed by the following formula: In the formula represents the input image, represents the extracted interaction region, Represents three sizes of cropping methods, Represents the three cropped image description pairs that constitute the internal knowledge .

[0021] Step 3: Build a knowledge base containing first-person perspective images and their corresponding description texts, and use a visual encoder to encode the images in the knowledge base to generate a unified visual feature representation; Step 4: Calculate the matching degree between the visual features in the knowledge base and the visual features of the input image through cosine similarity, and select the three groups of images with the highest similarity as external knowledge; The specific contents of this step are as follows: Collect first-person images according to specific task requirements and combine them with the annotated descriptions or descriptions generated by the large model to form image description pairs. All images are encoded into visual features using a visual encoder, and their cosine similarity with the visual features of the input image is calculated. The k groups of image description pairs with the highest similarity are taken to form external knowledge. . It can be expressed by the following formula: In the formula represents the input image, Represents the image to be matched, S represents the use of cosine similarity to calculate the visual feature similarity for screening, Represents the k groups of image description pairs that are screened out, which constitute the external knowledge .

[0022] Step 5: Integrate the internal knowledge extracted in step 2 and adaptively select an image of appropriate size and its corresponding description pair as the internal knowledge according to the position of the hand marker in the interaction area; The specific contents of this step are as follows: According to the number of hands in the internal knowledge, a set of interaction regions and their corresponding description texts are adaptively selected as the final internal knowledge: if there are no hands, the largest interaction region is selected; if there is one hand, the medium-sized interaction region is selected; if there are two hands, the smallest interaction region is selected.

[0023] Step 6: Integrate the external knowledge retrieved in step 4, remove redundant information, and filter out image-description pairs whose similarity with the input image exceeds a preset threshold. Combine the filtered external knowledge with the internal knowledge in step 5 to construct a first-person knowledge set; The specific contents of this step are as follows: Calculate the cosine similarity between each image in the external knowledge and remove highly similar redundant images to ensure the diversity of external knowledge. On this basis, select the images with similarity to the input image higher than the preset threshold from the remaining external knowledge. The image-description pairs are combined with the filtered internal knowledge to form the first-person knowledge set K. The filtering process of steps 5 and 6 is expressed as: In the formula represents internal knowledge, represents external knowledge, Indicates the similarity threshold used when filtering external knowledge. represents the screening process of internal knowledge, Represents the use of cosine similarity to sort and filter external knowledge.

[0024] Step 7: Use the causal model to model the reasoning process of the large model and inject the first-person knowledge set into the reasoning process of the large model; The specific contents of this step are as follows: During the inference process of a large model, given an input X (including an input image I and an input question Q), the output answer A is affected by two factors: the causal relationship between X and A and the unknown confounding factor U in the dataset. The influence of the confounding factor U is represented by the "backdoor path" X←U→A, and is modeled using the "backdoor adjustment" formula: In the formula Indicates the causal relationship between X and A, also known as the do operation. Indicates that the input X and unknown factors A is generated under the influence of Indicates the influence of confounding factors.

[0025] To alleviate the problem of degraded first-person task performance in large models due to a lack of first-person data during training and fine-tuning, a "front-door adjustment" approach is used to introduce first-person knowledge K into the inference process. This approach does not rely on unknown confounding factors U to express the relationship between X and A. The formula is as follows: This formula decomposes the causal relationship between X and A into and Two parts, represents the process of extracting first-person knowledge K based on the input image X, It represents the process of first-person knowledge K guiding model reasoning and generating answer A. According to steps 1 to 6, the causal relationship between the input image X and the first-person knowledge K can be measured by the following formula: In the formula Represents the process of extracting internal knowledge, and the subscript i represents i types of cutting methods. Represents the process of retrieving external knowledge from the knowledge base. The extraction process of internal knowledge and external knowledge constitutes the causal relationship between knowledge k and input X.

[0026] Step 8: Ask the same question to the large model multiple times, intervene in the large model's reasoning process through the intervention mechanism in step 7, and collect the answers generated by the large model multiple times; The specific contents of this step are as follows: After generating first-person knowledge through the above active learning method, we use the contextual learning method to inject knowledge into the model reasoning process, using the following format: This is the image and corresponding description of the character interaction area: Here are images similar to the test image and their corresponding descriptions: 〉. After asking the model N questions, the answer set generated by the model is .

[0027] Step 9: Use the encoder to encode all answers and the input image, calculate the certainty score of each answer, and select the answer with the highest score as the final output.

[0028] The specific contents of this step are as follows: Use a visual text encoder to encode N answer texts and the input image as corresponding features. Calculate the cosine similarity between the text features and exclude answers that are significantly different from the others. Use the CLIP model to score the similarity between the text features and the image features, using this as the certainty score for each answer. The text with the highest similarity to the image is selected as the final answer. After steps 7 to 9, the causal relationship between the first-person knowledge K and the final output answer A can be measured using the following formula: In the formula represents the N answers generated under the guidance of knowledge K, Represents the process of calculating the certainty score for the answer and the input image.

[0029] The present invention also relates to a system for improving the first-person reasoning capability of a large visual language model, comprising the following units: The acquisition unit is used to obtain the visual features of the input image, encode and preprocess the input image, encode the image into visual features using a visual encoder, and use the Grounding DINO model to select the interaction area; a first-person knowledge set construction unit, configured to construct a first-person knowledge set, the first-person knowledge set comprising internal knowledge and external knowledge, the internal knowledge consisting of images of appropriate sizes selected in the interaction area and their corresponding description pairs, and the external knowledge consisting of three groups of images having the highest similarity between visual features in the knowledge base and visual features of the input image; The reasoning unit is used to model the reasoning process of the large model using a causal model and inject the first-person knowledge set into the reasoning process of the large model. It asks the same question to the large model multiple times, intervenes in the reasoning process of the large model through the intervention mechanism, and collects the answers generated by the large model multiple times. The answer evaluation and screening unit is used to encode all answers and input images using a visual text encoder, calculate the certainty scores of all generated answers, and select the answer with the highest score as the final output.

[0030] To test the model's ability to analyze images from a first-person perspective, we use accuracy to determine the visual language model's ability to understand first-person images. We use GPT-4 to evaluate the responses generated by the visual language model. Specifically, GPT-4 compares the generated responses with the standard answers and determines the degree of consistency between the two through a structured scoring system, assigning each image a score of 1 (perfect match), 0.5 (partial match), or 0 (no match). The accuracy is then expressed as the percentage of all images in that category.

[0031] Four scenario cases are given below to more intuitively demonstrate the effects and advantages of the present invention: The first scenario case: Using the aforementioned solution, 10 common multimodal models were tested on the five major categories of Object, Activity, Localization, Forecasting, and Reasoning on the EgoThink dataset. These dataset categories represent different types of image annotation problems, and can comprehensively test the model's first-person understanding ability from five aspects: objects, behaviors, locations, predictions, and reasoning. The answer to each question is a short phrase. There are a total of 600 first-person pictures and annotations in the five categories, and the pictures are all screenshots of the Ego4D video dataset. The comparison of first-person image understanding ability is shown in Table 1 Table 1 Comparison of first-person image understanding effects on the EgoThink dataset Second scenario case: Using the aforementioned solution, the model was tested on the Planning category of the EgoThink dataset and compared with the original model and the effect of adding only some first-person knowledge. The Planning category includes two subcategories, Navigation and Assistant, with a total of 100 images and corresponding annotations. Different from the category requirements in Table 1, this category requires the large model to analyze the surrounding environment and specify a streamlined action plan for the specified next action. The results of the experiment are shown in Figure 5 .

[0032] Case study of the third scenario: There are currently methods to improve the reasoning ability of large models from the perspective of alleviating large model hallucinations. However, these methods are still lacking in improving the ability to understand from the first perspective. Below, we select the currently commonly used methods for alleviating hallucination problems, VCD and OPERA, to compare with the method of this invention, and compare the effects of the three methods on the EgoThink dataset after adding them to the two models. The comparison results are shown in Figure 2. Figure 6 .

[0033] Case study of scenario 4: Using the above solution, the method proposed in this invention is used on the EgoThink dataset to improve the first-person perspective understanding ability of the visual language large model. Figure 4As shown, for a given image and question, whether in the Activity category that answers short phrases or in the Planning category that answers specific plans, the method proposed by the present invention effectively improves the visual language large model's ability to understand first-person perspective images, and gives more accurate and practical answers. In the case of Activity, the method of the present invention helps the large model recognize that the object in the image is an iron, and further derives the answer that clothes are being ironed. In the case of Planning, the method of the present invention injects current scene information into the large model, allowing the large model to change from answering the general process of buying soy sauce to concluding that the soy sauce is on the shelf and that it can be picked up based on image information.

[0034] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for improving the first-person reasoning capability of a large visual language model, characterized by: The method comprises the following steps: Obtain visual features of the input image, encode and preprocess the input image, use the visual encoder to encode the image into visual features, and use the Grounding DINO model to select the interaction area; Constructing a first-person knowledge set, the first-person knowledge set including internal knowledge and external knowledge, the internal knowledge consisting of images of appropriate sizes selected from the interaction area and their corresponding description pairs, the description pairs generated by inputting scaled interaction area images into a large visual language model; the external knowledge consisting of three groups of images in the knowledge base whose visual features have the highest similarity with the visual features of the input image; Use causal models to model the reasoning process of the large model and inject first-person knowledge sets into the reasoning process of the large model; ask the same question to the large model multiple times, intervene in the reasoning process of the large model through intervention mechanisms, and collect the answers generated by the large model multiple times; Use the visual text encoder to encode all answers and input images, calculate the certainty scores of all generated answers, and select the answer with the highest score as the final output.

2. The method for improving the first-person reasoning ability of a large visual language model according to claim 1, characterized in that: The method comprises the following steps: Step 1: Encode and preprocess the input image, use the visual encoder to encode the image into visual features, and use the Grounding DINO model to select the interaction area; Step 2: Using an active learning strategy, the interaction area obtained in step 1 is scaled into several images of different sizes. The scaled interaction area images are input into the visual language model to generate corresponding description pairs, thereby extracting the implicit internal knowledge of the image itself; Step 3: Build a knowledge base containing first-person perspective images and their corresponding description texts, and use a visual encoder to encode the images in the knowledge base to generate a unified visual feature representation; Step 4: Calculate the matching degree between the visual features in the knowledge base and the visual features of the input image through cosine similarity, and select the three groups of images with the highest similarity as external knowledge; Step 5: Integrate the internal knowledge extracted in step 2 and adaptively select an image of appropriate size and its corresponding description pair as the internal knowledge according to the position of the hand marker in the interaction area; Step 6: Integrate the external knowledge retrieved in step 4, remove redundant information, and filter out image-description pairs whose similarity with the input image exceeds a preset threshold; combine the filtered external knowledge with the internal knowledge in step 5 to construct a first-person knowledge set; Step 7: Use the causal model to model the reasoning process of the large model and inject the first-person knowledge set into the reasoning process of the large model; Step 8: Ask the same question to the large model multiple times, intervene in the large model's reasoning process through the intervention mechanism in step 7, and collect the answers generated by the large model multiple times; Step 9: Use the visual text encoder to encode all answers and the input image, calculate the certainty score of each answer, and select the answer with the highest score as the final output.

3. The method for improving the first-person reasoning ability of a large visual language model according to claim 2 is characterized in that: Step 2 includes the following steps in more detail: screening based on whether the interactive area selected in step 1 contains hand information; screening based on whether the interactive area selected in step 1 contains hand information: if the image contains hand information, cropping the interactive area centered on the hand based on the proportional relationship between the hand area and the overall image; if the image does not contain hand information, cropping the lower middle part of the image as the default interactive area; and cropping according to three proportional sizes based on the distance from the interactive area to the edge of the image, as expressed by the following formula: In the formula represents the input image, represents the extracted interaction region, Represents three sizes of cropping methods, Represents the three groups of cropped image description pairs, which constitute the internal knowledge .

4. The method for improving the first-person reasoning ability of a large visual language model according to claim 2, characterized in that: Step 4 includes the following steps in more detail: collecting first-person images according to specific task requirements, and combining them with the annotated descriptions or descriptions generated by the large model to form image description pairs; encoding all images into visual features using a visual encoder, calculating their cosine similarity with the visual features of the input image, and taking the k groups of image description pairs with the highest similarity to form external knowledge. ; Expressed as follows: In the formula represents the input image, Represents the image to be matched, S represents the use of cosine similarity to calculate the visual feature similarity for screening, Represents the k groups of image description pairs that are screened out, which constitute the external knowledge .

5. The method for improving the first-person reasoning ability of a large visual language model according to claim 2, characterized in that: Step 5 includes in more detail: according to the number of hands in the internal knowledge, adaptively selecting a set of interaction regions and their corresponding description texts as the final internal knowledge: if there are no hands, select the largest interaction region; if there is one hand, select the medium-sized interaction region; if there are two hands, select the smallest interaction region.

6. The method for improving the first-person reasoning ability of a large visual language model according to claim 2, characterized in that: Step 6 includes in more detail: calculating the cosine similarity between each image in the external knowledge and removing highly similar redundant images; on this basis, screening out the images with similarity to the input image higher than a preset threshold from the remaining external knowledge. The image-description pairs are combined with the filtered internal knowledge to form the first-person knowledge set K; the screening process of steps 5 and 6 is expressed as: In the formula represents internal knowledge, represents external knowledge, Represents the similarity threshold used when filtering external knowledge; represents the screening process of internal knowledge, Represents the use of cosine similarity to sort and filter external knowledge.

7. The method for improving the first-person reasoning ability of a large visual language model according to claim 2, characterized in that: Step 7 includes the following details: During the inference process of the large model, given an input X, including an input image I and an input question Q, the output answer A is affected by two factors: the causal relationship between X and A and the unknown confounding factor U in the dataset; the influence of the confounding factor U is represented by the "backdoor path" X←U→A, and is modeled by the "backdoor adjustment" formula: In the formula Indicates the causal relationship between X and A, also known as the do operation; Indicates that the input X and unknown factors A is generated under the influence of Indicates the influence of confounding factors; The "front-door adjustment" method introduces first-person knowledge K into the reasoning process. This method does not rely on unknown confounding factors U to express the relationship between X and A. The formula is as follows: This formula decomposes the causal relationship between X and A into and Two parts, namely the causal relationship between k and X and the causal relationship between A and k; represents the process of extracting first-person knowledge K based on the input image X, The process of first-person knowledge K guiding model reasoning and generating answer A. According to steps 1 to 6, the causal relationship between the input image X and the first-person knowledge K can be measured by the following formula: In the formula Represents the process of extracting internal knowledge, and the subscript i represents i types of cutting methods. Represents the process of retrieving external knowledge from the knowledge base; the extraction process of internal knowledge and external knowledge constitutes the causal relationship between knowledge k and input X.

8. The method for improving the first-person reasoning ability of a large visual language model according to claim 2, characterized in that: Step 8 includes in more detail: After generating first-person knowledge through the above active learning method, use the contextual learning method to inject knowledge into the model reasoning process, using the following format: This is the image and corresponding description of the character interaction area: 〉; Here are images similar to the test image and their corresponding descriptions: 〈 〉; After asking the model N questions, the answer set generated by the model is .

9. The method for improving the first-person reasoning ability of a large visual language model according to claim 2, characterized in that: Step 9 includes the following steps in more detail: using a visual text encoder to encode N answer texts and the input image as corresponding features; calculating the cosine similarity between text features and excluding answers that are significantly different from other answers; using the CLIP model to score the similarity between text features and image features as the certainty score of each answer, and selecting the text with the highest similarity to the image as the final answer; after steps 7 to 9, the causal relationship between the first-person knowledge K and the final output answer A can be measured using the following formula: In the formula Indicates generating N answers under the guidance of knowledge K, Represents the process of calculating the certainty score of the answer and the input image.

10. A system for improving the first-person reasoning ability of a large visual language model, characterized by: The system includes the following units: The acquisition unit is used to obtain the visual features of the input image, encode and preprocess the input image, encode the image into visual features using a visual encoder, and use the Grounding DINO model to select the interaction area; a first-person knowledge set construction unit, configured to construct a first-person knowledge set, the first-person knowledge set comprising internal knowledge and external knowledge, the internal knowledge consisting of images of appropriate sizes selected from the interaction area and their corresponding description pairs, the description pairs being generated by inputting scaled interaction area images into a large visual language model, and the external knowledge consisting of three groups of images whose visual features in the knowledge base have the highest similarity to those of the input image; The reasoning unit is used to model the reasoning process of the large model using a causal model and inject the first-person knowledge set into the reasoning process of the large model; Ask the same question to the big model multiple times, intervene in the reasoning process of the big model through the intervention mechanism, and collect the answers generated by the big model multiple times; The answer evaluation and screening unit is used to encode all answers and input images using a visual text encoder, calculate the certainty scores of all generated answers, and select the answer with the highest score as the final output.

Images