A visual scene-oriented counterfactual verification memory construction method and system

Abstract

The application provides a visual scene-oriented counterfactual verification memory construction method and system, which can obtain a target visual question and answer sample, including a target image and target question and answer information for the target image; input the target image and the target question and answer information into a visual intelligent agent for rule understanding, output candidate memory rules containing at least causal visual factors and non-causal interference factors; perform counterfactual image editing on the target image by changing one of the non-causal interference factors and the causal visual factors while keeping the other unchanged, respectively generate an interference edited image and a causal edited image, verify the interference edited image and the causal edited image, in the case of passing the verification, determine the candidate memory rules as target semantic memories, and determine the interference edited image as a target counterexample memory, and combine the target visual question and answer sample together to update a new memory item to a memory library of the visual intelligent agent, thereby improving the quality of the memory library.

Description

Technical Field

[0001] This disclosure relates to the field of visual question answering technology, and in particular to a method and system for constructing counterfactual verification memories for visual scenarios. Background Technology

[0002] With the development of large language models and visual language models, visual agents are widely used in multimodal scenarios such as visual question answering and image-text reasoning. In order to enable agents that have been running for a long time to gradually accumulate experience in continuous tasks, the industry usually sets up an external memory to store historical interaction and reasoning process fragments, and calls them in subsequent tasks through retrieval-augmented generation (RAG). Therefore, the quality of the memory has a crucial impact on the performance of the agent.

[0003] However, in related technologies, agents are prone to writing other factors that co-occur with the answer but do not determine the answer into the memory bank. These low-quality memories reappear repeatedly during retrieval, leading to shortcut amplification, which reduces the quality of the memory bank, the robustness of reasoning, and the reasoning performance of the agent. Summary of the Invention

[0004] The main objective of this disclosure is to propose a counterfactual verification memory construction method and system for visual scenarios, which can improve the quality of the memory bank, the robustness of reasoning, and the reasoning performance of the intelligent agent.

[0005] To achieve the above objectives, a first aspect of this disclosure proposes a method for constructing counterfactual verification memories for visual scenes, comprising: Obtain a target visual question-and-answer example for a visual intelligent agent, wherein the target visual question-and-answer example includes a target image and target question-and-answer information for the target image; The target image and the target question-and-answer information are input into the visual agent for rule understanding, and candidate memory rules are output. The candidate memory rules include at least causal visual factors and non-causal interference factors. By changing the non-causal interference factor while keeping the causal visual factor unchanged, and by changing the causal visual factor while keeping the non-causal interference factor unchanged, counterfactual image editing is performed on the target image to generate interference-edited images and causal-edited images, respectively. The interference-edited image and the causal-edited image are verified based on the target question-and-answer information and the visual agent to obtain the verification results; If the verification result characterization verification passes, the candidate memory rule is determined as the target semantic memory, and the interference editing image is determined as the target counterexample memory. The target visual question answering example, the target semantic memory, and the target counterexample memory are combined into a new memory item and updated into the memory bank of the visual agent.

[0006] In some embodiments, prior to obtaining the target visual question-answering example of the visual agent, the method further includes: Obtain a target image and a target question for the target image, and generate a target retrieval key containing question semantics and image description based on the target image and the target question; Retrieve each candidate memory item from a preset memory bank, encode and generate a memory key for each candidate memory item, calculate the retrieval score between the target retrieval key and each memory key, and select the target memory key from multiple memory keys based on the size of each retrieval score. Extract visual question-and-answer examples, semantic memories, and counterexample memories under the memory item corresponding to the target memory key, and form retrieval prompt information. Input the retrieval prompt information, the target image, and the target question into the visual agent to generate the target answer. The target question and answer information, composed of the target question and the target answer, and the target image constitute a target visual question and answer example.

[0007] In some embodiments, the candidate memory rules and the semantic memory further include inference rule description text, and the step of inputting the retrieval prompt information, the target image, and the target question into the visual agent to generate the target answer includes: The reasoning rule description text, causal visual factors, and non-causal interference factors corresponding to the semantic memory in the retrieval prompt information, as well as the counterexample memory and the visual question-and-answer sample, are structured and organized, and input into the visual agent along with the target image and the target question. Based on the verified reasoning rule description text, learning the causal visual factors, avoiding the warnings of the non-causal interference factors, and learning the historical interaction references in the counterexample memory, multimodal reasoning is performed to generate the target answer.

[0008] In some embodiments, the target question-and-answer information includes a target question and a target answer; the verification of the interference-edited image and the causal-edited image based on the target question-and-answer information and the visual agent to obtain a verification result includes: Based on the target image, the fidelity of the interference-edited image and the causal-edited image is verified, a fidelity weight is generated, and fidelity filtering is performed on the interference-edited image and the causal-edited image based on the fidelity weight; The target question is input into the visual agent along with the interference-edited image and the causal-edited image after fidelity filtering, respectively, to generate the corresponding interference-edited answer and causal-edited answer. The interference invariance score is determined based on the fidelity weight, the target answer, and the interference edited answer after fidelity filtering; and the causal sensitivity score is determined based on the fidelity weight, the target answer, and the causal edited answer after fidelity filtering. The verification results are obtained based on the relationship between the interference invariance score and the preset first threshold, and the causal sensitivity score and the preset second threshold.

[0009] In some embodiments, determining the interference invariance score based on the fidelity weight, the target answer, and the interference-edited answer after fidelity filtering, and determining the causal sensitivity score based on the fidelity weight, the target answer, and the causal-edited answer after fidelity filtering, includes: For each of the interference-edited images that has completed fidelity filtering, it is determined whether the corresponding interference-edited answer is equivalent to the target answer to obtain a first determination result. The first determination result is weighted and summed using the fidelity weight corresponding to each image as a coefficient. The summation result is divided by the sum of the fidelity weights of all valid interference-edited images to obtain the interference invariance score. For each causal edited image that has completed fidelity filtering, it is determined whether the corresponding causal edit answer is equivalent to the target answer to obtain a second determination result. After inverting the second determination result, a weighted sum is performed based on the fidelity weight of each image. The sum is then divided by the sum of the fidelity weights of all valid causal edited images to obtain the causal sensitivity score.

[0010] In some embodiments, the candidate memory rule further includes a scope description text; the step of obtaining the verification result based on the relationship between the interference invariance score and a preset first threshold, and the causal sensitivity score and a preset second threshold, further includes: Based on the differences between the scope description text in the candidate memory rule and the scope description text under each memory item in the memory bank, the conflict score of the candidate memory rule is determined. The verification results are obtained based on the relationship between the interference invariance score and the preset first threshold, the causal sensitivity score and the preset second threshold, and the conflict score and the preset third threshold.

[0011] In some embodiments, after combining the target visual question-answering example, the target semantic memory, and the target counterexample memory into a new memory item and updating it in the memory bank of the visual agent, the method further includes: Under the preset prompt word budget constraint, the retention value of each memory item in the memory bank is calculated. The retention value is obtained by summing the replay benefit, reliability confidence lower bound, and applicable scope coverage benefit of the memory item, and then deducting the redundancy penalty and conflict penalty. Greedy pruning is performed based on the retention values ​​of each memory item, and conflicting memory items are either removed or put into a dormant state.

[0012] To achieve the above objectives, a second aspect of this disclosure proposes a counterfactual verification memory construction system for visual scenes, comprising: The question-and-answer sample acquisition module is used to acquire target visual question-and-answer samples of the visual intelligent agent. The target visual question-and-answer samples include target images and target question-and-answer information for the target images. The candidate rule generation module is used to input the target image and the target question-and-answer information into the visual agent for rule understanding and output candidate memory rules, wherein the candidate memory rules include at least causal visual factors and non-causal interference factors. The counterfactual editing module is used to perform counterfactual image editing on the target image when the non-causal interference factor is changed while the causal visual factor remains unchanged, and when the causal visual factor is changed while the non-causal interference factor remains unchanged, to generate interference-edited images and causal-edited images respectively. The counterfactual verification module is used to verify the interference-edited image and the causal-edited image based on the target question-and-answer information and the visual agent, and obtain the verification result; The memory construction module is used to determine the candidate memory rule as the target semantic memory and the interference editing image as the target negative example memory when the verification result characterization verification is passed, and to combine the target visual question answering example, the target semantic memory and the target negative example memory into a new memory item and update it into the memory bank of the visual agent.

[0013] To achieve the above objectives, a third aspect of this disclosure provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the counterfactual verification memory construction method for visual scenes described in the first aspect embodiment above.

[0014] To achieve the above objectives, a fourth aspect of this disclosure provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the counterfactual verification memory construction method for visual scenes described in the first aspect embodiment.

[0015] The beneficial effects of the embodiments disclosed herein include: By inputting the target image and target question-and-answer information into a visual agent to perform rule understanding, the output is a candidate memory rule containing both causal visual factors and non-causal interference factors. First, the causal attributes of various factors affecting the question-and-answer results are decomposed and defined. Then, using a counterfactual editing approach with controlled variables, two sets of control samples are constructed: one generating interference-edited images by changing non-causal interference factors while keeping causal visual factors unchanged, and the other generating causal-edited images by changing causal visual factors while keeping non-causal interference factors unchanged. These samples can be used to verify the causal validity of the memory rules. Finally, verification is completed based on the two types of counterfactual edited images combined with the target question-and-answer information. Only when the verification is successful and the answer of the candidate memory rule is confirmed to be determined by causal visual factors rather than co-occurring interference factors, is the corresponding target semantic memory and target counterexample memory combined into a new memory item and updated to the memory bank. This rigorous screening from the source of memory entry eliminates low-quality pseudo-memory methods based solely on superficial co-occurrence associations, fundamentally avoids the shortcut amplification problem, and effectively improves the overall quality of the memory bank. Ultimately, this ensures that the agent's subsequent retrieval and calls are all high-quality memories that have been causally verified, and will not trigger erroneous reasoning due to the accidental occurrence of non-causal interference factors, thereby significantly improving the robustness of reasoning and achieving the technical effect of improving the reasoning performance of visual agents. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of an application environment for the counterfactual verification memory construction method for visual scenes provided in this embodiment of the disclosure; Figure 2 This is a flowchart illustrating the counterfactual verification memory construction method for visual scenarios provided in this embodiment of the disclosure; Figure 3 yes Figure 2 A flowchart illustrating the process preceding step S101; Figure 4 yes Figure 2 A flowchart further includes step S104; Figure 5 yes Figure 4 A flowchart further included in step S303; Figure 6 yes Figure 4 A flowchart further included in step S304; Figure 7 yes Figure 2 A flowchart illustrating the further steps following step S105; Figure 8 This is a schematic diagram of the functional modules of the counterfactual verification memory construction system for visual scenes provided in this embodiment of the disclosure; Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this disclosure. Detailed Implementation

[0017] The accompanying drawings in the embodiments clearly and completely describe the technical solutions in the embodiments of this disclosure. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0018] It is understood that in the specific embodiments of this disclosure, visual question-and-answer examples, images, questions and related data are involved. When the above embodiments of this disclosure are applied to specific products or technologies, permission or consent can be obtained from the subject, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards.

[0019] Furthermore, when the embodiments of this disclosure require visual question-and-answer examples, images, questions, and related data, separate permission or separate consent can be obtained for the initial timing data, initial sample timing data, and related data through pop-up windows or by jumping to a confirmation page. After clearly obtaining separate permission or separate consent for the initial timing data, initial sample timing data, and related data, the necessary initial timing data, initial sample timing data, and related data for enabling the embodiments of this disclosure to operate normally can then be obtained.

[0020] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0021] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0022] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0023] Visual agents are artificial intelligence systems capable of receiving images and natural language questions, and outputting answers or reasoning. They can be built based on visual language models (VLMs), and therefore can also be referred to as agents, models, or large models in this embodiment. The parameters of the visual agents in this embodiment are frozen, and gradient updates are not performed.

[0024] A Visual-Language Model (VLM) is a multimodal artificial intelligence system capable of simultaneously understanding images / videos and text. It enables AI to interpret images and reason based on them by establishing cross-modal alignment between visual signals and linguistic semantics. Essentially, VLM combines the reasoning capabilities of a Large Language Model (LLM) with a visual encoder, allowing the model not only to see pixels but also to describe, explain, or answer questions about images using natural language.

[0025] External memory is an external storage structure used to store historical interactions, reasoning rules, and counterexamples. It further includes three types: episodic memory, semantic memory, and counterexample memory.

[0026] Causal visual factors refer to the core visual evidence that truly determines the answer in a visual question-and-answer task, such as the color, shape, spatial relationships, quantity, and text area of ​​an object.

[0027] Non-causal interfering factors refer to visual features that co-occur with the answer but do not determine the answer, such as background, lighting, style, layout, perspective, or co-occurring objects, which can easily cause spurious correlations.

[0028] Counterfactual image editing uses generative image editing models to modify the original image while maintaining or changing specific visual factors, generating control samples to verify causal relationships.

[0029] The interference invariance score measures the robustness of candidate rules to changes in non-causal interference factors, i.e., the proportion of answers that remain unchanged after interference is edited.

[0030] The causal sensitivity score measures the sensitivity of candidate rules to changes in causal visual factors, i.e., the proportion of answers that change after causal editing.

[0031] Visual intelligent agents need to continuously accumulate experience during long-term operation. However, traditional memory storage methods based on appearance similarity are prone to incorporating non-causal interference factors such as background and style into memory, leading to shortcut amplification and reduced reasoning robustness. To address this, embodiments of this disclosure provide a counterfactual verification memory construction method and system for visual scenes. Counterfactual verification acts as the gatekeeper for memory writing, ensuring that only rules that pass causal testing can enter long-term semantic memory.

[0032] Please see Figure 1 , Figure 1 A schematic diagram of the implementation environment for the counterfactual verification memory construction method for visual scenes provided in this embodiment of the disclosure includes: a terminal 11 and a server 12.

[0033] For example, server 12 can obtain target visual question-and-answer samples from terminal 11, the target visual question-and-answer samples including target images and target question-and-answer information for the target images; input the target images and target question-and-answer information into the visual agent for rule understanding, and output candidate memory rules, the candidate memory rules including at least causal visual factors and non-causal interference factors; perform counterfactual image editing on the target images by changing non-causal interference factors while keeping causal visual factors unchanged, and by changing causal visual factors while keeping non-causal interference factors unchanged, generating interference-edited images and causal-edited images respectively; verify the interference-edited images and causal-edited images based on the target question-and-answer information and the visual agent, and obtain verification results; if the verification results indicate that the verification is successful, determine the candidate memory rules as target semantic memories and the interference-edited images as target counterexample memories, and combine the target visual question-and-answer samples, target semantic memories and target counterexample memories into new memory items and update them in the memory bank of the visual agent.

[0034] Terminal 11 can be a mobile phone, computer, smart voice interaction device, smart wearable device, smart home appliance, vehicle terminal, etc., but is not limited to these. Terminal 11 can also independently execute the counterfactual verification memory construction method for visual scenes. Terminal 11 and server 12 can be directly or indirectly connected through wired or wireless communication, and this embodiment of the disclosure does not impose any limitations.

[0035] Server 12 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Additionally, server 12 can also be a node server in a blockchain network.

[0036] It should be noted that, Figure 1 The schematic diagram of the implementation environment shown is merely an example. The scenarios described in this disclosure are intended to more clearly illustrate the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided in this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in this disclosure are also applicable to similar technical problems.

[0037] Please see Figure 2 , Figure 2 This is a flowchart illustrating a method for constructing counterfactual verification memories for visual scenes, provided in an embodiment of this disclosure. This method can be applied to the server in the above embodiments, or executed jointly by a terminal and a server. The method includes steps S101 to S105: Step S101: Obtain target visual question-and-answer samples for the visual agent. The target visual question-and-answer samples include target images and target question-and-answer information for the target images. Step S102: Input the target image and target question-and-answer information into the visual agent for rule understanding, and output candidate memory rules. The candidate memory rules contain at least causal visual factors and non-causal interference factors. Step S103: When changing non-causal interference factors and keeping causal visual factors unchanged, and when changing causal visual factors and keeping non-causal interference factors unchanged, counterfactual image editing is performed on the target image to generate interference-edited images and causal-edited images respectively. Step S104: Verify the interference-edited image and the causal-edited image based on the target question-answering information and the visual agent to obtain the verification result; In step S105, if the verification result characterization verification passes, the candidate memory rule is determined as the target semantic memory, and the interference editing image is determined as the target counterexample memory. The target visual question answering example, the target semantic memory, and the target counterexample memory are combined into a new memory item and updated to the memory bank of the visual agent.

[0038] Regarding step S101 above, in the offline phase of the visual agent, this embodiment can acquire target visual question-and-answer examples for the visual agent. These examples include a target image and target question-and-answer information for that image. Specifically, a target visual question-and-answer example refers to a complete interaction record completed by the visual agent during the wake-up phase, including the original input target image I and target question Q, as well as the agent's output target answer A and reasoning basis ρ. The target question-and-answer information includes the target question Q and target answer A, which are the core content of this interaction. The purpose of acquiring these examples is to provide real and traceable reasoning cases for the offline processing phase, enabling the extraction of candidate memory rules. These examples originate from the visual agent's execution records in actual tasks and can reflect the agent's current reasoning ability and potential dependency biases.

[0039] It should be noted that, by obtaining complete target visual question-and-answer examples, the original interactive behavior of the intelligent agent is used as material for subsequent rule generation, so that the rule extraction does not rely on manual annotation, but is based on the successful experience of the intelligent agent itself, thus realizing the autonomous accumulation of experience.

[0040] Regarding step S102 above, this embodiment of the disclosure can input the target image and target question-and-answer information into a visual agent for rule understanding, and output candidate memory rules. The candidate memory rules at least include causal visual factors and non-causal interference factors. Here, the candidate memory rule refers to a structured reasoning hypothesis extracted from an interaction record, i.e., the current target visual question-and-answer example, and its form is h = ( , (r, scope), where causal visual factors The core visual evidence that truly determines the answer, such as the color area of ​​the hat or the left-right positional relationship between two objects, is not a causal confounding factor. The environmental features that co-occur with the answer but are not necessary conditions, such as a beach background or shooting angle. The inference rule description text r is a brief natural language description of the rule, such as checking the color of the target object itself when answering a color question. The scope description text scope limits the type of task or visual reasoning skill to which the rule applies, such as object attribute recognition or spatial relationship judgment.

[0041] In some embodiments, the candidate memory rule h may not contain the inference rule description text r and the scope description text scope. Subsequently, when similar inference rule description text r and scope description text scope are needed, they can be based on the available causal visual factors. Non-causal interference factors Therefore, this disclosure does not impose specific limitations on the embodiments thereof.

[0042] Furthermore, the reason why the target visual question-answering example needs to include the original input target image I and target question Q, as well as the agent's output target answer A and reasoning basis ρ, is that this information together constitutes a complete and traceable causal reasoning chain. The target image I and target question Q are the original inputs to the wake-up phase task, serving as the foundational material for subsequent counterfactual editing and re-answering; without them, the task scenario cannot be reproduced or altered. The target answer A, as the agent's response, is not only a result signal for evaluating the correctness of reasoning but also a reference benchmark for judging whether the edited answer is equivalent to the original answer in counterfactual verification, and is the core basis for calculating the interference invariance score and causal sensitivity score. The reasoning basis ρ records the visual evidence and reasoning path relied upon by the agent when arriving at the answer, which is crucial for candidate rule generation in the offline phase. By analyzing ρ, the system can accurately locate which visual factors are the true causal evidence determining the answer. Which are the distracting factors that co-occur with the answer but are negligible? This allows for the extraction of candidate memory rules with clear causal hypotheses, avoiding blind guessing based solely on answers. Therefore, recording complete input, output, and reasoning triples provides ample and interpretable context for offline memory organization and counterfactual verification.

[0043] Therefore, visual agents can automatically decompose causal visual factors and non-causal interference factors by analyzing which regions in the image are strongly correlated with the answer and which environmental features may constitute spurious correlations. Only by distinguishing between the two types of factors can the invariance and sensitivity of the rules be tested separately.

[0044] It should be noted that the embodiments of this disclosure explicitly decompose candidate memory rules into causal factors and interfering factors, providing operable objects for counterfactual intervention, making the verification process have a clear experimental design intent, and avoiding a general evaluation of the right or wrong of the rules.

[0045] Regarding step S103 above, embodiments of this disclosure can perform counterfactual image editing on the target image by changing non-causal interference factors while keeping causal visual factors unchanged, and by changing causal visual factors while keeping non-causal interference factors unchanged, respectively generating interference-edited images and causal-edited images. Counterfactual image editing refers to calling a generative image editing model G to modify the original image I according to specified editing instructions, generating hypothetical images that may appear without changing the true causal relationship. Furthermore, the generative image editing model G can also be a visual agent V, or other models can be used as the subject of counterfactual image editing; this disclosure does not impose specific limitations on this.

[0046] The image being edited by interference is one of the images in the family of images that are edited by interference. By changing non-causal interference factors If the background is changed from a beach to a grassy area, the direction of the light is adjusted, and the shooting angle is changed, while strictly maintaining the causal visual factor z+, multiple interference edit images can be generated for the same target image to form an interference edit image. However, considering the computational resource cost, only one can be generated. A causal edit image is one image in a family of causal edit images. By changing or removing causal visual factors Examples of such actions include changing the color of a hat from red to blue, removing an object from the frame, or altering the relative positions of two objects, while strictly maintaining non-causal interference factors. For the same target image, multiple causal edited images can be generated without changing the original image, forming a family of causal edited images. Alternatively, considering computational resource costs, there can be only one family of images. These two families of images constitute a control experiment for causality testing. The former tests whether the rule is robust to noise variations, while the latter tests whether the rule truly depends on the claimed evidence. If the rule is a correct causal law, then perturbation editing should not change the answer, while causal editing should change the answer.

[0047] It should be noted that the embodiments of this disclosure simulate the experimental design of the controlled variable method by constructing two complementary families of edited images, so that the candidate rule must exhibit the expected behavior under interventions in two opposite directions in order to pass the verification, thereby effectively distinguishing between true causal relationships and spurious correlations.

[0048] Regarding step S104 above, this embodiment of the disclosure can verify the interfering edited image and the causal edited image based on the target question-and-answer information and the visual agent, and obtain the verification result. Verification refers to operations such as using a parameter-frozen visual agent V to re-answer the edited image, and quantifying the causal validity of the candidate rules based on the change in the answer. Specifically, for the interfering edited image, the expected answer is to remain unchanged; for the causal edited image, the expected answer is to change.

[0049] Furthermore, the verification process involves calculating the interference invariance score Inv(h) and the causal sensitivity score Nec(h) and comparing them with a preset threshold to determine whether the rule passes the verification. This process does not use real labels but uses the original answer A as a reference, so no additional labeled data is required.

[0050] Besides comparing the interference invariance score Inv(h) and the causal sensitivity score Nec(h) with a threshold, other verification methods such as statistical hypothesis testing or Bayesian inference can be used. For example, paired t-tests or McNemar tests can be used to compare the significance of the difference in answer distribution between the interference-edited image set and the original image set, obtaining a p-value. If the p-value is greater than the significance level, such as 0.05, the invariance hypothesis is accepted. Similarly, the significance of answer changes can be tested on the causal-edited image set; if the p-value is less than the significance level, the sensitivity hypothesis is accepted. Another approach is evidence assessment based on Bayesian factors, which calculates the prior probability ratio of the observed current answer distribution under the hypotheses of a true and a false causal rule. When the Bayesian factor exceeds a preset threshold, such as 10, the rule is considered validated. These alternative methods can quantify the confidence of the verification results and provide a more flexible decision boundary than fixed thresholds, but the computational complexity is relatively high, making them suitable for high-risk application scenarios with extremely low tolerance for false positives.

[0051] It should be noted that, by having the same frozen visual agent respond again on the edited image, the present disclosure eliminates the interference of model parameter changes on the results, making the verification results depend entirely on the causal quality of the rules themselves, thus ensuring the fairness of the test.

[0052] Regarding step S105 above, this embodiment of the disclosure can, if the verification result characterization verification passes, determine the candidate memory rule as the target semantic memory and the interference edited image as the target negative example memory, and combine the target visual question-answering example, target semantic memory, and target negative example memory into a new memory item and update it in the visual agent's memory bank. Here, target semantic memory refers to inference rules that can be stored long-term after passing counterfactual verification; it enters the semantic memory bank S for retrieval and prompting in future tasks. Target negative examples (Hard Negatives), also known as factual memories, refer to successfully verified interference edited images. Although the answer remains unchanged, these images demonstrate the diversity of interference factors such as background and style; they are stored in the negative example memory bank H as a shortcut warning to remind the model not to rely on these interference factors during future retrievals. The original interaction records, i.e. target visual question-and-answer examples, will serve as plot memory, semantic memory as rules, and counterexample memory as counterexamples. The three will be stored together to form a complete experience loop. The reason for saving the three types of memory at the same time is that plot memory preserves the integrity of the original case, semantic memory provides reusable reasoning rules, and counterexample memory strengthens the robustness of the rules to interference factors through negative examples.

[0053] It should be noted that the terms "counterexample memory" and "fact memory" in this disclosure are essentially the same concept, both referring to edited images that have passed the interference invariance verification. They can also be called "fact memory" because although these images change non-causal interference factors such as background and lighting, the answer remains unchanged. This fact itself constitutes strong supporting evidence for the causal validity of the rule; that is, it concretely proves that the answer is determined by causal visual factors, not interference factors. Unlike the term "counterexample memory," which emphasizes negative examples and warnings against shortcuts, "fact memory" emphasizes the positive verification facts carried by these samples. They are experimentally tested real cases that can corroborate the robustness of the rule. The two naming methods describe the essence of the same type of memory item from different perspectives. Counterexamples serve as warnings against false correlations, while facts provide factual evidence for verifying causal invariance. In actual system implementation, this type of memory plays a dual role of positive support and negative warning.

[0054] It should be noted that the embodiments disclosed herein simultaneously write verified rules and counterexamples into memory, which tells the model what causal factors it should pay attention to, and also reminds the model not to be affected by which interfering factors, forming a mechanism that combines positive guidance and negative warning, which greatly improves the practical value of memory.

[0055] Furthermore, this embodiment of the disclosure distinguishes between the wake-up phase and the offline organization phase, primarily based on considerations of computational resource efficiency and the differences in task nature. The wake-up phase corresponds to the agent's online reasoning process, requiring real-time response to new visual question-answering tasks. Therefore, it must ensure low latency and high throughput, and can only perform lightweight memory retrieval and single forward inference, unable to tolerate high-overhead operations such as counterfactual editing and multiple model inferences. The offline organization phase occurs during system idle periods or after task batches, when computational resources are sufficient to perform computationally intensive operations such as image editing, fidelity verification, and repeated question-answering, and to conduct rigorous causal checks on candidate rules. In addition, rule extraction itself requires accumulating a sufficient number of interaction records as material, and offline organization can batch process historical memory, extracting statistically significant patterns from it. Therefore, separating the two ensures both the real-time nature of online interaction and the depth and reliability of memory updates, achieving a balance between fast response and deep thinking.

[0056] In summary, this embodiment of the present disclosure, through the execution of the counterfactual verification memory construction method for visual scenes in steps S101 to S105, inputs the target image and target question-and-answer information into a visual agent to perform rule understanding, and outputs candidate memory rules that simultaneously contain causal visual factors and non-causal interference factors. First, the causal attributes of various factors affecting the question-and-answer results are decomposed and defined. Then, using a counterfactual editing approach with controlled variables, two sets of control samples are constructed: one to generate interference edited images by changing non-causal interference factors while keeping causal visual factors unchanged, and the other to generate causal edited images by changing causal visual factors while keeping non-causal interference factors unchanged. These samples can be used to verify the causal validity of the memory rules. Finally, based on two types of... The counterfactual editing image is combined with the target question-and-answer information to complete the verification. Only when the verification is passed and it is confirmed that the answer of the candidate memory rule is determined by causal visual factors rather than co-occurring interference factors, is the corresponding target semantic memory and target counterexample memory combined into a new memory item and updated to the memory bank. This completes the strict screening from the source of memory entry, eliminating low-quality pseudo-memory methods based solely on surface co-occurrence associations. It avoids the shortcut amplification problem from the root, effectively improving the overall quality of the memory bank. Ultimately, the agent's subsequent retrieval and calls are all high-quality memories that have been causally verified, and will not trigger erroneous reasoning due to the accidental occurrence of non-causal interference factors. This significantly improves the robustness of reasoning and achieves the technical effect of improving the reasoning performance of visual agents.

[0057] Please see Figure 3 , Figure 3 yes Figure 2 A flowchart illustrating the process preceding step S101. In some embodiments, steps S201 to S204 may be included before obtaining the target visual question-answering example of the visual agent: Step S201: Obtain the target image and the target question for the target image, and generate a target retrieval key containing question semantics and image description based on the target image and the target question encoding; Step S202: Obtain each candidate memory item from the preset memory bank, encode and generate memory keys for each candidate memory item, calculate the retrieval score between the target retrieval key and each memory key, and select the target memory key from multiple memory keys based on the size of each retrieval score. Step S203: Extract visual question-and-answer examples, semantic memories, and counterexample memories under the memory item corresponding to the target memory key, and form retrieval prompt information. Input the retrieval prompt information, target image, and target question into the visual agent to generate the target answer. Step S204: Based on the target question and target answer information, and the target image, a target visual question and answer example is formed.

[0058] In the above steps, embodiments of this disclosure can acquire a target image and a target question for the target image during the wake-up phase of the visual agent, and generate a target retrieval key containing question semantics and image description based on the target image and target question encoding. The target retrieval key b is a compact representation for retrieving information related to the current task, used to quickly find memory items in the memory bank. It can be obtained by concatenating or encoding the semantic vector of the target question Q, the global description of the target image I, local object or text region features, task labels, candidate answer types, and visual relationship categories. The encoding can use the embedding layer of a visual language model or a separate text encoder. The purpose of designing the retrieval key is to extract the key information of the task into a comparable vector, enabling similar tasks to retrieve relevant memories. For example, for the question "What is the color of the hat in the image?", the retrieval key can include embedded representations of keywords such as color, hat, and object attributes.

[0059] Next, candidate memory items are retrieved from a pre-defined memory bank, and memory keys for each candidate memory item are generated. The retrieval score between the target retrieval key and each memory key is calculated, and the target memory key is selected from multiple memory keys based on the magnitude of each retrieval score. Candidate memory items include rules s from semantic memory, counterexamples h from counterexample memory, and anchoring examples e from episodic memory. Each memory item has its retrieval key b_m pre-stored during writing.

[0060] The retrieval score is calculated as follows: R(m|I,Q)=α×sim(b,b_m)+β×LCB(θ_m)+γ×cov(m;I,Q)-ζ×conflict(m)-ξ×cost(m), where sim(b,b_m) is the similarity between the target retrieval key b and the memory key b_m, which can be represented by cosine similarity. The larger the value, the more semantically relevant the memory item is to the current task. LCB(θ_m) is the lower confidence bound of the reliability of the memory item, and LCB is taken as the lower quantile of the posterior distribution, such as 0.1. Quantiles are used to conservatively estimate the intervention effectiveness of rules, avoiding overconfidence caused by a few validation samples. `cov(m; I,Q)` measures the coverage of current task factors by the rule, such as whether the causal factors mentioned in the rule are identifiable in the current image. `conflict(m)` indicates the degree of contradiction between the memory item and existing conflict indexes or stronger rules. `cost(m)` represents the text length or image anchor cost required to include the memory item in the prompt. `α`, `β`, `γ`, `ζ`, and `ξ` are configurable weight parameters that can be set according to actual needs. The system selects a preset number of semantic rules, counterexample warnings, and anchor samples from high to low search scores; the corresponding memory keys are the target memory keys.

[0061] Furthermore, in order to improve the speed and efficiency of retrieval, the retrieval score can also be obtained directly based on the similarity between the target retrieval key b and the memory key b_m. This embodiment of the present disclosure does not impose specific limitations on this.

[0062] It should be noted that the retrieval score in this embodiment of the disclosure not only considers semantic similarity, but also introduces multiple factors such as reliability confidence lower bound, coverage, conflict and cost, so that the retrieval results are more inclined to select memory items with high reliability, low conflict and low cost, thereby improving the efficiency of the use of prompt words.

[0063] Subsequently, visual question-and-answer examples, semantic memories, and counterexample memories corresponding to the target memory key are extracted and combined to form retrieval prompts. These retrieval prompts, the target image, and the target question are then input into the visual agent to generate the target answer. The retrieval prompts are natural language prompts organized from the selected memory items. For example, the retrieval prompts can be rendered into three blocks: a verified rule block containing the explanation text of the reasoning rules in the semantic memory, causal visual factors, and non-causal interference factors; a known counterexample warning block containing counterexample memories, i.e., the interference edited images and their explanation text in the counterexample memories; and an optional anchored example block containing representative historical interactions in the episodic memory. The prompt context, the original target image I, and the target question Q are then input into the parameter-frozen visual language model V, and the model outputs the target answer A and the reasoning basis ρ.

[0064] In some embodiments, candidate memory rules and semantic memories also include inference rule description text. Step S203, when generating the target answer, may further include step S2031: The inference rule description text corresponding to the semantic memory in the retrieval prompt information, causal visual factors and non-causal interference factors, as well as counterexample memories and visual question-and-answer examples, are structurally organized and input into the visual agent along with the target image and target question. This allows for multimodal inference based on the verified inference rule description text, learning causal visual factors, avoiding warnings from non-causal interference factors, and learning historical interaction references from counterexample memories, thereby generating the target answer.

[0065] Next, this embodiment of the disclosure further constructs a target visual question-and-answer example based on the target question and target answer, and the target image. Specifically, after the reasoning is completed, the interaction record e=(I,Q,A,ρ,o,b,κ) is appended to the plot memory E, where o is an optional result signal, such as obtained through human feedback, and κ is metadata such as task type and timestamp. This record constitutes the target visual question-and-answer example in step S101, for use in the offline processing stage.

[0066] It should be noted that, by storing the reasoning results and the basis together in the plot memory, the embodiments of this disclosure preserve the complete experiential context, providing rich material for rule generation in the offline stage and avoiding information loss caused by only storing the answers.

[0067] In some embodiments, the target question-and-answer information includes the target question and the target answer. (See [link to relevant documentation]). Figure 4 , Figure 4 yes Figure 2 The flowchart further includes step S104. In some embodiments, the process of verifying the interfering edited image and the causal edited image based on the target question-answering information and the visual agent to obtain the verification result may also include steps S301 to S304: Step S301: Based on the target image, perform fidelity verification on the interference-edited image and the causal-edited image, generate fidelity weights, and perform fidelity filtering on the interference-edited image and the causal-edited image based on the fidelity weights; Step S302: Input the target question and the interference-edited image and causal-edited image that have been filtered for fidelity into the visual agent to answer again, and generate the corresponding interference-edited answer and causal-edited answer; Step S303: Determine the interference invariance score based on the fidelity weight, the target answer, and the interference edited answer after fidelity filtering; and determine the causal sensitivity score based on the fidelity weight, the target answer, and the causal edited answer after fidelity filtering. Step S304: Based on the relationship between the interference invariance score and the preset first threshold, and the causal sensitivity score and the preset second threshold, the verification result is obtained.

[0068] In the above steps, embodiments of this disclosure can perform fidelity verification on interfering edited images and causal edited images based on the target image, generate fidelity weights, and perform fidelity filtering on interfering edited images and causal edited images based on the fidelity weights. The fidelity verification is performed by an editing verifier W, which can be the visual agent V in the above embodiments, or other models can be used as the editing verifier alone for verification operations; this disclosure does not impose specific limitations on this.

[0069] Furthermore, the editor verifier W can perform three types of checks on each edited image. First, it checks whether the protected factor is preserved, such as verifying that the hat color has indeed not changed. Second, it checks whether the target factor has been changed as required, such as verifying that the background has indeed been replaced. Third, it checks whether the edited image is clear, natural, and still answers the original question. The verifier outputs fidelity weights. ∈[0,1], The closer the fidelity weight is to 1, the higher the editing quality. In this embodiment, the system can discard edited images with a fidelity weight lower than the fidelity threshold, such as 0.5, to prevent low-quality images from interfering with the verification results. The set of interfering edited images that pass verification is denoted as... The set of causal edited images is .

[0070] It should be noted that the embodiments disclosed herein use fidelity filtering to ensure that only high-quality edited images participate in subsequent verification, thereby avoiding false negative or false positive results caused by poor image editing quality and improving the reliability of verification.

[0071] Subsequently, the target question, along with the interference-edited image and the causal-edited image that have passed fidelity filtering, are input into the visual agent for a re-answer, generating corresponding interference-edited and causal-edited answers. Specifically, for each interference-edited image that passes fidelity filtering... The objective question Q and the image are re-inputted into the visual language model V with frozen parameters to obtain the answer. Similarly, for causal editing images Get the answer The actual label is not used during the re-answering process; instead, the original wake-up phase target answer A is used as the reference answer. This is because this embodiment is concerned with whether the rule leads to a change in the answer, rather than the absolute correctness of the answer itself.

[0072] Next, the interference invariance score is determined based on the fidelity weight, the target answer, and the interference-edited answer after fidelity filtering. Similarly, the causal sensitivity score is determined based on the fidelity weight, the target answer, and the causal editing answer after fidelity filtering. Specifically, embodiments of this disclosure can calculate the required interference invariance score and causal sensitivity score using a preset function. Alternatively, a pre-trained neural network model can be used to output the required interference invariance score when given the fidelity weight, the target answer, and the interference-edited answer after fidelity filtering. Furthermore, when given the causal editing answer based on the fidelity weight, the target answer, and the causal editing answer after fidelity filtering, the required causal sensitivity score can be output.

[0073] Finally, the verification result is obtained based on the relationship between the interference invariance score and a preset first threshold, and the causality sensitivity score and a preset second threshold. The magnitudes of the first and second thresholds can be set according to actual needs, and this embodiment does not impose specific limitations on them. In this embodiment, the verification result is considered passed only when all these conditions are met simultaneously.

[0074] It should be noted that, by setting independent thresholds, the requirements for interference invariance and causal sensitivity are controlled separately, allowing users to adjust the strictness for different task scenarios. For example, in high-risk scenarios, the threshold can be increased to reduce the probability of misjudging false rules as true.

[0075] Please see Figure 5 , Figure 5 yes Figure 4 The flowchart further includes step S303. In some embodiments, the process of determining the interference invariance score based on the fidelity weight, the target answer, and the interference edited answer after fidelity filtering, and determining the causal sensitivity score based on the fidelity weight, the target answer, and the causal edited answer after fidelity filtering, may further include steps S401 to S402: Step S401: For each interference editing image that has completed fidelity filtering, determine whether its corresponding interference editing answer is equivalent to the target answer to obtain the first judgment result. Use the fidelity weight corresponding to each image as a coefficient to perform a weighted summation on the first judgment result. Divide the summation result by the sum of the fidelity weights of all valid interference editing images to obtain the interference invariance score. Step S402: For each causal edited image that has completed fidelity filtering, determine whether its corresponding causal edit answer is equivalent to the target answer to obtain a second determination result. After inverting the second determination result, perform a weighted summation based on the fidelity weight of each image. Divide the summation result by the summation of the fidelity weights of all valid causal edited images to obtain the causal sensitivity score.

[0076] In the above steps, the embodiments of this disclosure can calculate the required interference invariance score and causality sensitivity score by setting a preset function.

[0077] Specifically, the formulas for calculating the interference invariance score Inv(h) and the causality sensitivity score Nec(h) are as follows: ; ; in, To ensure fidelity, The first judgment result is obtained by determining whether the corresponding interference edited answer is equivalent to the target answer. If the re-answer is equivalent to the original answer A, the result is set to 1; otherwise, it is set to 0. The second judgment result is obtained by determining whether the corresponding causal edit answer is equivalent to the target answer. The value is 1 if the re-answer is not equivalent to the original answer A, otherwise it is 0.

[0078] The above function uses option letter normalization, case and punctuation normalization, numerical format normalization, or the official evaluator to determine equivalence based on task type. The denominator is the sum of the fidelity weights of all perturbation-edited images that pass fidelity verification. The perturbation invariance score Inv(h) represents the proportion of the answer that remains unchanged under conditions of changing perturbation factors. Ideally, a true causal rule should be insensitive to changes in perturbation, and Inv(h) should be close to 1. The causal sensitivity score Nec(h) represents the proportion of the answer that changes under conditions of changing causal evidence. Ideally, a true causal rule should be sensitive to causal evidence, and Nec(h) should be close to 1.

[0079] It should be noted that the present invention calculates the score by weighted average. The fidelity weight ensures that high-quality edited images contribute more, while the influence of low-quality edited images is suppressed. At the same time, the two scores correspond to invariance and sensitivity, respectively, forming a complementary causal test.

[0080] In some embodiments, candidate memory rules also include a scope description text; see [link to relevant documentation]. Figure 6 , Figure 6 yes Figure 4 The flowchart further included in step S304, in the process of obtaining the verification result based on the relationship between the interference invariance score and a preset first threshold, and the causality sensitivity score and a preset second threshold, may also include steps S501 to S502: Step S501: Based on the differences between the applicable scope description text in the candidate memory rule and the applicable scope description text under each memory item in the memory bank, determine the conflict score of the candidate memory rule; Step S502: Based on the relationship between the interference invariance score and the preset first threshold, the causal sensitivity score and the preset second threshold, and the conflict score and the preset third threshold, the verification result is obtained.

[0081] In the above steps, this embodiment of the disclosure also considers the magnitude of the conflict score during the verification process. Further, the conflict score of the candidate memory rule can be determined based on the difference between the scope description text in the candidate memory rule and the scope description text under each memory item in the memory bank. The conflict score Conflict(h) is used to measure the degree of contradiction between the candidate rule and stronger rules in the existing memory bank.

[0082] Specifically, the scope of the candidate memory rule is compared with the scope of each rule in the existing semantic memory. If there is a high degree of overlap in scope but opposite rule conclusions (e.g., both are applicable to color recognition tasks, but one says to look at the hat color and the other says to look at the clothing color), or if there is an essential contradiction in causal factors, then the conflict score is high. The conflict score can also be directly quantified by the proportion of inconsistencies between the prediction results of the candidate rule and the existing rule on the same test sample; this disclosure does not impose specific limitations on this.

[0083] Specifically, in this embodiment of the disclosure, the first threshold can be set as: The second threshold is The third threshold is Only when the interference invariance score Inv(h) ≥ Causal sensitivity score Nec(h) ≥ Conflict score (h) < When these three conditions are met, candidate memory rules are considered to have passed counterfactual verification and can be promoted to long-term semantic memory.

[0084] It should be noted that the embodiments of this disclosure introduce a conflict detection mechanism to prevent new rules from contradicting existing high-reliability rules, maintain the logical consistency of the memory, and at the same time control the conflict tolerance through a threshold to allow for the existence of an appropriate degree of supplementary rules.

[0085] Please see Figure 7 , Figure 7 yes Figure 2 The flowchart further includes steps S105. In some embodiments, after combining the target visual question-answering example, target semantic memory, and target counterexample memory into a new memory item and updating it into the visual agent's memory bank, steps S601 to S602 may also be included: Step S601: Under the preset prompt budget constraint, calculate the retention value of each memory item in the memory bank. The retention value is obtained by summing the replay benefit, reliability confidence lower bound, and applicable scope coverage benefit of the memory item, and then deducting the redundancy penalty and conflict penalty. Step S602: Perform greedy pruning based on the retention values ​​of each memory item, and select conflicting memory items to remove them or put them into a dormant state.

[0086] Specifically, the reserved values ​​provided in the embodiments of this disclosure The calculation formula is: ; in, To include the replay benefits of this memory item in the replay task, i.e., the representative task selected from historical memory, such as improved accuracy or increased answer likelihood. To provide a lower bound for reliability confidence, LCB is taken as the lower quantile of the posterior distribution, which is used to conservatively estimate the intervention effectiveness of the rule and avoid overconfidence in a small number of validation samples. The contribution of this memory item to the scope of under-covered tasks is called the scope coverage gain, which means whether it can provide new patterns for currently rare or uncovered task types. This indicates the redundancy of a higher reliability memory item. The minus sign in the formula indicates a conflict penalty. For example, if two rules give the same causal factors under the same applicable scope, the redundancy is higher. This indicates the degree of conflict with counterexamples or new rules. A minus sign in the formula indicates a penalty for the degree of conflict. , , , These are the corresponding weight parameters, and the size of each weight parameter can be set according to actual needs.

[0087] Furthermore, the lower bound of reliability It can be estimated using the Beta-Bernoulli method: ; in, The estimated reliability of each memory item s in the memory bank; The corresponding historical verification dataset; the Beta probability distribution is the statistical model selected in this embodiment of the disclosure. Since the rule verification result is a binary Bernoulli trial of pass / fail, the Beta distribution and the Bernoulli distribution are conjugate priors, which are suitable for continuously iteratively updating the credible probability of the rules. The initial prior parameters preset for the system are the first initial shape parameters of the Beta distribution, which are fixed hyperparameters. They represent the number of times the initial verification passes the prior parameters before any counterfactual verification is carried out, and are used for rule initialization. Another preset initial prior parameter for the system is the second initial shape parameter of the Beta distribution, which is also a fixed hyperparameter. It represents the number of initial verification failures preset by the system before any counterfactual verification is performed. The initial reliability distribution that constitutes the rules; Let s be the cumulative number of valid passes. The rule is considered to have passed the verification if it simultaneously meets the first threshold, the second threshold, and the third threshold in each counterfactual verification. This represents the cumulative number of valid failures for s.

[0088] Next, a greedy pruning process is performed based on the retention values ​​of each memory item, selecting conflicting memory items for removal or putting them into a dormant state. Specifically, under the condition that the total cost constraint does not exceed a preset cost budget threshold, this disclosure greedily selects memory items from high to low retention values ​​Π(m), removing memory items with low retention values. If two rules conflict and cannot be resolved by merging, the rule with the higher reliability confidence lower bound is retained, and the other rule can be deleted or put into a dormant state as a counterexample for diagnosis when similar conflicts recur. Based on this, the pruned memory bank remains compact, reliable, and diverse, effectively solving the problem of infinite memory expansion.

[0089] It should be noted that, through budget pruning and reliability confidence lower bound mechanisms, this embodiment maintains a high-value, low-redundancy, and conflict-free compact memory within a limited prompt term budget, significantly improving retrieval efficiency and inference stability.

[0090] The following is a complete explanation of the counterfactual verification memory construction method for visual scenarios, using specific examples, such as a hat color question-and-answer task: During the wake-up phase, the system receives image I (a person wearing a red hat standing on a beach) and question Q (what color is the hat in the image). The system generates a search key, retrieves a verified rule from the memory bank, extracts the corresponding causal visual factor ("when answering a color question, check the color area of ​​the target object itself"), the non-causal interference factor ("ignore the background environment"), and the inference rule description text ("do not guess the hat color based on the beach background"). This information is organized into a prompt and input along with I and Q into the frozen visual language model. The model outputs the answer A = "red," and this interaction record is stored in the episodic memory.

[0091] During the offline processing phase, the system selects the record, invokes the visual language model to generate candidate memory rules, and then invokes the image editing model to generate interference edit images, such as changing the background from a beach to grass, snow, or city streets, while keeping the hat red. It also generates causal edit images, such as changing the hat color to blue, green, or yellow, while keeping the beach background unchanged. Each edited image is then validated for fidelity, discarding those that fail to meet quality standards. The frozen visual language model then re-answers the question; if the answer remains red under the interference edit image, the answer changes accordingly under the causal edit image. Next, the interference invariance score and causal sensitivity score are calculated, while conflict scores are checked. If a rule passes validation, it is promoted to semantic memory, and the interference edit image is stored as a negative example to warn against relying on the background in the future.

[0092] In subsequent tasks, when encountering similar questions, such as the question "What is the color of the hat?", but the background changes to indoors or other environments, the system will search for the rule and counterexamples, prioritizing the hat color over the background, thus consistently outputting the correct answer.

[0093] Please see Figure 8 This disclosure also provides a counterfactual verification memory construction system for visual scenes, which can implement the above-described counterfactual verification memory construction method for visual scenes. The counterfactual verification memory construction system for visual scenes includes: The question-and-answer sample acquisition module 801 is used to acquire target visual question-and-answer samples of the visual intelligent agent. The target visual question-and-answer samples include target images and target question-and-answer information for the target images. The candidate rule generation module 802 is used to input the target image and target question-and-answer information into the visual agent for rule understanding and output candidate memory rules. The candidate memory rules contain at least causal visual factors and non-causal interference factors. The counterfactual editing module 803 is used to perform counterfactual image editing on the target image when changing the non-causal interference factors while keeping the causal visual factors unchanged, and when changing the causal visual factors while keeping the non-causal interference factors unchanged, to generate interference-edited images and causal-edited images respectively. The counterfactual verification module 804 is used to verify the interference-edited image and the causal-edited image based on the target question-and-answer information and the visual intelligent agent, and obtain the verification result; The memory construction module 805 is used to determine the candidate memory rules as target semantic memory and the interference editing image as target counterexample memory when the verification result characterization verification is passed, and to combine the target visual question answering example, target semantic memory and target counterexample memory into a new memory item and update it into the memory bank of the visual agent.

[0094] In summary, the counterfactual verification memory construction system for visual scenes executes the counterfactual verification memory construction method for visual scenes described in the above embodiments. It inputs the target image and target question-and-answer information into a visual agent to perform rule understanding, outputting candidate memory rules that simultaneously contain causal visual factors and non-causal interference factors. First, it decomposes and defines the causal attributes of various factors affecting the question-and-answer results. Then, using a counterfactual editing approach with controlled variables, it generates interference-edited images by changing non-causal interference factors while retaining causal visual factors unchanged, and by changing causal visual factors while retaining non-causal interference factors unchanged, generating causal-edited images. This constructs two sets of control samples that can be used to verify the causal validity of the memory rules. Finally, based on… Two types of counterfactual edited images are combined with target question-and-answer information to complete verification. Only when the verification passes and it is confirmed that the answer of the candidate memory rule is determined by causal visual factors rather than co-occurring interference factors, is the corresponding target semantic memory and target counterexample memory combined into a new memory item and updated to the memory bank. This rigorous screening is carried out from the source of memory entry, eliminating low-quality pseudo-memory methods based solely on superficial co-occurrence associations. This avoids the shortcut amplification problem at its root, effectively improving the overall quality of the memory bank. Ultimately, this ensures that the agent's subsequent retrieval and calls are all high-quality memories that have been causally verified, and will not trigger erroneous reasoning due to the accidental occurrence of non-causal interference factors. This significantly improves the robustness of reasoning and achieves the technical effect of improving the reasoning performance of visual agents.

[0095] The specific implementation of the counterfactual verification memory construction system for visual scenes is basically the same as the specific embodiment of the counterfactual verification memory construction method for visual scenes described above, and will not be repeated here. Subject to meeting the requirements of the embodiments of this disclosure, the counterfactual verification memory construction system for visual scenes may also be equipped with other functional modules to implement the counterfactual verification memory construction method for visual scenes described above.

[0096] This disclosure also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the above-described counterfactual verification memory construction method for visual scenes. This electronic device can include any smart terminal such as a tablet computer or an in-vehicle computer.

[0097] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 to execute the counterfactual verification memory construction method for visual scenes according to the embodiments of this disclosure. The 903 input / output interface is used to implement information input and output. The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904); The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0098] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for constructing counterfactual verification memories for visual scenes.

[0099] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0100] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0101] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0102] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0103] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0104] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such orders can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0105] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0106] In the embodiments provided in this disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0107] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0109] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0110] The preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present disclosure. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present disclosure shall be within the scope of the claims of the present disclosure.

Claims

1. A method for constructing counterfactual verification memories for visual scenes, characterized in that, include: Obtain a target visual question-and-answer example for a visual intelligent agent, wherein the target visual question-and-answer example includes a target image and target question-and-answer information for the target image, and the target question-and-answer information includes a target question and a target answer; The target image and the target question-and-answer information are input into the visual agent for rule understanding, and candidate memory rules are output. The candidate memory rules include at least causal visual factors and non-causal interference factors. By changing the non-causal interference factor while keeping the causal visual factor unchanged, and by changing the causal visual factor while keeping the non-causal interference factor unchanged, counterfactual image editing is performed on the target image to generate interference-edited images and causal-edited images, respectively. Based on the target image, the fidelity of the interference-edited image and the causal-edited image is verified, a fidelity weight is generated, and fidelity filtering is performed on the interference-edited image and the causal-edited image based on the fidelity weight; The target question is input into the visual agent along with the interference-edited image and the causal-edited image after fidelity filtering, respectively, to generate the corresponding interference-edited answer and causal-edited answer. The interference invariance score is determined based on the fidelity weight, the target answer, and the interference edited answer after fidelity filtering; and the causal sensitivity score is determined based on the fidelity weight, the target answer, and the causal edited answer after fidelity filtering. The verification results are obtained based on the relationship between the interference invariance score and the preset first threshold, and the causal sensitivity score and the preset second threshold. If the verification result characterization verification passes, the candidate memory rule is determined as the target semantic memory, and the interference editing image is determined as the target counterexample memory. The target visual question answering example, the target semantic memory, and the target counterexample memory are combined into a new memory item and updated into the memory bank of the visual agent.

2. The method for constructing counterfactual verification memories for visual scenes according to claim 1, characterized in that, Before acquiring the target visual question-answering example of the visual agent, the method further includes: Obtain a target image and a target question for the target image, and generate a target retrieval key containing question semantics and image description based on the target image and the target question; Retrieve each candidate memory item from a preset memory bank, encode and generate a memory key for each candidate memory item, calculate the retrieval score between the target retrieval key and each memory key, and select the target memory key from multiple memory keys based on the size of each retrieval score. Extract visual question-and-answer examples, semantic memories, and counterexample memories under the memory item corresponding to the target memory key, and form retrieval prompt information. Input the retrieval prompt information, the target image, and the target question into the visual agent to generate the target answer. The target question and answer information, composed of the target question and the target answer, and the target image constitute a target visual question and answer example.

3. The method for constructing counterfactual verification memories for visual scenes according to claim 2, characterized in that, The candidate memory rules and the semantic memory also include inference rule description text. The step of inputting the retrieval prompt information, the target image, and the target question into the visual agent to generate the target answer includes: The reasoning rule description text, causal visual factors, and non-causal interference factors corresponding to the semantic memory in the retrieval prompt information, as well as the counterexample memory and the visual question-and-answer sample, are structured and organized, and input into the visual agent along with the target image and the target question. Based on the verified reasoning rule description text, learning the causal visual factors, avoiding the warnings of the non-causal interference factors, and learning the historical interaction references in the counterexample memory, multimodal reasoning is performed to generate the target answer.

4. The method for constructing counterfactual verification memories for visual scenes according to claim 1, characterized in that, The process of determining the interference invariance score based on the fidelity weight, the target answer, and the interference-edited answer after fidelity filtering, and determining the causal sensitivity score based on the fidelity weight, the target answer, and the causal-edited answer after fidelity filtering, includes: For each of the interference-edited images that has completed fidelity filtering, it is determined whether the corresponding interference-edited answer is equivalent to the target answer to obtain a first determination result. The first determination result is weighted and summed using the fidelity weight corresponding to each image as a coefficient. The summation result is divided by the sum of the fidelity weights of all valid interference-edited images to obtain the interference invariance score. For each causal edited image that has completed fidelity filtering, it is determined whether the corresponding causal edit answer is equivalent to the target answer to obtain a second determination result. After inverting the second determination result, a weighted sum is performed based on the fidelity weight of each image. The sum is then divided by the sum of the fidelity weights of all valid causal edited images to obtain the causal sensitivity score.

5. The method for constructing counterfactual verification memories for visual scenes according to claim 1, characterized in that, The candidate memory rule also includes an explanation of its applicable scope; the verification result obtained based on the relationship between the interference invariance score and a preset first threshold, and the causal sensitivity score and a preset second threshold, further includes: Based on the differences between the scope description text in the candidate memory rule and the scope description text under each memory item in the memory bank, the conflict score of the candidate memory rule is determined. The verification results are obtained based on the relationship between the interference invariance score and the preset first threshold, the causal sensitivity score and the preset second threshold, and the conflict score and the preset third threshold.

6. The method for constructing counterfactual verification memories for visual scenes according to claim 1, characterized in that, After combining the target visual question-answering example, the target semantic memory, and the target counterexample memory into a new memory item and updating it in the memory bank of the visual agent, the method further includes: Under the preset prompt word budget constraint, the retention value of each memory item in the memory bank is calculated. The retention value is obtained by summing the replay benefit, reliability confidence lower bound, and applicable scope coverage benefit of the memory item, and then deducting the redundancy penalty and conflict penalty. Greedy pruning is performed based on the retention values ​​of each memory item, and conflicting memory items are either removed or put into a dormant state.

7. A counterfactual verification memory construction system for visual scenes, characterized in that, include: The question-and-answer sample acquisition module is used to acquire target visual question-and-answer samples of the visual intelligent agent. The target visual question-and-answer samples include target images and target question-and-answer information for the target images. The target question-and-answer information includes target questions and target answers. The candidate rule generation module is used to input the target image and the target question-and-answer information into the visual agent for rule understanding and output candidate memory rules, wherein the candidate memory rules include at least causal visual factors and non-causal interference factors. The counterfactual editing module is used to perform counterfactual image editing on the target image when the non-causal interference factor is changed while the causal visual factor remains unchanged, and when the causal visual factor is changed while the non-causal interference factor remains unchanged, to generate interference-edited images and causal-edited images respectively. The counterfactual verification module is used to perform fidelity verification on the interference-edited image and the causal-edited image based on the target image, generate fidelity weights, and perform fidelity filtering on the interference-edited image and the causal-edited image based on the fidelity weights; the target question is then input into the visual agent along with the interference-edited image and the causal-edited image after fidelity filtering for re-answering, generating corresponding interference-edited answers and causal-edited answers; The interference invariance score is determined based on the fidelity weight, the target answer, and the interference edited answer after fidelity filtering; and the causal sensitivity score is determined based on the fidelity weight, the target answer, and the causal edited answer after fidelity filtering. The verification results are obtained based on the relationship between the interference invariance score and the preset first threshold, and the causal sensitivity score and the preset second threshold. The memory construction module is used to determine the candidate memory rule as the target semantic memory and the interference editing image as the target negative example memory when the verification result characterization verification is passed, and to combine the target visual question answering example, the target semantic memory and the target negative example memory into a new memory item and update it into the memory bank of the visual agent.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the counterfactual verification memory construction method for visual scenes as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the counterfactual verification memory construction method for visual scenes as described in any one of claims 1 to 6.

Images