A training-free agent context adaptation method and system for open-world self-view action recognition
By using the multi-agent collaborative framework EgoDirector, the generalization and adaptability issues of self-perspective action recognition in open-world environments are solved, achieving training-free adaptive action recognition, reducing costs and improving recognition accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-09
- Publication Date
- 2026-07-14
AI Technical Summary
Traditional self-perspective action recognition methods have limited generalization ability in open, dynamic real-world environments, struggle to adapt to new action-object combinations, face perceptual uncertainty and combinatorial explosion problems, and rely on large-scale labeled datasets, resulting in high costs and low adaptability.
The EgoDirector multi-agent collaborative framework is adopted, which combines object perception, action reasoning and activity prediction agents with visual context and semantic context engines to achieve training-free adaptation, dynamically update the knowledge base and policy base, and solve combinatorial explosion and perceptual uncertainty.
Significantly improves recognition accuracy and robustness in open-world environments, reduces data collection and model retraining costs, and enables rapid deployment and continuous learning capabilities of the system in real-world scenarios.
Smart Images
Figure CN122392124A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of computer vision and artificial intelligence, and in particular to a training-free agent context adaptation method and system for open-world self-perspective action recognition. Background Technology
[0002] Egocentric Action Recognition (EAR), an important research direction in computer vision and artificial intelligence, aims to accurately identify and understand human actions from first-person perspective videos, and is key to promoting the development of embodied intelligence and augmented reality technologies.
[0003] With continuous technological advancements, EAR (Extended Application Analyzer) has demonstrated enormous application potential in various fields such as smart homes, medical assistance, and industrial automation. However, traditional EAR methods face serious challenges in terms of performance and generalization ability when dealing with open, dynamic real-world environments.
[0004] Traditional EAR methods mainly rely on supervised learning on large-scale labeled datasets, using deep learning models to identify predefined action categories.
[0005] However, this method has the following limitations: 1. Static Knowledge Dilemma: After training, the knowledge of traditional models is frozen in the model parameters, making it difficult to adapt to the constantly emerging new action-object combinations in the open world, thus limiting generalization ability. For example, when encountering new actions or objects not found in the training set, the model performance will significantly decrease.
[0006] 2. Perceptual uncertainty: Factors such as camera movement, rapid changes in perspective, and hand-object occlusion commonly found in self-view videos cause visual cues to become fragmented and blurred, making accurate recognition based on raw pixels extremely difficult.
[0007] 3. Verb-Noun Combination Explosion: Real-world activities consist of an infinite number of combinations of atomic verbs (such as cutting and cleaning) and diverse interactive objects. No fixed dataset can cover this huge combination space, which means that the model must constantly face unseen verb-noun pairs.
[0008] Self-perspective action recognition in open-world environments faces two core challenges: 1. Level 3 Open Environment: According to the classification by Kundu et al., Level 3 open environments require systems to identify verbs, nouns, combinations, and domains that are completely unknown. This is currently the most challenging setting in the EAR (Extreme Errata) field. Traditional methods experience a sharp decline in performance under these conditions, making it difficult to meet the needs of practical applications.
[0009] 2. Combinatorial explosion problem: As the types of verbs and nouns increase, the number of possible verb-noun combinations grows exponentially. Traditional methods are unable to effectively handle this combinatorial explosion problem, resulting in a significant drop in recognition accuracy.
[0010] Although recent research has begun to focus on the open-set EAR problem and has attempted to improve performance by introducing external knowledge or leveraging the zero-shot capability of visual language models (VLMs), the following shortcomings still exist: 1. Pre-training dependency: Many methods rely on pre-training on specific self-perspective action datasets and often perform semantic search within a closed set embedding space, which limits their adaptability to new environments.
[0011] 2. Static video-text pairs: Existing methods often directly match the original video frame sequence with static video-text pairs, leading to context dilution problems and making it difficult to handle complex time processes and interactive contexts in self-view videos.
[0012] 3. Lack of dynamic adaptation: Traditional methods lack dynamic context adaptation mechanisms, making it difficult to continuously update knowledge during the reasoning process to cope with the uncertainties in the open world.
[0013] To address the aforementioned issues, this invention proposes a novel unsupervised multi-agent collaborative training framework called EgoDirector. By reconstructing open-world self-perspective action recognition as a collaborative reasoning process, it overcomes the limitations of traditional methods in open environments. EgoDirector coordinates specialized agents for object, action, and activity reasoning and is equipped with a self-evolving context adaptation mechanism, achieving self-perspective action recognition without training, significantly improving the system's performance and generalization capabilities in open-world environments. Summary of the Invention
[0014] The purpose of this invention is to address the shortcomings of existing technologies by proposing a training-free agent context adaptation method and system for open-world self-perspective action recognition.
[0015] The objective of this invention is achieved through the following technical solution: a training-free agent context adaptation method for open-world self-perspective action recognition, the method comprising: Initialize the knowledge base and reflection strategy library, and configure the intelligent agent and runtime environment; The received video is preprocessed, each frame of the image is segmented, and the segmentation results are formatted into visual cues, so that the visual cues and the images form a two-source pair as visual context. The intelligent agent performs reasoning, and the knowledge base is updated when new objects are identified during the reasoning process. The reasoning process includes: Using an object-aware agent, we synthesize interaction-centric visual cues from the visual context and a set of candidate nouns from a knowledge base to extract a ranked set of interactive objects. The action reasoning agent receives a set of objects, visual context, and structured prior knowledge from a knowledge base. For each object, it retrieves its verb-noun association cache and concept-action profile, combines visual evidence and semantic association with policies provided by the policy library, generates possible nouns for each object, and combines nouns with verbs to construct an action hypothesis space. The activity prediction agent aggregates visual context, candidate noun set of knowledge base, interactive objects and hypothesis space, scores and ranks each candidate action in hypothesis space, and generates prediction results. The reflective policy library is updated after each round of reasoning by the agent; the final prediction result is formatted as a JSON array and output.
[0016] Furthermore, the initialization of the knowledge base includes: Generate a set of high-frequency everyday nouns using a large language model ; Extract object categories from existing public self-perspective datasets to form Set; merge and To form an initial core noun set = ; for For each noun 'o' in the code, a structured configuration file is automatically generated using a visual language model. Object-Action Associated Caching Build a basic knowledge base ; in This represents a set of known concepts, where 'o' represents a key action or object. For concept action files; For verb-noun association caching; The concept action profile includes visual appearance features, action and interaction patterns, and scene context; The verb-noun association cache maps the object noun o to a set of potential verbs. As A quick index and supplement.
[0017] Furthermore, updating the knowledge base includes: During the reasoning process, if the object-aware agent recognizes a new term... , making The system will then automatically invoke the visual language model and create a new structured configuration file in real time based on the name and visual description of the new term. and associated cache This allows for the seamless integration of newly generated knowledge components into the existing knowledge base. ← This mechanism ensures that the system continuously accumulates knowledge, enabling it to use newly acquired information to make accurate inferences in subsequent tasks.
[0018] Furthermore, the reflection strategy library includes: Initialize a hierarchical policy memory reflection, which is divided into three levels, corresponding to the coarse-grained object-scene matching strategy used by the object perception agent, the medium-grained hand-object interaction pattern strategy used by the action reasoning agent, and the fine-grained action sequence and intent inference strategy used by the activity prediction agent. The initial reflection strategy is empty, or a cold start is performed from a small amount of labeled data, expert knowledge, or a general prior strategy generated by LLM. The strategy entry format includes: category, strategy text description content, frame_strategy cue template for frame-level visual evidence, and frequency of validation pass.
[0019] Furthermore, updating the reflection strategy library includes... Based on the application effect of the strategy and newly identified objects and actions, the reflection strategy library is dynamically updated, including adding strategies, modifying existing strategies, or deleting invalid strategies; the reflection strategy agent is invoked to generate reflection strategies for the three agents respectively based on the output of the three agents, the visual context, and the structured prior knowledge of the knowledge base; The strategy generated by the agent is added to the corresponding level of reflection strategy in the form of frequency=1; Frequency accumulation and semantic merging: When the number of strategy entries at any level exceeds a preset threshold, a merge and refinement process is triggered. Calculate the text embeddings for all strategies at this level and calculate the pairwise cosine similarity. Traverse in descending order of frequency, starting with the highest frequency strategy: if the similarity between a subsequent strategy and the current strategy is greater than a preset value and they belong to the same category, they are considered to be mergeable. The frequencies are accumulated, and the strategy with the highest original frequency is retained as the representative, while the rest are discarded. If the merged data still exceeds the maximum capacity, the top entries are truncated by frequency.
[0020] Furthermore, the preprocessing includes: Time-sample the input video V and extract the T-frame RGB image I. ; Each frame of image is processed using a pre-trained hand-object segmentation model. Segmentation is performed to obtain mask information for the hand and the object; For each frame Format the segmentation results as visual cues : in Represents background pixels, Representative category The bounding box, where H represents a hand and O represents an object; Ultimately, the visual context is constructed as a set of two-source pairs: .
[0021] Furthermore, the workflow of the object-aware intelligent agent is as follows: Receive visual context And the candidate noun set N of the knowledge base; Upon receiving the visual context, the visual language model is invoked to perform analysis based on three strict constraints: Interaction-centric filtering: Strictly instructing agents to focus on The H and O masks highlight the areas that meet the criteria for clear physical interaction, while suppressing background noise; Based on the provided policy library, the top k most frequent policies corresponding to the object-aware agent are extracted as part of the system prompts, guiding the object-aware agent to pay more attention to the frequently verified object occurrence patterns and typical interaction cues. Hybrid Recognition Strategy: To balance accuracy and generalization ability, the agent employs a hybrid search strategy, based on a candidate list from the knowledge base. Prioritize matching objects; to suit open-world settings, explicitly allow the generation of new terms when strong visual evidence indicates objects outside a predefined set. ; Output list The extracted object set is sorted in descending order of the strength of visual interaction evidence. Encoded as a machine-readable JSON object.
[0022] Furthermore, the workflow of the action reasoning agent is specifically as follows: integrating the detected object set O and dynamic visual cues from the object perception agent. Structured prior knowledge of the knowledge base And strategies from the Reflective Strategy Library The hypothesis space is constructed by inferring the most reasonable operation verb v for each object provided by OPA. For each object o in O, the AI retrieves its corresponding object-action association cache and concept-action profile from the knowledge base, and the output is formatted as a strictly defined, machine-parseable JSON list wrapper.
[0023] Furthermore, the workflow of the activity prediction agent includes: performing conditional scoring and ranking. in This represents the confidence score for activity a. To reflect on the strategy library, high-frequency fine-grained action sequences and intention strategies are used as counterfactual checks or ranking weights to eliminate erroneous assumptions; scoring function Each candidate is evaluated based on the following three priority criteria: Visual-hypothesis alignment: Validating whether the proposed action is visually aligned with the hypothesis. The captured hand interaction pattern matches; Semantic rationality: utilizing prior knowledge Check the logical compliance of the action-object pair in the given scene context; Confidence aggregation: Combine the detection intensity from the object-aware agent and the generation likelihood from the action-reasoning agent to form a unified confidence metric; The final output is a strictly formatted JSON array containing a list of activities sorted in descending order of confidence level.
[0024] According to another aspect of the specification, a system for implementing the method is also provided, comprising: an object-aware agent, an action-reasoning agent and an activity-predicting agent, a visual context engine, a semantic context engine and a reflective policy library; The visual context engine eliminates background interference to create clean, focused visual inputs for key interactions, namely hands and objects. The semantic context engine is used to build and update the knowledge base; The reflexive strategy library is used to provide strategy suggestions during agent reasoning.
[0025] The beneficial effects of this invention are: This invention significantly reduces the complexity of single-step reasoning by designing three functionally complementary and clearly defined dedicated agents, gradually narrowing the search space and solving the combinatorial explosion problem through multi-agent collaborative reasoning. A Visual Context Engine (VCE) is designed, which uses explicit hand-object interaction segmentation technology to eliminate irrelevant background areas in video in real time, constructing clean visual input focused on key interactions. This effectively combats perceptual uncertainty common in self-view videos, providing a high-quality visual foundation for subsequent agents. A Semantic Context Engine (SCE) maintains an online, dynamically expandable structured action knowledge base. During reasoning, it automatically detects and learns new nouns and verbs, integrating them into the knowledge system to form a continuously growing open knowledge graph, addressing the static knowledge dilemma and improving the ability to recognize unknown actions. A reflective strategy library is then designed to supervise and check each recognition process and provide the agent with the correct strategy, further improving the system's generalization and stability.
[0026] The method proposed in this invention is a "training-free" framework, completely eliminating the reliance on large-scale, high-quality labeled datasets found in traditional methods. In practical applications (such as smart homes and industrial automation production lines), enterprises do not need to invest heavily in data collection and manual annotation, nor do they need to perform expensive model retraining or fine-tuning for new scenarios or actions. The system is ready to use immediately after deployment, significantly reducing the initial construction and subsequent maintenance costs of AI solutions and accelerating the transition of technology from the laboratory to real-world scenarios.
[0027] Through multi-agent collaboration and a self-evolutionary mechanism based on a reflective strategy library, this invention demonstrates significantly higher recognition accuracy and robustness than traditional methods in open, dynamic real-world environments. At the application level, this means: Reduced human intervention: For example, in medical assistance or industrial monitoring scenarios, the system can accurately identify unseen combined actions such as "using new tools for cutting", which greatly reduces the cost of manual review or false alarm handling caused by model failure.
[0028] Ensuring the stability of critical tasks: In scenarios with extremely high real-time requirements, such as autonomous driving or monitoring of hazardous operations, the system can dynamically adapt to newly emerging objects or actions without interrupting services, ensuring business continuity and safety.
[0029] This invention constructs an open-world recognition system with continuous learning capabilities: through the continuous evolution of the Semantic Context Engine (SCE) and the Reflective Strategy Library, the system becomes "smarter with use" as its frequency of use increases after deployment, building core technological advantages in specific vertical fields. Attached Figure Description
[0030] Figure 1 This is a schematic diagram of the overall framework of the method provided in the embodiments of the present invention. Detailed Implementation
[0031] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.
[0032] like Figure 1 As shown, the core concept of this invention is to reformulate the complex EAR (Extremely Experiential Learning) task as a reasoning process executed collaboratively by multiple specialized intelligent agents. These agents are empowered by a self-evolving context engine, enabling them to continuously adapt to a dynamically open environment. The framework includes four specialized agents: Object-Aware Agent (OPA), Action-Inference Agent (ARA), Activity-Predicting Agent (APA), and Reflective Policy Agent. These agents are supported by three foundational engines: Visual Context Engine (VCE), Semantic Context Engine (SCE), and Reflective Policy Library (RPA).
[0033] To elaborate on the specific implementation of EgoDirector in open-world self-centered action recognition, the following will provide a detailed explanation of several key aspects, including system initialization, input processing, multi-agent collaborative reasoning, dynamic context adaptation, and output generation.
[0034] 1. System Initialization During the system startup phase, EgoDirector needs to perform a series of initialization operations to build the basic knowledge base and configuration environment in preparation for the subsequent reasoning process.
[0035] Knowledge base initialization: A knowledge base is defined as a collection of semantic entries indexed by object nouns: ,in This represents a set of known concepts. Each entry consists of two complementary components: Conceptual Action Files ( ): For each key action or object o, a detailed descriptive profile is generated using the implicit knowledge of the large language model. These archives systematically organize three-dimensional information: (1) visual appearance features ( ): Describe typical object materials (such as metal), shapes (such as cylinders), colors (such as silver), and their partial representations from a first-person perspective. This provides prior knowledge for OPA to complete visual information under perceptual uncertainty (such as occlusion, blur). For example, even if only part of the faucet is visible, such a description will guide the model to focus on its reflective properties and typical location. (2) Action and interaction modes ( ): Enumerate high-probability verbs associated with objects (e.g., turn on, turn off) and precisely describe hand movement trajectories (e.g., rotate, push and pull) and typical action sequences (e.g., activate-use-deactivate). This approach directly decouples objects and links them to potential actions, which is a key strategy for handling combinatorial explosion. It allows the system to index the set of potential actions by objects, rather than remembering all combinations. (3) Scene context ( This clarifies typical scenarios (e.g., kitchen, classroom) and objects that frequently occur simultaneously (e.g., washbasin, detergent). This provides strong contextual constraints for action recognition, and scene consistency is crucial disambiguation evidence when visual perception becomes blurred.
[0036] Object action associated cache ( To achieve fast response, a lightweight key-value cache is maintained, mapping object 'o' to a set of potential action verbs. The cache serves as Its rapid indexing and supplementation allow for the instant retrieval of action hypotheses for any detected object and significantly improve reasoning efficiency in open-world settings.
[0037] The initialization of the knowledge base includes two parts: concept set construction and knowledge generation, specifically: Generate a set of high-frequency everyday nouns using a large language model (LLM) These terms cover categories of objects commonly found in everyday life.
[0038] Extract object categories from existing public self-centralized datasets to form gather.
[0039] merge and To form an initial core noun set = .
[0040] for For each noun 'o' in the code, the Visual Language Model (VLM) is invoked to automatically generate its structured configuration file. Object-Action Associated Caching Build a basic knowledge base This process effectively builds a comprehensive foundational knowledge base, which serves as the core knowledge base when the system starts up.
[0041] Model loading: The pre-trained Qwen3-VL-8B-Instruct model is loaded as the core VLM for each agent, which is used to perform specific visual-language reasoning tasks.
[0042] Load the hand-object segmentation model (VISOR-HOS) to extract the interaction area between the hand and the object from video frames.
[0043] Strategy library (reflection) initialization: Simultaneously, a hierarchical strategy memory bank is initialized for reflection, divided into three levels (level one, level two, and level three), corresponding to "coarse-grained object-scene matching strategy", "medium-grained hand-object interaction mode strategy", and "fine-grained action sequence and intent inference strategy", respectively.
[0044] The initial reflection can be empty, or a cold start can be performed using a general prior strategy generated from a small amount of labeled data / expert knowledge / LLM. The strategy entry format includes: category, content (strategy text description), frame_strategy (clue template for frame-level visual evidence), and frequency (verification pass frequency, initially 1 or a preset value).
[0045] 2. Input Processing When the system receives a self-centered video input, it first uses a visual context engine for preprocessing to extract useful visual features.
[0046] The spatial configuration of hands and objects, and their dynamic interaction, constitute the essential characteristics of first-person perspective action. To address the challenge of combinatorial explosion, we decompose videos into basic <verb-noun> interaction units. This strategy transforms the problem into combinatorial structural units, freeing the system from the constraints of memorizing holistic concepts, thereby enabling it to learn generative behavioral patterns applicable to new objects.
[0047] Specifically, we constructed an explicit visual context. This is used to guide the agent to focus on interaction primitives. Specifically, given an input video V, we first employ a temporal sampling strategy to temporally sample the input video V and extract T frames of RGB images I. The sampling frequency is adjusted according to actual needs to balance computational efficiency and information integrity.
[0048] Each frame of image is processed using a pre-trained hand-object segmentation model. The parts are segmented to obtain mask information for the hand and the object.
[0049] For each frame Format the segmentation results as visual cues (Prompt Mask) enhances the accuracy of perception by filtering background noise and labeling the positions of hands and objects. in Representative to Perform a segmentation operation. Represents background pixels, Representative category The bounding box, This indicates that when b is identified in the bounding box When inside, add category label c to b. This means that when p is identified as the background, the background is removed. Specifically, this formatting involves two key operations: (1) noise filtering: setting the background to black to eliminate environmental interference; (2) explicit labeling: to eliminate entity ambiguity, labeling 'H' (hand) and 'O' (object) in the upper left corner of the corresponding bounding box.
[0050] Ultimately, the visual context is constructed as a set of two-source pairs: This formula ensures that the agent receives data from the original frame. To obtain rich appearance details, and from the interaction mask This provides clear structural guidance.
[0051] 3. Multi-agent cooperative reasoning The core of EgoDirector lies in its multi-agent collaborative reasoning mechanism, which achieves a complete process from video input to action recognition through the collaborative work of three agents: OPA, ARA, and APA.
[0052] Let V represent the input first-person perspective video. This framework utilizes two sets of dynamic contexts: the visual context extracted from the VCE. and semantic knowledge maintained by SCE Combine with relevant strategies in the reflection strategy library It provides structured configuration files and self-evolution capabilities for each agent; the inference process adopts a progressive decomposition strategy and is modeled as a sequential mapping process. Each arrow represents a reasoning step performed by a specific agent that performs conditional mapping based on a shared visual and semantic context. Specifically, OPA first analyzes visual information V to identify a set of interactive objects. Subsequently, ARA deduced that... Relevant reasonable action assumption space Finally, APA integrates multimodal evidence to rank these hypotheses and generate the final activity prediction. This divide-and-conquer approach provides a structured foundation for handling combinatorial explosion problems. EgoDirector's core innovation lies in the collaborative design of its context engine. The Visual Context Engine (VCE) provides basic visual units that combat perceptual uncertainty through hand-object interaction segmentation. The SCE, acting as a self-expanding knowledge base, stores structured configuration files. Crucially, the SCE possesses self-evolutionary capabilities: it automatically integrates new objects and knowledge, forming a continuously evolving context ecosystem, thereby mitigating context collapse and enabling continuous updates without parameter training.
[0053] Specifically, the intelligent agents include: Object-Aware Agent (OPA): An OPA is the basic perceptual unit responsible for deconstructing the visual environment into semantic entities. Its main purpose is to synthesize semantic entities from... Interaction-centric visual cues and from Previous candidate noun set Extract a set of ranking interaction objects, denoted as .
[0054] The workflow includes: Receive visual context (Including RGB frames and visual cues) and a candidate noun set N from SCE.
[0055] The core VLM performs condition generation and ranking tasks. Through interaction center filtering, hybrid recognition strategy and evidence-based ranking, it generates and outputs the top-ranked set of interaction objects O.
[0056] Formally, the perception process is modeled as a conditional generation and sorting function: Where K is a fixed output size (e.g., K=5). Upon receiving the visual context, OPA invokes its core VLM to perform this analysis based on three strict constraints: (1) Interaction-centric filtering: strictly instructing the agent to focus on The areas highlighted by the "H" and "O" masks identify objects that meet the "clear physical interaction" standard while suppressing background noise. (2) Based on the strategies provided in the provided strategy library The perception of the guided object (3) Hybrid recognition strategy: In order to balance accuracy and generalization ability, the agent adopts a hybrid search strategy. It is based on the candidate list of SCE ( The matching objects are prioritized. However, to accommodate open-world settings, the generation of new terms is explicitly allowed when strong visual evidence indicates objects outside the predefined set. (4) Evidence-based ranking: Output list The visual interaction evidence is sorted in descending order of strength to ensure that the most prominent objects are processed first by the downstream agents.
[0057] To achieve seamless collaboration, the extracted object set Encoded as a machine-readable JSON object (e.g., This structured output serves as a clear semantic interface, transmitting the decoupled perception results to the downstream ARA.
[0058] Inject reflective strategies into reasoning prompts: Extract the top k most frequent strategies from the level one of reflections and use them as part of few-shot examples or system prompts to guide OPA to pay more attention to frequently verified object occurrence patterns and typical interaction cues, thereby improving the recall rate of rare / long-tail objects in open worlds.
[0059] Action-Based Inference Agent (ARA): As a specialized inference engine, ARA is responsible for bridging the gap between detected objects and potential actions. Unlike simple classification, ARA addresses the "action-object" combinatorial explosion problem by generating a comprehensive, structured action hypothesis space, represented as... .
[0060] The workflow includes: Receives an object set O from the OPA and dynamic visual cues from the VCE. and structured prior knowledge from SCE .
[0061] For each object o∈O, retrieve its object-action association cache. and concept action configuration files Combining visual evidence and semantic associations, from The strategy library provides strategies to generate the most likely action verb v for each object.
[0062] Reflection (Level two) includes: injecting the top k most frequent medium-granularity hand-object interaction strategies into cues to help ARA quickly match typical interaction patterns (such as "holding a knife → cutting", "pinching a soft object → kneading"), reducing the illusion of rare actions.
[0063] The reasoning process follows the forced generation principle. Formally, the agent constructs the hypothesis space by inferring the most reasonable operational verb v for each object provided by the OPA. : This strategy aims to maximize recall. Even when visual interaction evidence is ambiguous (e.g., occlusion issues), the agent can rely on scene context and common-sense knowledge to infer the most likely action, preventing premature rejection of potential real-world activities.
[0064] The output is formatted as a strictly defined, machine-parseable JSON list wrapper (e.g., ...). This standardized interface provides a computable candidate set for downstream APAs. This transforms the open-ended identification problem into a ranking and verification task.
[0065] Activity Prediction Agent (APA): The APA is the final arbiter and decision-maker. Unlike the upstream agent, which focuses on generating candidates, the APA is responsible for the action hypotheses provided by the ARA. The key evaluation and global ranking. Formally, the agent aggregates a set of modules from all the preceding modules ( , The complete pool of evidence derived from the intermediate inference states of OPA and ARA, and The strategy provided here aims to map the hypothesis space to a list of ranking activities. .
[0066] The workflow includes: Receive intermediate inference states from VCE, SCE, OPA, and ARA.
[0067] Reference reflection level three: Use high-frequency fine-grained action sequences and intent strategies as counterfactual checks or ranking weighting criteria to help APA eliminate semantically incoherent but visually plausible erroneous assumptions.
[0068] Based on three criteria—visual-hypothesis alignment, semantic plausibility, and confidence aggregation—each candidate action in the hypothesis space H is scored and ranked to generate the final activity prediction. .
[0069] in This represents the confidence score for activity a. Scoring function. Each candidate was evaluated based on three priority criteria: (1) Visual hypothesis alignment: verifying whether the proposed action is aligned with the visual hypothesis alignment. The captured hand interaction patterns are visually aligned (e.g., rejecting actions incompatible with hand grip). (2) Semantic rationality: utilizing It checks the logical consistency of action-object combinations in a given scene context (e.g., rejecting "washing books" even if visual cues are unclear). (3) Confidence aggregation: It combines the detection strength of OPA and the generation likelihood of ARA to form a unified confidence metric. Operationally, APA builds an overall evaluation cue for its embedded VLM to calculate these scores. The final output is a strictly formatted JSON array containing a list of ranked activities (e.g., This probabilistic ranking enables the system to transparently address the ambiguity of the open world, providing a measurable basis for the final decision.
[0070] 4. Dynamic Context Adaptation To address the uncertainty in open-world environments, EgoDirector introduces a dynamic context adaptation mechanism, which continuously updates knowledge and strategies through the self-evolution capabilities of SCE (Semantic Context Evolver) and a reflective online experience replay and refinement mechanism.
[0071] New object recognition and knowledge base update (original content retained): During the reasoning process, if OPA identifies a new object, , making If this happens, the SCE's self-evolution mechanism will be triggered immediately.
[0072] During the evolution process: The system automatically invokes VLM to create new structured configuration files in real time based on the name and visual description of the new object. And associated cache C_new.
[0073] Seamlessly integrate the newly generated knowledge components into the existing knowledge base, i.e. ← This mechanism ensures that the system continuously accumulates knowledge, enabling it to use newly acquired information to make accurate inferences in subsequent tasks.
[0074] Online updates and refinement of reflection strategies (with new core components): In the Reflective Strategy Library (RPA) module: (1) Initialize the reflection policy library and pre-store 3 empty initialization policies; the reflection data structure adopts a three-level classification storage ('one' / 'two' / 'three'), where 'one' corresponds to the policy of the object-aware agent (OPA). "two" corresponds to the policy of the Action Reasoning Agent (ARA). "three" corresponds to the policy of the Activity Prediction Agent (APA). Each level stores a policy dictionary, with the policy ID as the key and the following values: category: action category, content: policy text description, and frequency: frequency of large model feedback.
[0075] (2) During the action recognition process, based on the frequency, the three policies with the highest feedback frequency are retrieved from the reflection policy library and applied. Input is given to the object-aware intelligent agent. Input to the action reasoning agent Input is given to the Activity Prediction Agent (APA); (3) Based on the application effect of the strategy and newly identified objects and actions, the reflection strategy library is dynamically updated, including adding strategies, modifying existing strategies, or deleting invalid strategies; RPA generates reflections for the first three agents based on the outputs of the first three agents and the outputs of the Visual Context Engine (VCE) and Semantic Context Engine (SCE), respectively. , , .
[0076] (4) Provide a strategy merging and similarity calculation mechanism to ensure that the strategies in the rethinking strategy library are both comprehensive and efficient, avoiding redundancy. When there are more than 200 strategies in the strategy library, the merging mechanism will be triggered. The similarity of each strategy will be calculated pairwise using Qwen3-Embedding-0.6B. When the similarity is greater than 0.85, the strategies will be merged, frequency will be incremented by 1, strategies with higher original frequency will be retained, and strategies with lower frequency will be deleted. If the total number of strategies is still greater than 50 after merging, strategies with a frequency ranking after 50 will be truncated and all subsequent strategies will be deleted.
[0077] The workflow includes: After a complete inference cycle, the system inputs all outputs from OPA, ARA, and APA, along with all outputs from the visual context engine and semantic context engine, into the reflexive policy agent to extract high-quality policy recommendations at three levels: Level One: Object-Scene Association and Priority Clues Level Two: Hand-Object Interaction Patterns and Typical Verb Triggering Conditions Level Three: Action Sequences, Causal Relationships, and Intent Inference Templates The three strategies output in this inference are added to the corresponding level of reflection in the form of frequency=1 (if there is an ID conflict, it is considered a new variant).
[0078] Frequency accumulation and semantic merging: When the number of strategy entries at any level exceeds a preset threshold (e.g., 200), the merging and refining process is triggered.
[0079] Calculate text embeddings for all strategies at this level (using a high-performance sentence embedding model, such as qwen3-embed or an equivalent model), and compute pairwise cosine similarity.
[0080] Traverse in descending order of frequency, starting with the most frequent strategy: if a subsequent strategy has a similarity ≥ 0.85 with the current strategy and the same category, it is considered merging. The frequencies are accumulated, and the strategy with the highest original frequency is retained as the representative, while the rest are discarded (to achieve automatic deduplication and elimination of low-value strategies).
[0081] If the merged data still exceeds the maximum capacity (e.g., 50 entries), truncate by frequency and keep only the top entries.
[0082] Through the above mechanism, the reflection gradually converges into a refined, high-frequency, effective policy memory bank that covers long-tail scenarios in the open world, and serves as a strong prior in subsequent inference to continuously improve the zero-shot / few-shot generalization ability of OPA, ARA, and APA.
[0083] 5. Output Generation After multi-agent collaborative reasoning and dynamic context adaptation, EgoDirector generates the final activity prediction results.
[0084] Result formatting: The final activity prediction A* is formatted as a JSON array containing the identified actions and their confidence scores, for example: {"actions": [{"noun": "onion", "verb": "cut", "action": "cut onion", "confidence": 0.95}, ...]}.
[0085] Output transmission: The formatted results are then passed to downstream applications or systems, such as robot control modules or augmented reality interfaces, to enable corresponding interactions or feedback.
[0086] Through the above specific implementation methods, EgoDirector achieves high efficiency, accuracy, and adaptability in open-world autocentric action recognition, providing strong support for the development of embodied intelligence and augmented reality technologies.
[0087] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.
[0088] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A training-free agent context adaptation method for open-world self-perspective action recognition, characterized in that, The method includes: Initialize the knowledge base and reflection strategy library, and configure the intelligent agent and runtime environment; The received video is preprocessed, each frame of the image is segmented, and the segmentation results are formatted into visual cues, so that the visual cues and the images form a two-source pair as visual context. The intelligent agent performs reasoning, and the knowledge base is updated when new objects are identified during the reasoning process. The reasoning process includes: Using an object-aware agent, we synthesize interaction-centric visual cues from the visual context and a set of candidate nouns from a knowledge base to extract a ranked set of interactive objects. The action reasoning agent receives a set of objects, visual context, and structured prior knowledge from a knowledge base. For each object, it retrieves its verb-noun association cache and concept-action profile, combines visual evidence and semantic association with policies provided by a policy library, generates hypothetical action verbs for each object, and constructs a hypothesis space based on the action verbs. The activity prediction agent aggregates visual context, candidate noun set of knowledge base, interactive objects and hypothesis space, scores and ranks each candidate action in hypothesis space, and generates prediction results. The reflective policy library is updated after each round of reasoning by the agent; the final prediction result is formatted as a JSON array and output.
2. The method for training-free agent context adaptation for open-world self-perspective action recognition according to claim 1, characterized in that, The initial knowledge base includes: Generate a set of high-frequency everyday nouns using a large language model ; Extract object categories from existing public self-perspective datasets to form Set; merge and To form an initial core noun set = ; for For each noun 'o' in the code, a structured configuration file is automatically generated using a visual language model. Caching associated with verbs and nouns Build a basic knowledge base ; in This represents a set of known concepts, where 'o' represents a key action or object. For concept action files; Associate cache with object actions; The concept action profile includes visual appearance features, action and interaction patterns, and scene context; The verb-noun association cache maps the noun 'o' to a set of potential verbs. As A quick index and supplement.
3. The training-free agent context adaptation method for open-world self-perspective action recognition according to claim 2, characterized in that, The updating of the knowledge base includes: During the reasoning process, if the object-aware agent recognizes a new term... , making The system will then automatically invoke the visual language model and create a new structured configuration file in real time based on the name and visual description of the new term. and associated cache This allows for the seamless integration of newly generated knowledge components into the existing knowledge base. ← This mechanism ensures that the system continuously accumulates knowledge, enabling it to use newly acquired information to make accurate inferences in subsequent tasks.
4. The method for training-free agent context adaptation for open-world self-perspective action recognition according to claim 1, characterized in that, The reflection strategy library includes: Initialize a hierarchical policy memory reflection, which is divided into three levels, corresponding to the coarse-grained object-scene matching strategy used by the object perception agent, the medium-grained hand-object interaction pattern strategy used by the action reasoning agent, and the fine-grained action sequence and intent inference strategy used by the activity prediction agent. The initial reflection strategy is empty, or a cold start is performed from a small amount of labeled data, expert knowledge, or a general prior strategy generated by LLM. The strategy entry format includes: category, strategy text description content, frame_strategy cue template for frame-level visual evidence, and frequency of validation pass.
5. The training-free agent context adaptation method for open-world self-perspective action recognition according to claim 4, characterized in that, The update of the reflection strategy library includes Based on the effectiveness of the policy application and newly identified objects and actions, the reflective policy library is dynamically updated, including adding policies, modifying existing policies, or deleting invalid policies; reflective policies are generated for the three agents respectively based on the outputs of the three agents, the visual context, and the structured prior knowledge of the knowledge base. The strategy generated by the agent is added to the corresponding level of the reflective strategy in the form of a pass rate of 1. Frequency accumulation and semantic merging: When the number of strategy entries at any level exceeds a preset threshold, a merge and refinement process is triggered. Calculate the text embeddings for all strategies at this level and calculate the pairwise cosine similarity. Traverse the strategies in descending order of verification pass frequency, starting with the most frequent strategies: if the similarity between a subsequent strategy and the current strategy is greater than a preset value and they belong to the same category, they are considered to be mergeable. The verification pass frequency is accumulated, and the strategy with the highest original verification pass frequency is retained as the representative, while the rest are discarded. If the merged system still exceeds the maximum capacity, the remaining capacity will be truncated based on the frequency of successful verification.
6. The training-free agent context adaptation method for open-world self-perspective action recognition according to claim 1, characterized in that, The preprocessing includes: Time-sample the input video V and extract the T-frame RGB image I. ; Each frame of image is processed using a pre-trained hand-object segmentation model. Segmentation is performed to obtain mask information for the hand and the object; For each frame Format the segmentation results as visual cues : in Representative to Perform a segmentation operation. Represents background pixels, Representative category The bounding box, where H represents a hand and O represents an object; This indicates that when b is identified in the bounding box When inside, add category label c to b. This means removing the background when p is identified as the background. Ultimately, the visual context is constructed as a set of two-source pairs: 。 7. The method for training-free agent context adaptation for open-world self-perspective action recognition according to claim 6, characterized in that, The specific workflow of the object-aware intelligent agent is as follows: Receive visual context And the candidate noun set N of the knowledge base; Upon receiving the visual context, the visual language model is invoked to perform analysis based on three strict constraints: Interaction-centric filtering: Strictly instructing agents to focus on The H and O masks highlight the areas that meet the criteria for clear physical interaction, while suppressing background noise; Based on the provided policy library, the top k most frequent policies corresponding to the object-aware agent are extracted as part of the system prompts, guiding the object-aware agent to pay more attention to the frequently verified object occurrence patterns and typical interaction cues. Hybrid Recognition Strategy: To balance accuracy and generalization ability, the agent employs a hybrid search strategy, based on a candidate list from the knowledge base. Prioritize matching objects; to suit open-world settings, explicitly allow the generation of new terms when strong visual evidence indicates objects outside a predefined set. ; Output list The extracted object set is sorted in descending order of the strength of visual interaction evidence. Encoded as a machine-readable JSON object.
8. The training-free agent context adaptation method for open-world self-perspective action recognition according to claim 7, characterized in that, The specific workflow of the action reasoning agent is as follows: integrating the detected object set O and dynamic visual cues from the object perception agent. Structured prior knowledge of the knowledge base And strategies from the Reflective Strategy Library The hypothesis space is constructed by inferring the most reasonable operation verb v for each object provided by OPA. For each object o in O, the AI retrieves its corresponding object-action association cache and concept-action profile from the knowledge base, and the output is formatted as a strictly defined, machine-parseable JSON list wrapper.
9. A training-free agent context adaptation method for open-world self-perspective action recognition according to claim 8, characterized in that, The workflow of the activity prediction agent includes: performing conditional scoring and ranking. in This represents the confidence score for activity a. To reflect on the strategy library, high-frequency fine-grained action sequences and intention strategies are used as counterfactual checks or ranking weights to eliminate erroneous assumptions; scoring function Each candidate is evaluated based on the following three priority criteria: Visual-hypothesis alignment: Validating whether the proposed action is visually aligned with the hypothesis. The captured hand interaction pattern matches; Semantic rationality: utilizing prior knowledge Check the logical compliance of the action-object pair in the given scene context; Confidence aggregation: Combine the detection intensity from the object-aware agent and the generation likelihood from the action-reasoning agent to form a unified confidence metric; The final output is a strictly formatted JSON array containing a list of activities sorted in descending order of confidence level.
10. A system for implementing the method according to any one of claims 1-9, characterized in that, include: Object-aware agent, action-reasoning agent, activity-predicting agent, and reflective policy agent; visual context engine, semantic context engine, and reflective policy library; The reflective strategy agent generates reflective strategies for the three agents based on the outputs of the three agents, the visual context, and the structured prior knowledge of the knowledge base, thereby updating the reflective strategy base. The visual context engine eliminates background interference to create clean, focused visual inputs for key interactions, namely hands and objects. The semantic context engine is used to build and update the knowledge base; The reflexive strategy library is used to provide strategy suggestions during agent reasoning.