An agent training method for embodied navigation decision understanding
By combining a panoramic, densely labeled dataset with an adaptive reward function, the problems of poor generalization ability and rigid decision-making of embodied navigation agents are solved, and the agents are able to navigate autonomously in complex environments.
Patent Information
- Application Number
- CN202511448998.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2025-10-11
- Publication Date
- 2026-03-20
- Estimated Expiration
- 2045-10-11
AI Technical Summary
In existing technologies, embodied navigation agents suffer from poor generalization ability and rigid decision-making during training, making them unable to perform effective autonomous navigation in complex environments.
By generating a panoramic, densely labeled navigation training dataset and an adaptive interval-aware hybrid reward function, the agent is guided to shift from imitating a single path to understanding the decision space.
The intelligent agent exhibits excellent generalization performance and robustness in complex environments, and is able to make reasonable navigation decisions in unseen environments, achieving true autonomous navigation.
Smart Images
Figure CN121257641B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the field of embodied artificial intelligence, machine learning and robotics, in particular to an agent training method for embodied navigation decision understanding, aiming to improve the generation of training data for the autonomous navigation ability of robots or virtual agents and a reward function assignment method used in the reinforcement learning framework. BACKGROUND
[0002] Autonomous navigation of embodied agents is a basic ability to achieve their tasks in complex and unknown environments. Currently, the mainstream technology paradigm for training large visual-linguistic models to empower embodied navigation agents is imitation learning, that is, the agent learns and replicates a single "optimal" trajectory predefined by an expert. This paradigm is called "path imitation".
[0003] However, the "path imitation" paradigm has a fundamental technical flaw. It simplifies a complex, continuous decision-making embodied navigation task into a sequence-to-sequence replication of a single correct path. This greatly limits the agent's exploration ability and generalization ability in new environments. Real-world embodied navigation scenarios are often open and variable, and there are often multiple equally effective paths from point A to point B. Rigidly imitating the only path prevents the agent from understanding the subtle differences between different choices, suppressing its initiative to explore other feasible solutions and compromising its decision-making robustness in unseen scenarios. Therefore, the agent trained in this way is essentially a "repeater" of a path, rather than a "decision maker" who can think and judge independently, making it difficult to cope with any unexpected situations other than the preset trajectory in practical applications, and unable to achieve true autonomous navigation. SUMMARY
[0004] The purpose of the present application is to overcome the series of problems such as poor generalization ability and rigid decision-making caused by the "path imitation" paradigm in the prior art, and to provide an agent training method for embodied navigation decision understanding, aiming to guide the agent to transition from the low-level ability of "imitating a single path" to the high-level intelligence of "understanding the decision space".
[0005] The specific technical solutions to achieve the purpose of the present application are as follows:
[0006] An agent training method for embodied navigation decision understanding, the method comprising the following specific steps:
[0007] Step 1: Generate a navigation training dataset with panoramic dense annotation
[0008] A1: Generation of feasible candidate actions and definition of state
[0009] In a three-dimensional simulation environment (such as Habitat-sim), for an agent at any time step... status (including its position and attitude) and first-person visual observation First, an "action proposal module" analyzes its sensor data (such as depth images) to generate a set of all physically and visually feasible candidate actions. Each candidate action represents a high-level movement command, such as (turn 30 degrees, move forward 2 meters), and is visualized as a numbered arrow in the agent's field of vision.
[0010] A2: Panoramic Dense Annotation and Trajectory Data Acquisition
[0011] This is a crucial step in the process. First, for the candidate action set... Every action in The system invokes a "global A* planner" that possesses global map information of the environment, and calculates the potential cost of reaching the goal after performing the action using the following function:
[0012]
[0013] in, The function is based on the current attitude. and actions Predict the new landing posture after implementation; It represents the final target location; the A* function calculates the shortest geodesic distance from the new landing attitude to the target point. ;
[0014] Subsequently, the agent typically selects the action with the shortest geodesic distance to generate an optimal navigation trajectory. During this process, to enrich the diversity of the dataset and better train decision understanding capabilities, a backtracking mechanism is introduced when the agent encounters multiple suboptimal actions with similar geodesic distances at a decision point—a decision ambiguity point. This backtracking mechanism reflects the state of that decision point. The complete distance vector will be recorded; after the current navigation task ends, the agent will return to these recorded fuzzy decision points and actively execute one of the unexplored suboptimal actions to generate a new navigation trajectory branch, and perform panoramic dense annotation on each step of the navigation trajectory branch.
[0015] A3: Structured storage of data samples
[0016] The agent at time step First-person visual observation The set of all candidate actions a complete vector composed of the action and the geodesic distance corresponding to each action , which are combined together into a structured training data sample and stored in the final dataset; through the above process, a dataset containing multiple effective paths and intensive supervision signals is finally formed;
[0017] Step 2: Set up the reward function
[0018] Reward function is composed of two parts, where is an action selection response generated by the agent policy; the first part is the continuous basic score , regardless of the action selected by the agent, a basic score will be obtained; the score is calculated by applying the Softmax function to the negative values of the geodesic distances of all candidate actions, and the mathematical expression is:
[0019]
[0020] where is the geodesic distance of the selected action, is a temperature hyperparameter used to control the sharpness of the score distribution; this part of the reward provides a smooth, relative ranking of all options, even if the action is not optimal, it can also get the corresponding score according to its good or bad degree;
[0021] The second part is the dynamic reward bonus modulated by decision certainty, which is the key innovation of this reward function, which makes the reward signal "intelligent". First, the agent will calculate a "decision certainty factor" to quantify the degree of clarity of the current decision; the factor is defined by the normalized distance difference between the optimal action, i.e. the closest distance, set as and the suboptimal action, i.e. the second closest distance, set as :
[0022]
[0023] where, is a small constant to prevent the denominator from being zero; if the optimal option is significantly better than all other options, i.e. the gap is large, the value is close to 1, indicating that this is a "high certainty" decision point; on the contrary, if multiple options are similar in quality, i.e. the gap is small, the value is close to 0, indicating that this is a "low certainty" or "ambiguous" decision point;
[0024] Reward function is composed of the basic score and the dynamic bonus, and the complete expression is:
[0025]
[0026] wherein, an index representing the optimal action, is an indicator function, which is 1 when the condition is true, and 0 otherwise, is a pre-set maximum reward bonus coefficient;
[0027] Step 3: Reinforcement fine-tuning based on agent decision understanding training
[0028] Reinforcement fine-tuning (RFT) is performed on the agent using the dataset generated in step 1 to align the decision understanding capability; specifically:
[0029] C1: Response generation and scoring
[0030] During the training process, for a state given by the environment, the agent generates a set (e.g., G) of different candidate output sequences, i.e., corresponding to selecting different actions;
[0031] C2: Reward function mechanism
[0032] For each generated action, the corresponding geodesic distance label in the dataset is used to calculate its reward value through an interval perception hybrid reward function; the function consists of two parts: a continuous basic score calculated by a softmax function based on all action distances, and an additional reward dynamically adjusted by decision certainty (distance gap between optimal and suboptimal actions), which is triggered only when the agent selects the optimal action.
[0033] C3: Strategy update
[0034] Finally, a group-level reward strategy optimization algorithm (such as GRPO) is used to update the network parameters of the agent according to the reward value; the goal of the optimization algorithm is to maximize the expected total reward of the set of actions generated by the agent, thereby guiding the agent to learn to evaluate the relative merits of all available paths, achieving the transition from "path imitation" to "decision understanding".
[0035] The core of the present application lies in two mutually supporting and closely related technical innovations:
[0036] A novel dataset generation process that provides panoramic supervision signals: the present application proposes a completely new dataset construction method, the core of which is to provide accurate and quantitative merit evaluation for all feasible options of the agent at each decision point, thereby constructing a training dataset containing dense supervision signals. This dataset provides the necessary data foundation for the agent to learn how to make complex decision trade-offs.
[0037] An adaptive interval-aware hybrid reward assignment method used with the data set: the present application proposes a novel reward function specially designed to effectively utilize the rich panoramic supervision information in the above-mentioned data set. During the training process of the agent, the function can adaptively provide dynamic feedback signals according to the difficulty and certainty of the current decision, thereby achieving efficient and accurate shaping of the decision-making ability of the agent.
[0038] The present application has the following beneficial effects:
[0039] The application of the two core innovations of the present application to the training process of the agent can bring the following significant beneficial effects: the agent realizes the paradigm shift from imitation to understanding, the agent learns to evaluate the relative value of all options, and no longer memorizes the path, but truly learns how to reason and weigh in the complex decision-making space; the agent has excellent generalization performance, as it masters the general decision evaluation ability rather than the path memory specific to the training environment, the agent can make reasonable navigation judgments in completely new environments that have never been seen before, and exhibits strong generalization ability and robustness; the agent is provided with an efficient strategy alignment mechanism, the interval-aware hybrid reward function provides more rich, effective and adaptive learning signals than the traditional binary (right / wrong) reward or simple distance reward, significantly accelerating the alignment process of the model strategy to the ultimate goal of "intelligent decision-making". BRIEF DESCRIPTION OF DRAWINGS
[0040] Figure 1 The present application is a whole flowchart of the method;
[0041] Figure 2 The present application is a flowchart of reward calculation and strategy update in the reinforcement fine-tuning stage. DETAILED DESCRIPTION
[0042] In order to facilitate the understanding of the present application, the present application will be described in detail below in combination with the drawings and examples.
[0043] The specific technical solutions of the present application are described as follows:
[0044] Step 1: Generation of feasible candidate actions
[0045] In a three-dimensional simulation environment (such as habitat-sim), for the state (including its position, pose and first-person visual observation) of the agent at any time step , first analyze its sensor data (such as depth image) through an "action proposal module" to generate a set of physically and visually feasible candidate actions Each candidate action represents a high-level movement command, such as (turn 30 degrees, move forward 2 meters), and is visualized as a numbered arrow in the agent's field of vision;
[0046] Step 2: Panoramic Dense Annotation
[0047] This is a crucial step in the process. For the candidate action set generated in step 1... Every action in Each agent invokes a "global A* planner," which possesses a global map of the environment. This planner calculates the optimal course of action for the agent. Then, the shortest geodesic distance from its new location to the final target. This distance provides a precise and objective quantification of the merits of the action.
[0048] Step 3: Structured storage of data samples
[0049] The agent at time step Visual observation, the set of all candidate actions And a complete vector consisting of the geodesic distance corresponding to each action. These are combined to form a structured training data sample.
[0050] Step 4: Dataset Augmentation Based on Backtracking Mechanism
[0051] To further enrich the dataset, this process also includes a backtracking mechanism. When the agent encounters multiple suboptimal actions with similar geodesic distances at a decision point (i.e., a decision ambiguity point), the agent records its current state. After the current embodied navigation task ends, the agent returns to these recorded state points and executes other feasible paths that were not previously selected, thereby exploring and recording more diverse but equally effective navigation trajectories.
[0052] Through the above process, the dataset generated by this invention is no longer a "standard answer set" containing only a single correct answer, but a "decision quality reference table" that provides quantitative evaluation of all options for each decision point, providing unprecedentedly rich supervisory information for the agent to learn and understand the decision space.
[0053] The second core aspect of this invention is a novel reward function specifically designed to effectively utilize the panoramic supervision signals in the aforementioned dataset during the reinforcement learning training phase. Its core idea is adaptability: the nature and intensity of the reward should be dynamically adjusted based on the "determinism" of the decision itself.
[0054] The reward function It consists of two parts, among which is the response of the agent policy generation, which includes action selection The first part is the continuous base score Regardless of which action the agent chooses, it will get a base score. This score is calculated by applying a Softmax function to the negative values of the geodesic distances of all candidate actions, and its mathematical expression is:
[0055]
[0056] where is the geodesic distance of the selected action, is a temperature hyperparameter that controls the sharpness of the score distribution. This part of the reward provides a smooth, relative ranking of all options, even if the non-optimal action can also get a score according to its good or bad degree.
[0057] The second part is the dynamic reward bonus modulated by decision certainty, which is the key innovation of this reward function, which makes the reward signal "intelligent". First, the agent will calculate a "decision certainty factor" to quantify the degree of clarity of the current decision. This factor is defined by the normalized distance difference between the optimal action (closest distance, set as ) and the suboptimal action (second closest distance, set as ):
[0058]
[0059] where, is a small constant to prevent the denominator from being zero. If the optimal option is significantly better than all other options (gap), the value is close to 1, indicating that this is a "high certainty" decision point. Conversely, if multiple options are similar in quality (small gap), the value is close to 0, indicating a "low certainty" or "ambiguous" decision point.
[0060] The final hybrid reward is composed of the base score and this dynamic bonus, and its complete expression is:
[0061]
[0062] where, represents the index of the optimal action, is an indicator function (1 when the condition is true, otherwise 0), is the preset maximum reward bonus coefficient.
[0063] The ingenuity of this design lies in: in the high certainty scenario ( high), if the agent chooses the only optimal action, it will get a high base score plus an extra large reward bonus, forming a very strong positive learning signal. In low certainty / ambiguous scenarios (value low), the dynamic reward bonus part is almost zero. At this time, the reward is mainly determined by the base score. For several equally good choices, their base scores will be close, and the agent will not be punished for not choosing that "optimal but not outstanding" option. This effectively protects and encourages the agent to make valuable exploration, rather than stubbornly favoring a certain option in ambiguous situations.
[0064] Embodiments
[0065] Reference Figure 1 , the whole process of the embodiment.
[0066] S100: First, data preparation is performed to construct a dataset containing two subsets of SFT and RFT. This dataset generates an RFT dataset containing about 11000 trajectories in the habitat-sim simulation platform using HM3D and other scene datasets. At the same time, in order to solve the cold start problem of subsequent training, an SFT dataset containing the complete thinking chain and decision of the successful embodied navigation of a powerful teacher model Qwen-QvQ is additionally generated.
[0067] S110: Perform policy initialization. A pre-trained visual-linguistic model is fine-tuned on the SFT subset. The data format of the SFT subset is , where is the text sequence generated by the teacher model containing the reasoning process and the final action <think> ...< / think> <answer> k< / answer> This step aims to enable the model to learn basic environmental understanding and behavior patterns.
[0068] S120: Perform policy alignment (RFT). This step is the core of the invention, and its internal process is described in detail in Figure 2 .
[0069] S130: Deployment and navigation. The model trained in S120 is deployed on the agent to enable it to receive visual input and high-level goals in real or simulated environments and make autonomous decisions and navigate.
[0070] Reference Figure 2 , the reinforcement fine-tuning sub-process in step S120 of the invention.
[0071] S200: Sample response generation. For an input in the RFT dataset, use the current policy to sample a different output sequence
[0072] S210: Calculate interval-aware hybrid reward. For each generated sequence , parse the selected action . Then, using the pre-calculated geodesic distance vector in S100, calculate the reward of the action through the interval-aware hybrid reward function. For specific calculations, refer to the corresponding part of the specification.
[0073] S220: Update the policy. Based on the reward calculated in S210, use the objective function of the GRPO algorithm to perform gradient update on the parameters of the policy . Repeat S200 to S210 until the model converges.
Claims
1. A method for training an intelligent agent for embodied navigation decision understanding, characterized in that, The method includes the following specific steps: Step 1: Generate a navigation training dataset with panoramic dense annotations A1: Generation of feasible candidate actions and definition of state In a three-dimensional simulation environment, for an agent at any time step status Including its position and attitude and first-person visual observation First, an "action proposal module" analyzes the sensor data to generate a set of all physically and visually feasible candidate actions. Each candidate action represents a high-level movement command and is visualized as a numbered arrow in the agent's view. A2: Panoramic Dense Annotation and Trajectory Data Acquisition First, for the candidate action set Every action in It invokes a "global A* planner" that possesses global map information of the environment, and calculates the potential cost of reaching the goal after performing the action using the following function: ； in, The function is based on the current attitude. and actions Predict the new landing posture after implementation; It represents the final target location; the A* function calculates the shortest geodesic distance from the new landing attitude to the target point. ; Subsequently, the agent selects the action with the shortest geodesic distance to generate an optimal navigation trajectory. When the agent encounters multiple suboptimal actions with similar geodesic distances at a decision point (a decision ambiguity point), a backtracking mechanism is introduced: the state of that decision point... The complete distance vector will be recorded; after the current navigation task ends, the agent will return to these recorded fuzzy decision points and actively execute one of the unexplored suboptimal actions to generate a new navigation trajectory branch, and perform panoramic dense annotation on each step of the navigation trajectory branch. A3: Structured storage of data samples The agent at time step First-person visual observation The set of all candidate actions And a complete vector consisting of the geodesic distances corresponding to each action. These are combined to form a structured training data sample, which is then stored in the final dataset, forming a dataset containing multiple effective paths and dense supervision signals. Step 2: Set the reward function reward function It consists of two parts, among which It is a policy generation mechanism for intelligent agents that includes action selection. Response: Part 1, Continuous Basic Score Regardless of which action the agent chooses, it will receive a base score; this score is calculated by applying the Softmax function to the negative values of the geodesic distances of all candidate actions, and its mathematical expression is: ； in It is the geodesic distance of the selected action. It is a temperature hyperparameter used to control the sharpness of the score distribution; this part of the reward provides a smooth, relative ranking of all options, so that even non-optimal actions can receive corresponding scores according to their quality. The second part, dynamic reward enhancement for decision determinism modulation, begins with the agent calculating a "decision determinism factor". This factor is used to quantify the clarity of the current decision; it is determined by the optimal action, i.e., the closest one, and is set as . The second-best move, i.e., the one closest to the target, is denoted as... Defined by the normalized distance difference between them: ； in, It is a small constant to prevent the denominator from being zero; if the optimal option is significantly better than all other options, i.e., the difference is large, then... A value close to 1 indicates a "high-certainty" decision point; conversely, if multiple options are similar in merit (i.e., the differences are small), then... A value close to 0 indicates that this is a decision point with "low certainty" or "fuzzy" uncertainty; reward function It is composed of a base score and dynamic addition, and its complete expression is: ； in, The index representing the optimal action. It is an indicator function that returns 1 when the condition is true and 0 otherwise. It is the preset maximum reward bonus coefficient; Step 3: Training the agent's decision-making understanding based on reinforcement fine-tuning The agent is reinforced and fine-tuned using the dataset generated in step 1 to align its decision-making and understanding capabilities; specifically: C1: Response Generation and Scoring During training, given a state in the environment, the agent generates a set of different candidate output sequences, which correspond to different actions. C2: Mechanism of Reward Function For each generated action, its reward value is calculated using the geodesic distance label corresponding to it in the dataset and the interval-aware hybrid reward function. C3: Strategy Update Finally, a group-level reward strategy optimization algorithm is adopted to update the agent's network parameters based on the reward value. The goal of the optimization algorithm is to maximize the expected total reward of a set of actions generated by the agent, thereby guiding the agent to learn to evaluate the relative merits of all optional paths and realize the transformation from "path imitation" to "decision understanding".