A Long-Range Visual Language Navigation Method and System
By extracting structured navigation skills from successful trajectories and combining them with dynamic historical reconstruction, the problem of decision instability in long-range visual language navigation is solved, achieving more stable navigation decisions and reducing costs.
Patent Information
- Application Number
- CN202611261026.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-19
- Publication Date
- 2026-11-17
AI Technical Summary
In long-range visual-language navigation tasks, existing technologies struggle to handle issues such as task phase switching, repeated access, and premature stopping stably. They also lack reusable, triggerable structured navigation skills and dynamic history reconstruction mechanisms.
Structured navigation skills are extracted from the successful trajectories of the first navigation model to build an external navigation skill library. Combined with a dynamic historical reconstruction mechanism, applicable skills are triggered to provide decision constraints for the second navigation model, including landmark conversion, premature stop suppression, repeated access avoidance, and direction constraints.
It improves decision stability in long-range navigation, reduces training and deployment costs, enhances the transferability and cross-environment applicability of navigation skills, and reduces historical redundancy and dilution of critical information.
Smart Images

Figure CN123115288A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of artificial intelligence and robot navigation technology, specifically to a long-range visual language navigation method and system. Background Technology
[0002] Visual language navigation is a class of agent tasks that combine natural language understanding, visual perception, and sequential decision-making. Its goal is to enable an agent to progressively observe the surrounding scene, select reachable locations, and ultimately arrive at the target area described in the instructions, based on natural language commands given by the user, within an indoor or virtual environment.
[0003] Taking R2R indoor visual-language navigation as an example, the indoor environment is usually abstracted as a connected graph composed of multiple reachable viewpoints. At each step, the agent receives visual observations from the current viewpoint, candidate navigable viewpoints, and natural language instructions, and combines these with the already executed navigation trajectory to determine which candidate navigable viewpoint to move to next, or outputs a stop action when it believes it has reached the target location.
[0004] With the development of large language models and multimodal large models, existing methods have begun to explore the use of pre-trained models for visual language navigation. These methods typically organize natural language instructions, current observation summaries, candidate navigable viewpoints, and historical navigation records as text input. The model then generates the current reasoning process and the next navigation action through an alternating process of reasoning and action. Compared to traditional models that require dedicated training of navigation strategies, these methods can leverage the language understanding, common-sense reasoning, and task planning capabilities of pre-trained models to achieve zero-shot or few-shot navigation decisions to a certain extent.
[0005] In long-range visual-language navigation tasks, the navigation process often involves multiple rooms, multiple intermediate landmarks, and multiple stages of action. The agent not only needs to select the next candidate navigable viewpoint based on the current observation, but also needs to continuously determine which stage of the instruction it is currently in, which landmarks have appeared, which areas have been explored, whether it should enter a new space, whether it should avoid returning to a visited area, and whether the stopping conditions are met.
[0006] Relying solely on current observations and linear historical records is insufficient to reliably support the aforementioned long-term decision-making. On one hand, the agent needs to reuse certain local navigation skills, such as switching task phases after passing intermediate landmarks, suppressing stopping when the target is not confirmed, avoiding repeated visits, and selecting candidate navigable viewpoints based on direction words. On the other hand, the agent also needs to organize the continuously growing historical reasoning, action, and observation information to reduce historical redundancy and provide a clearer task progress status for skill triggering and action decisions.
[0007] Therefore, introducing triggerable, reusable, and constrainable navigation skills into the visual language navigation framework, and combining them with a dynamic history reconstruction mechanism to maintain the progress of long-term tasks, is an important direction for improving the decision-making stability of long-term indoor navigation agents.
[0008] One of the prior art techniques known to the inventor that is relatively close to this invention is the non-patent document "NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models". This document discloses a visual language navigation method based on large language models. Its main idea is to transform the visual language navigation task into a step-by-step reasoning and action selection problem that can be handled by large language models. In each navigation step, the system organizes information such as natural language navigation instructions, text descriptions of current environmental observations, historical navigation records, and future explorable directions into model inputs. The large language model then generates current state reasoning and selects the next navigation action.
[0009] Specifically, these methods typically employ an interactive decision-making framework similar to ReAct. In each round of navigation, the model first generates inferences related to the current state and then outputs the corresponding action selection. After the environment executes the action, it returns new observation information, and the system then appends the new inferences, actions, and observation records to the historical record for use in the next round of navigation decisions. Essentially, it combines textualized observation input, linear historical trajectory recording, and progressive inference decision-making using a large language model, enabling a pre-trained large language model to complete a certain degree of visual-language navigation tasks without requiring a dedicated navigation strategy model.
[0010] In addition to the technologies described above, the inventors have also considered other prior art related to some of the technical features of this invention. For example, AgentFold-like methods focus on dynamic context management in long-range agent tasks. Their core idea is to proactively organize or fold historical interaction records to alleviate the problems of excessively long contexts and redundant historical information in long-range tasks. AgentVLN or other skill-invoking navigation agent methods attempt to introduce skill libraries, tools, or modular capabilities into the visual-language navigation process to improve the agent's decision-making ability in complex navigation tasks.
[0011] The aforementioned technologies provide references in different directions for large-model-based navigation decision-making, long-term history management, and skill-invoking agents, but still do not cover the complete technical solution proposed in this invention. NavGPT-like methods provide a ReAct-style navigation decision-making framework based on large language models, but mainly rely on linear historical records and model-based reasoning, lacking a mechanism to extract structured navigation skills from successful trajectories and form action constraints during online decision-making. AgentFold-like methods focus on context compression and history management, but do not establish executable navigation skill enhancement mechanisms for candidate navigable viewpoint selection, stop action verification, and repeated access avoidance in indoor visual-language navigation. Although skill-invoking navigation methods involve external skills or modular capabilities, they typically do not form a stable closed loop between the task progress state obtained from dynamic history reconstruction and skill triggering, skill matching, and action enhancement decision-making.
[0012] While the aforementioned existing technologies can utilize pre-trained models or navigation agents to complete visual language navigation decisions to a certain extent, and have explored aspects such as historical management, experience reuse, or skill invocation, they still lack an executable skill enhancement and historical state coordination mechanism for long-range indoor visual language navigation tasks.
[0013] Specifically, in long-range navigation, agents often need to handle tasks such as switching between mission phases after passing intermediate landmarks, deciding when to stop before the target is confirmed, repeatedly avoiding visited areas, and selecting candidate navigable viewpoints with directional constraints. Existing ReAct-style navigation methods typically rely on models to reason based on current observations and linear historical records, lacking reusable, triggerable, and constrained structured navigation skills for current action selection. Therefore, in complex indoor environments, problems such as untimely task phase switching, repeated detours, or premature stopping are prone to occur.
[0014] Meanwhile, while some history compression or context management methods can alleviate the problem of excessively long historical data to some extent, their focus is usually on reducing the length of the input context or organizing historical records, without establishing executable skill constraints for critical action decisions in indoor navigation, such as candidate navigable viewpoint selection, stop action verification, and repeated access avoidance. Therefore, history compression alone is still insufficient to reliably solve the action selection problem in critical navigation states.
[0015] Furthermore, while some experience retrieval or skill invocation methods attempt to utilize external experience or skills to assist agent decision-making, they typically provide similar historical trajectories or general skills as textual references to the model. They lack a complete mechanism for extracting structured skill units from key turning points in the successful trajectories of the first navigation model and combining this with the current task progress state for skill triggering, skill matching, and decision enhancement. Therefore, their constraints on key decisions such as stop action verification, repeated access avoidance, task phase switching, and directional constraint execution remain insufficiently clear.
[0016] Therefore, a skill enhancement decision-making method for long-range visual-language navigation is needed. This method can extract reusable, triggerable, and executable structured navigation skills from the successful trajectories of the first navigation model or historical successful trajectories. Combined with the task progress status formed by dynamic historical reconstruction, the matched navigation skills are introduced as decision constraints into the action selection process of the second navigation model, thereby improving the stability of key state decisions in long-range navigation. Summary of the Invention
[0017] To address the problems existing in the prior art, this invention proposes a long-range visual-language navigation method and system. The technical solution of this invention is as follows.
[0018] This invention provides a long-range visual-language navigation method, comprising the following steps: S1. Obtain the navigation trajectory generated by the first navigation model in the historical visual language navigation task, and filter the successful trajectory or high-quality trajectory from the historical navigation trajectory according to the preset task success conditions and trajectory quality conditions. S2. Identify key navigation turning points in the successful or high-quality trajectory based on the navigation skill template; S3. Based on the identified key navigation turning points, generate structured navigation skill units according to the corresponding navigation skill templates; S4. Perform de-contextualization processing on the structured navigation skill unit and construct an external navigation skill library; S5. The second navigation model performs online navigation and dynamic history reconstruction. S6. Trigger applicable navigation skills based on the current navigation status; S7. The triggered navigation skills are used as decision constraints to participate in the action selection of the second navigation model; S8. Output the final navigation action and update the navigation state.
[0019] Preferably, the first navigation model is a model with visual language navigation capabilities. The historical visual language navigation task can be an indoor visual language navigation task, a robot navigation task, a virtual environment navigation task, or other navigation tasks based on natural language instructions. The navigation trajectory includes at least natural language navigation instructions, viewpoint observation information for each navigation step, a set of candidate navigable viewpoints, inference content and actions generated by the first navigation model, environmental return observations after action execution, and the final navigation result. The task success condition can be determined based on the final stopping position, the arrival status of the target area, and the completion status of intermediate landmarks. The trajectory quality condition can be determined based on information such as repeated visits, invalid detours, and navigation path length.
[0020] Preferably, the navigation skill templates include at least the following five categories: The first category is the landmark switching skill template, which describes the local decision-making ability of an agent to switch from the current instruction stage to the next instruction stage after observing or passing an intermediate landmark. The second category is the premature stop inhibition skill template, which describes the agent's ability to avoid outputting a stop action and continue comparing candidate navigable viewpoints for local decision-making before the final goal is confirmed. The third category is the repeated access avoidance skill template, which describes the agent's local decision-making ability to avoid returning to an invalid region when the candidate action points to a recently visited region and there is no clear need to backtrack. The fourth category is the spatial transfer skill template, which describes the agent's local decision-making ability to choose to enter a new space when no target is found in the current area and spatial transfer clues such as doorways, corridors, stairs, and open areas appear in the observation. The fifth category is the direction constraint skill template, which describes the agent's local decision-making ability to select candidate navigable viewpoints consistent with the direction constraints when there are direction constraints such as left turn, right turn, straight ahead, and return in natural language navigation instructions.
[0021] Preferably, step S2 specifically includes the following steps: S21. Divide the successful trajectory or high-quality trajectory into multiple local trajectory segments according to the number of navigation steps, instruction fragments, or local time windows; S22. Identify key landmarks, direction words, target areas, spatial transformation words, and phased actions in the natural language navigation instructions from the local trajectory segments; S23. Combining current viewpoint observation information, candidate navigable viewpoint set and historical trajectory changes, determine whether there are key behaviors such as phase switching, spatial transfer, direction execution, stop judgment or repeated access avoidance in local trajectory segments; S24. Match the identified key behaviors with preset navigation skill templates to obtain key navigation turning points and their corresponding skill template types.
[0022] Preferably, step S3 specifically includes the following steps: S31. Determine the skill type of the structured navigation skill unit based on the skill template type corresponding to the key navigation inflection point; S32. Extract applicable instruction patterns based on natural language navigation instructions and local trajectory segments corresponding to key navigation turning points; S33. Extract visual observation conditions based on the observation information before and after the key navigation turning point; S34. Extract historical state conditions based on the historical trajectory changes before and after key navigation turning points; S35. Based on the successful actions and avoided actions of the first navigation model at key navigation inflection points, generate candidate action constraints, stopping rules, and negative action rules. S36. Determine the action intervention method based on the skill type.
[0023] Preferably, step S4 specifically includes the following steps: S41. Delete the specific house number, scene number, dataset internal environment identifier, candidate navigable viewpoint number, specific path number, and complete path or action sequence in the structured navigation skill unit to avoid navigation skills relying on fixed navigation routes in a single environment. S42. Abstract the landmarks, objects, spaces and positional relationships in a specific environment into general state expressions such as intermediate landmarks, target areas, visited areas, new spaces, spatial transfer cues and directional cues, while retaining applicable instruction patterns, visual observation conditions, historical state conditions, candidate action constraints, stopping rules and negative action rules. S43. Based on skill type, triggering conditions, applicable status and action constraints, deduplicate, merge and filter the de-contextualized structured navigation skill units, and write the structured navigation skill units that meet the preset requirements into the external navigation skill library for skill triggering and calling when the second navigation model is navigating online. For skill units with unclear applicable conditions, low reliability of source trajectory or highly repetitive content, reduce their priority or do not write them into the external navigation skill library.
[0024] Preferably, the conditions for triggering the dynamic history reconstruction are at least one of the following: the length of the historical record exceeds a preset threshold, a fixed navigation step interval is reached, no new key landmarks or spatial clues are observed for several consecutive steps, repeated visits, back-and-forth movement or path stagnation occur, the second navigation model is preparing to output a stop action, and the current candidate action has the risk of backtracking, inconsistent direction, mismatched stage or premature stop.
[0025] Preferably, the triggering rules for the applicable navigation skills are as follows: when the structured task progress status indicates that an intermediate landmark has appeared or the corresponding sub-target has been completed, but the subsequent target has not yet been completed, a landmark conversion skill is triggered; when the second navigation model has a tendency to stop, but the structured task progress status lacks confirmation information of the final target or target area, a premature stop suppression skill is triggered; when there is a viewpoint pointing to a recently visited area among the candidate navigable viewpoints, and there is no clear need to backtrack in the current task stage, a revisit avoidance skill is triggered; when a doorway, corridor, staircase, open area, or other spatial transfer clues appear in the current viewpoint observation, and the target has not yet been confirmed, a revisit avoidance skill is triggered. When a navigation skill is recognized, spatial transfer skills are triggered. When a natural language navigation instruction contains directional requirements and there are corresponding directional differences among candidate navigable viewpoints, directional constraint skills are triggered. When a navigation skill meets the triggering conditions, the system outputs the triggered navigation skill and its triggering basis for step S7 to generate corresponding navigation decision constraints. When multiple navigation skills meet the triggering conditions simultaneously, the order of use can be determined based on the current task stage, action risk type, skill source reliability, or preset skill priority, or multiple navigation skills can be triggered simultaneously. When no navigation skill meets the triggering conditions, the current navigation step is executed according to the decision process without additional skill constraints.
[0026] Preferably, step S7 specifically includes the following steps: S71. Based on the skill type of the triggered navigation skill and the current triggering reason, generate candidate action constraints, stopping rules and negative action rules that are adapted to the current navigation state, and add them to the decision context of the second navigation model. S72. Adjust the selection priority of candidate navigable viewpoints based on the candidate action constraints in the triggered navigation skill; S73. When the second navigation model tends to output a stop action, determine whether to allow the stop based on the target confirmation status, the sub-targets to be completed, and the stop evidence in the structured task progress status; if the final target or target area has not yet been confirmed, or there are still sub-targets that have not been completed, then suppress the stop action and require the second navigation model to continue to select the next action from the candidate navigable viewpoints. S74. After the second navigation model generates the initial navigation action, it determines whether the initial navigation action conforms to the candidate action constraints, stopping rules, and negative action rules in the triggered navigation skill. When the initial navigation action does not conform to the corresponding skill constraints, the violated constraint content and corresponding reasons are provided to the second navigation model again, requiring it to regenerate the corrected navigation action.
[0027] On the other hand, the present invention provides a long-range visual language navigation system, applied to the aforementioned long-range visual language navigation method, characterized in that it comprises the following modules: The historical navigation trajectory processing module is used to obtain the navigation trajectory generated by the first navigation model in the historical visual language navigation task, and to filter successful or high-quality trajectories based on the task completion status and trajectory quality. The structured navigation skill construction module is used to analyze the successful or high-quality trajectory, identify key navigation turning points by combining preset navigation skill templates, and extract applicable instruction patterns, visual observation conditions, historical state conditions, successful actions, and actions to be avoided from the key navigation turning points to generate structured navigation skills with trigger conditions, candidate action constraints, stopping rules, and negative action rules. It is also used to de-contextualize, deduplicate, merge, and filter the structured navigation skills, and write the processed structured navigation skills into an external navigation skill library. The dynamic history reconstruction module is used to dynamically reconstruct the continuously accumulated reasoning content, action selection and environmental return observations when the second navigation model performs long-range visual language navigation tasks. It organizes the earlier original linear history into a structured task progress state, while retaining the original trajectory of the most recent steps. The navigation skill triggering and constraint decision module is used to determine whether the current navigation state meets the triggering conditions of one or more structured navigation skills in the external navigation skill library based on the current natural language navigation instructions, current viewpoint observation information, candidate navigable viewpoint set, structured task progress status, and the most recent original trajectory when necessary. When the triggering conditions are met, the module outputs the triggered navigation skill, the triggering reason, and its action constraints. It also adjusts the selection priority of candidate navigable viewpoints, constrains stopping actions, and verifies or re-decides preliminary actions that do not meet the skill constraints, so that the navigation skill participates in the determination of the current navigation action of the second navigation model. The joint navigation decision and state update module is used to construct a joint navigation decision context by combining the current natural language navigation command, current viewpoint observation information, candidate navigable viewpoint set, structured task progress status, most recent original trajectory, and triggered navigation skill constraints, and input it into the second navigation model; it is also used to receive candidate navigable viewpoints or stop actions output by the second navigation model, obtain new observation information after the environment executes the actions, and update the historical navigation trajectory, structured task progress status, skill trigger status, and visited viewpoint information.
[0028] Compared with the prior art, the beneficial effects of the present invention are mainly reflected in the following aspects: 1. This invention improves decision-making stability under critical states in long-range navigation. This invention extracts structured navigation skills from successful or high-quality trajectories of a first navigation model and triggers corresponding skills based on the current state during online navigation of a second navigation model. Through skills such as landmark transformation, premature stop suppression, repeated access avoidance, spatial transfer, and direction constraints, it provides executable decision constraints for the second navigation model under critical states such as task phase switching, stop judgment, candidate navigable viewpoint selection, and direction execution, thereby reducing problems such as untimely phase switching, repeated detours, and premature stopping.
[0029] 2. This invention improves the transferability of strong model capabilities. This invention does not require the second navigation model to directly mimic the complete trajectory of the first navigation model, nor does it store specific environment numbers, specific viewpoint paths, or complete action sequences. Instead, it extracts de-environmentalized local navigation skills from key navigation turning points. Therefore, the local problem-solving capabilities demonstrated by the first navigation model in successful trajectories can be reused as structured skills in new environments, new instructions, and new sets of candidate navigable viewpoints.
[0030] 3. This invention mitigates the generalization limitations of conventional experience base methods. Conventional experience bases typically provide similar historical tasks or complete successful trajectories as textual experience for model reference, which is prone to dependence on specific scenarios and paths. The external navigation skill base of this invention stores structured skill units with triggering conditions, applicable states, and action constraints. These units can participate in action selection as decision constraints when the current navigation state meets the conditions, thus exhibiting clearer executability and better cross-environment applicability compared to conventional experience retrieval methods.
[0031] 4. This invention reduces historical redundancy and dilution of key information in long-range navigation. Through a dynamic history reconstruction mechanism, this invention organizes the continuously growing reasoning content, action selections, and environmental observation information into a structured task progress state, reducing contextual redundancy caused by excessively long original linear historical records. This allows the second navigation model to more clearly utilize information such as the current state, path progress, visited or avoided areas, effective navigation cues, and next decision biases.
[0032] 5. This invention improves the accuracy of navigation skill triggering and action enhancement. The structured task progress status obtained from dynamic historical reconstruction can clearly record the current navigation progress and candidate action risks, providing a state basis for skill triggering, skill matching, and action verification. For example, when the target has not yet been confirmed but the model is preparing to stop, a premature stop suppression skill can be triggered; when an intermediate landmark has appeared but the subsequent target has not yet been completed, a landmark conversion skill can be triggered; when a candidate action points to a recently visited area, a revisit avoidance skill can be triggered. This improves the stability of skill triggering and the targeting of action enhancement.
[0033] 6. This invention reduces the training and deployment costs of the second navigation model. By reusing the local navigation capabilities of the first navigation model through an external navigation skill library and an online skill triggering mechanism, this invention eliminates the need for supervised fine-tuning or retraining of the second navigation model. Therefore, this method helps reduce training costs, improves the flexibility of skill library updates, and is applicable to navigation models of different sizes and types.
[0034] 7. This invention has good scalability and applicability. It is not only applicable to indoor visual-language navigation tasks such as R2R, but can also be extended to robot indoor navigation based on natural language commands, embodied intelligent agent navigation, virtual environment navigation, and other intelligent agent navigation scenarios requiring long-range task memory, skill reuse, and phased decision-making. Attached Figure Description
[0035] Figure 1 This is an overall flowchart of a long-range visual language navigation method provided in an embodiment of the present invention; Figure 2 A flowchart illustrating the construction process of an external navigation skill library according to an embodiment of the present invention; Figure 3 This is a flowchart of online skill triggering and action enhancement decision-making provided in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating dynamic history reconstruction and structured task progress status generation according to an embodiment of the present invention; Figure 5 This is a schematic diagram comparing the predicted trajectory before and after dynamic history reconstruction enhancement according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the input and output of dynamic history reconstruction according to an embodiment of the present invention; Figure 7 This is a schematic diagram comparing the original reasoning trajectory content before and after dynamic history reconstruction, as provided in an embodiment of the present invention. Detailed Implementation
[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0037] This invention relates to a method for generating successful or high-quality trajectories using a first navigation model in an indoor navigation scenario, and extracting the local navigation decision-making capabilities therein as structured navigation skills; and enhancing key navigation decisions such as candidate navigable viewpoint selection, stop action judgment, repeated access avoidance, task phase switching, and direction constraint execution through an external navigation skill library when the second navigation model performs online navigation decisions.
[0038] Furthermore, this invention also relates to a dynamic history reconstruction mechanism for compressing and organizing historical reasoning, action selection, and environmental observation information continuously generated during navigation to form a structured task progress state. This structured task progress state serves two purposes: firstly, it reduces contextual redundancy and dilution of key information during long-range navigation; secondly, it provides a state basis for triggering, matching, and enhancing external navigation skills, thereby improving the stability of long-range navigation decisions.
[0039] This invention is particularly suitable for indoor visual language navigation tasks such as R2R (Room-to-Room), and can also be extended to robot indoor navigation based on natural language commands, embodied intelligent agent navigation, virtual environment navigation, and other intelligent agent navigation scenarios that require long-range task memory, skill reuse, and phased decision-making. Example 1
[0040] The overall process of the long-range visual language navigation method provided in this embodiment is as follows: Figure 1 As shown in the figure, this embodiment proposes a skill enhancement decision-making and dynamic history reconstruction method for long-range visual language navigation tasks. The method includes an offline navigation skill base construction phase and an online navigation enhancement decision-making phase.
[0041] In this embodiment, the first navigation model, the second navigation model, and the history reconstruction model perform different functions. The first navigation model is used to execute historical visual language navigation tasks offline, generate successful or high-quality trajectories, and provide trajectory sources for subsequent structured navigation skill extraction. The second navigation model is used to execute new long-range visual language navigation tasks online, and outputs the next candidate navigable viewpoint or stop action with the assistance of an external navigation skill library and structured task progress status. The history reconstruction model is used to organize the inference content, action selection, and environmental return observations continuously generated by the second navigation model during online navigation, and generate a structured task progress status. The history reconstruction model can be the same as the second navigation model or it can be a model independent of the second navigation model; its main function is to compress, summarize, and represent historical information in a state-based manner, rather than directly outputting navigation actions.
[0042] During the offline navigation skill library construction phase, the system uses the first navigation model to execute historical visual language navigation tasks and obtain successful or high-quality trajectories. Then, according to the preset navigation skill template, the system extracts structured navigation skills from the key navigation turning points in the successful or high-quality trajectories and stores the structured navigation skills in the external navigation skill library after de-contextualization processing.
[0043] During the online navigation enhancement decision-making phase, the second navigation model executes a new long-range visual-language navigation task. The system maintains the current navigation history during the second navigation model's inference process and forms a structured task progress status through dynamic history reconstruction. Subsequently, based on the current natural language command, current observations, candidate navigable viewpoints, historical navigation trajectories, and structured task progress status, it determines whether to trigger navigation skills from the external navigation skill library. When the triggering condition is met, the matched navigation skill is introduced as a decision constraint into the second navigation model's action selection process, thereby assisting the second navigation model in outputting the next candidate navigable viewpoint or ceasing action.
[0044] The core process of this embodiment includes the following steps. The overall process is as follows: Figure 1 As shown, this invention demonstrates the overall process from generating a successful or high-quality trajectory using the first navigation model, to extracting structured navigation skills, constructing an external navigation skill library, and then to online reasoning, dynamic history reconstruction, skill triggering, action enhancement decision-making, and final action output using the second navigation model. The construction process of the external navigation skill library is as follows: Figure 2 As shown, this demonstrates how a successful or high-quality trajectory of the first navigation model undergoes trajectory filtering, trajectory segmentation, key navigation turning point identification, matching of five types of navigation skill templates, generation of structured navigation skill units, de-contextualization processing, and deduplication and merging to ultimately form an external navigation skill library. The online skill triggering and action enhancement decision-making process is as follows: Figure 3 As shown, this demonstrates how the second navigation model triggers matching navigation skills based on current natural language commands, current viewpoint observations, candidate navigable viewpoints, historical navigation trajectories, and / or structured task progress status during online navigation. The triggered skills are then transformed into decision constraints, which are used to assist the second navigation model in outputting the final action through skill constraint injection, stop action verification, repeated access avoidance, stage switching guidance, spatial transfer guidance, direction constraint enhancement, or re-decision. The dynamic historical reconstruction and structured task progress status generation process is as follows: Figure 4 The diagram illustrates how the reasoning content, action choices, and environmental return observations from the original linear historical record are organized into a structured task progress state through a dynamic history reconstruction module. This structured task progress state can include information such as the current state, path progress, visited or avoided areas, valid navigation clues, and the next decision bias.
[0045] S1: Obtain and filter the historical navigation trajectories of the first navigation model. Obtain the navigation trajectory generated by the first navigation model in a historical visual language navigation task. The first navigation model is a model with visual language navigation capabilities, and the historical visual language navigation task can be an indoor visual language navigation task, a robot navigation task, a virtual environment navigation task, or other navigation tasks based on natural language commands.
[0046] The navigation trajectory includes at least natural language navigation instructions, viewpoint observation information for each navigation step, a set of candidate navigable viewpoints, inference content and actions generated by the first navigation model, environmental return observations after action execution, and the final navigation result. Successful or high-quality trajectories are selected from the historical navigation trajectories based on preset task success and trajectory quality conditions. The task success conditions can be determined based on the final stopping position, target area arrival status, and intermediate landmark completion status; the trajectory quality conditions can be determined based on information such as repeated visits, invalid detours, and navigation path length.
[0047] The successful or high-quality trajectories obtained in this step are used for subsequent identification of key navigation inflection points and extraction of transferable structured navigation skills, rather than for enabling the second navigation model to directly mimic the complete navigation trajectory or reproduce specific action sequences.
[0048] S2: Identify key navigation turning points based on navigation skill templates Analyze the successful or high-quality trajectories selected in step S1 to identify key navigation turning points.
[0049] The critical navigation inflection point refers to a local decision-making location that has a significant impact on navigation success. This location typically corresponds to navigation behaviors in natural language commands, such as stage changes, spatial transformations, direction execution, stop judgments, or revisit avoidance.
[0050] This invention provides pre-defined navigation skill templates to guide the identification of key navigation turning points and subsequent structured skill generation. The navigation skill templates include at least the following five categories: The first category is the landmark switching skill template, which describes the local decision-making ability of an agent to switch from the current instruction stage to the next instruction stage after observing or passing an intermediate landmark. The second category is the premature stop inhibition skill template, which describes the agent's ability to avoid outputting a stop action and continue comparing candidate navigable viewpoints for local decision-making before the final goal is confirmed. The third category is the repeated access avoidance skill template, which describes the agent's local decision-making ability to avoid returning to an invalid region when the candidate action points to a recently visited region and there is no clear need to backtrack. The fourth category is the spatial transfer skill template, which describes the agent's local decision-making ability to choose to enter a new space when no target is found in the current area and spatial transfer clues such as doorways, corridors, stairs, and open areas appear in the observation. The fifth category is the direction constraint skill template, which describes the agent's local decision-making ability to select candidate navigable viewpoints consistent with the direction constraints when there are direction constraints such as left turn, right turn, straight ahead, and return in natural language navigation instructions.
[0051] Furthermore, step S2 may include the following sub-steps: S21: Divide the successful or high-quality trajectory into multiple local trajectory segments according to navigation steps, instruction fragments, or local time windows; S22: Identify key landmarks, direction words, target areas, spatial transition words, and phased actions in the natural language navigation instructions from the local trajectory segments; S23: Combining current viewpoint observation information, candidate navigable viewpoint set and historical trajectory changes, determine whether there are key behaviors such as phase switching, spatial transfer, direction execution, stop judgment or repeated access avoidance in local trajectory segments; S24: Match the identified key behaviors with preset navigation skill templates to obtain key navigation turning points and their corresponding skill template types.
[0052] For example, when a natural language navigation instruction requires "entering the corridor after passing the kitchen," and the successful trajectory shows that the first navigation model, after observing the kitchen, no longer stays near the kitchen but chooses a candidate navigable viewpoint leading to the corridor or a new space, this segment can be identified as a landmark transition-type critical navigation inflection point. As another example, when the final goal of the natural language navigation instruction has not yet appeared, and the first navigation model refuses to stop and continues forward in the successful trajectory, this segment can be identified as a premature stop inhibition-type critical navigation inflection point.
[0053] S3: Generate Structured Navigation Skill Units For the key navigation turning points identified in step S2, structured navigation skill units are generated according to the corresponding navigation skill templates.
[0054] The structured navigation skill unit is neither a complete navigation trajectory nor ordinary textual experience, but rather a triggerable, reusable, and executable skill unit used to describe action decision-making methods in a certain type of local navigation state. This skill unit is used to express the local problem-solving capabilities demonstrated by the first navigation model in a successful trajectory, rather than recording the complete path answer of the first navigation model in a specific environment.
[0055] The structured navigation skill unit includes at least the following fields: 1. Skill type, used to indicate the category of the navigation skill template to which this skill belongs; 2. Triggering conditions, used to determine whether the skill is applicable to the current navigation state; 3. Applicable instruction pattern, used to describe the natural language navigation instruction fragments applicable to this skill; 4. Visual observation conditions, used to describe landmarks, objects, spatial structures, or directional cues that should or should not appear in the current observation; 5. Historical status conditions, used to describe completed sub-objectives, current task stages, visited areas, repeated visit status, or target confirmation status; 6. Candidate action constraints, used to describe candidate navigable viewpoint types that should be prioritized, avoided, or prohibited; 7. Stopping rules, which describe the conditions that allow or inhibit a stopping action; 8. Negative action rules are used to describe actions that should be avoided, such as backtracking, repetitive access, inconsistent direction, or premature stopping. 9. Action intervention methods, used to describe how this skill participates in the action decision-making of the second navigation model during online navigation; 10. Skill source information, used to record which type of trajectory segment, skill template, or key navigation turning point the skill originated from.
[0056] Furthermore, step S3 may include the following sub-steps: S31: Determine the skill type of the structured navigation skill unit based on the skill template type corresponding to the key navigation inflection point; S32: Extract applicable instruction patterns based on natural language navigation instructions and local trajectory segments corresponding to key navigation turning points; S33: Extract visual observation conditions based on observation information before and after key navigation turning points; S34: Extract historical state conditions based on the historical trajectory changes before and after key navigation turning points; S35: Based on the successful actions and avoided actions of the first navigation model at key navigation inflection points, generate candidate action constraints, stopping rules, and negative action rules; S36: Based on the applicable scenario of the skill, determine the action intervention method, which may include skill constraint injection, stop action verification, repeated access avoidance prompt, direction constraint prompt, stage switching prompt or re-decision prompt.
[0057] For example, for landmark transformation skills, the triggering condition may include "the intermediate landmark has already appeared in the current observation or historical observation, and the subsequent target has not yet been completed"; the candidate action constraint may include "prioritizing candidate navigable viewpoints that lead to the next target, new area, corridor, doorway, or conform to the subsequent instruction fragment"; the negative action rule may include "avoiding staying near completed landmarks, returning to completed areas, or stopping directly".
[0058] For example, for skills that suppress premature stopping, the triggering condition may include "the second navigation model has a tendency to stop, but the final target confirmation information has not yet been recorded in the structured task progress status"; the stopping rule may include "suppressing the stopping action when the final target or target area has not been confirmed, and requiring continued comparison of candidate navigable viewpoints".
[0059] S4: Decontextualize structured navigation skill units and build an external navigation skill library. The structured navigation skill units generated in step S3 are de-contextualized to remove information that is only applicable to specific scenarios or specific navigation paths, and the processed structured navigation skill units are written into the external navigation skill library.
[0060] Furthermore, step S4 may include the following sub-steps: S41: Delete the specific house number, scene number, dataset internal environment identifier, candidate navigable viewpoint number, specific path number, and complete path or action sequence in the structured navigation skill unit to avoid the navigation skill from relying on a fixed navigation route in a single environment. S42: Abstracts landmarks, objects, spaces and positional relationships in a specific environment into general state expressions such as intermediate landmarks, target areas, visited areas, new spaces, spatial transfer cues and directional cues, while retaining applicable instruction patterns, visual observation conditions, historical state conditions, candidate action constraints, stopping rules and negative action rules. S43: Based on skill type, triggering conditions, applicable status, and action constraints, de-contextualized structured navigation skill units are deduplicated, merged, and filtered. Structured navigation skill units that meet preset requirements are written into an external navigation skill library for skill triggering and invocation during online navigation by the second navigation model. For skill units with unclear applicable conditions, low reliability of source trajectories, or highly repetitive content, their priority can be reduced or they can be excluded from the external navigation skill library.
[0061] Through the above processing, the local decision-making capability demonstrated by the first navigation model in specific successful trajectories is transformed into structured navigation skills that do not depend on specific environment numbers, specific viewpoints, or complete paths. The external navigation skill library stores local navigation skills with clear triggering conditions, applicable states, and action constraints, rather than complete historical tasks, specific successful trajectories, or similar cases for reference only. Therefore, these skills can participate as decision constraints in the action selection of the second navigation model in new navigation tasks and environments.
[0062] S5: The second navigation model performs online navigation and dynamic history reconstruction. During the online navigation phase, the second navigation model performs a new long-range visual-language navigation task. The system acquires the current natural language navigation instructions, current viewpoint observation information, a set of candidate navigable viewpoints, and historical navigation trajectories in the current task.
[0063] The second navigation model can be a large language model to be enhanced, a multimodal large model, or other models with visual language navigation capabilities. The historical navigation trajectory can include the inference content, executed actions, action inputs, environmental return observations, and visited viewpoint information generated by the second navigation model in the current task.
[0064] During the execution of long-range navigation tasks by the second navigation model, the system dynamically reconstructs the continuously accumulated historical navigation trajectory based on fixed step intervals or event triggering conditions, generating a structured task progress status.
[0065] The dynamic history reconstruction serves two purposes: firstly, it compresses and organizes the historical records accumulated during long-range navigation, reducing contextual redundancy and dilution of key information; secondly, it extracts task progress information related to skill triggering, providing a state basis for subsequent skill matching and action enhancement. For example... Figure 6 The diagram illustrates the input and output of dynamic history reconstruction in this embodiment of the invention. It demonstrates how the history reconstruction model outputs a structured task progress status given natural language navigation instructions and the original trajectory history. This structured task progress status may include information such as the current status, path progress, visited or avoided areas, valid navigation clues, and the next decision bias. Figure 7 The diagram shows a comparison of the original inference trajectory before and after dynamic historical reconstruction. It illustrates the original inference trajectory formed by the second navigation model before historical reconstruction, as well as the structured task progress status generated after dynamic historical reconstruction, thus demonstrating the role of dynamic historical reconstruction in compressing historical redundancy, retaining key navigation information, and supporting subsequent navigation decisions.
[0066] Furthermore, the conditions that trigger dynamic history reconstruction may include at least one of the following: S51: The length of the historical record exceeds the preset threshold; S52: Achieve a fixed navigation step interval; S53: No new key landmarks or spatial clues were observed over several consecutive steps; S54: Repeated visits, back-and-forth movement, or path stalling have occurred; S55: The second navigation model is preparing to output a stop action; S56: The current candidate action is at risk of rollback, inconsistent direction, mismatched stage, or premature termination.
[0067] Furthermore, the structured task progress status includes at least the following fields: The current state describes the current navigation phase, current location or neighboring area of the agent, and task-related environmental information in the current observations. Path progress is used to record the completed and incomplete tasks in natural language navigation instructions; the path progress can further include completed sub-goals and incomplete sub-goals. Visited areas or areas to be avoided are used to record viewpoints, spaces or areas that the agent has already visited, and to mark areas that should not be returned to repeatedly based on the current task stage; Effective navigation cues are used to record important landmarks, visual cues, spatial cues, and target object cues that are relevant to subsequent navigation decisions; The next decision bias is used to provide directions, regions, or candidate navigable viewpoint types that should be prioritized when selecting the next candidate action, based on the current state, path progress, visited areas, and valid navigation cues, as well as directions, regions, or candidate navigable viewpoint types that should be avoided.
[0068] It should be noted that the structured task progress status is not a simple compressed text of the entire historical record, but a status representation generated for long-range navigation task progress and skill triggering requirements. This status can be used alone to improve the long-range history utilization capability of the second navigation model, or it can be combined with an external navigation skill library to determine whether the triggering conditions of a certain type of navigation skill are currently met.
[0069] S6: Trigger applicable navigation skills based on the current navigation state When the second navigation model makes online navigation decisions, the system determines whether the current navigation state meets the triggering conditions of one or more structured navigation skill units in the external navigation skill library based on the current natural language navigation instructions, current viewpoint observation information, candidate navigable viewpoint set, structured task progress status, and the most recent original trajectory fragment when necessary.
[0070] The navigation skill is not triggered by retrieving historical cases based on the general semantic similarity between the current task and historical tasks. Instead, it matches the current navigation state with the triggering conditions, applicable instruction modes, visual observation conditions, and historical state conditions in the structured navigation skill unit, and triggers the corresponding navigation skill when the corresponding conditions are met.
[0071] Furthermore, the corresponding navigation skill can be triggered according to at least one of the following rules: S61: When the progress status of a structured task indicates that an intermediate landmark has appeared or the corresponding sub-objective has been completed, but the subsequent objective has not yet been completed, trigger a landmark transformation skill; S62: When the second navigation model has a tendency to stop, but the structured task progress status lacks confirmation information of the final goal or target area, the premature stop inhibition skill is triggered. S63: When there is a viewpoint among the candidate navigable viewpoints that points to a recently visited area, and there is no clear need to backtrack in the current task phase, trigger a repeated visit avoidance skill. S64: When a doorway, corridor, staircase, open area, or other spatial movement clues appear in the current viewpoint observation, and the target has not yet been confirmed, a spatial movement skill is triggered; S65: When a natural language navigation instruction contains directional requirements and there are corresponding directional differences among candidate navigable viewpoints, a directional constraint skill is triggered.
[0072] When a navigation skill meets the triggering conditions, the system outputs the triggered navigation skill and its triggering basis, which is then used by step S7 to generate the corresponding navigation decision constraints. When multiple navigation skills meet the triggering conditions simultaneously, the order of use can be determined based on the current task stage, action risk type, skill source reliability, or preset skill priority, or multiple navigation skills can be triggered simultaneously. When no navigation skill meets the triggering conditions, the current navigation step is executed according to the decision process without additional skill constraints.
[0073] S7: The triggered navigation skills are used as decision constraints in the action selection of the second navigation model. When at least one navigation skill is triggered in step S6, the system extracts the triggering reason, applicable state, candidate action constraints, stopping rules, and negative action rules from the triggered navigation skill and converts them into navigation decision constraints corresponding to the current navigation step. These navigation decision constraints can take the form of natural language constraints, structured field constraints, or a combination of both, and participate in the action selection of the second navigation model as part of the joint navigation decision context.
[0074] Furthermore, step S7 may include the following sub-steps: S71: Generate and add skill constraints. Based on the skill type of the triggered navigation skill and the current triggering reason, generate candidate action constraints, stopping rules, and negative action rules that are adapted to the current navigation state, and add them to the decision context of the second navigation model so that the second navigation model explicitly considers the skill constraints when generating navigation actions.
[0075] S72: Limit or adjust candidate action selection. Adjust the selection priority of candidate navigable viewpoints based on the constraints of candidate actions in the triggered navigation skill. Specifically, the priority of candidate navigable viewpoints that meet the current task stage, subsequent objectives, spatial transfer requirements, or directional requirements can be increased, while the priority of candidate navigable viewpoints that point to recently visited areas, completed task areas, incorrect directions, or are inconsistent with the current task stage can be decreased or excluded.
[0076] S73: Constraint Stop Action. When the second navigation model tends to output a stop action, it determines whether to allow a stop based on the target confirmation status, uncompleted sub-targets, and stop evidence in the structured task progress status. If the final target or target area has not yet been confirmed, or there are still uncompleted sub-targets, the stop action is suppressed, and the second navigation model is required to continue selecting the next action from candidate navigable viewpoints.
[0077] S74: Perform a consistency check on action constraints. In one implementation, after the second navigation model generates a preliminary navigation action, the system determines whether the preliminary navigation action conforms to the candidate action constraints, stopping rules, and negative action rules in the triggered navigation skill. When the preliminary navigation action does not conform to the corresponding skill constraints, the violated constraint content and corresponding reasons are provided back to the second navigation model, requiring it to regenerate the corrected navigation action.
[0078] Through the above processing, the triggered navigation skills are not provided to the second navigation model as historical experience unrelated to the current decision, but rather participate in determining the current navigation action by limiting candidate action selection, constraining stopping conditions, and verifying preliminary actions. The second navigation model thus invokes and utilizes the local decision-making capabilities formed by the first navigation model under critical navigation states, without needing to reproduce the complete navigation trajectory of the first navigation model in a specific environment.
[0079] S8: Output the final navigation action and update the navigation state. The second navigation model outputs the final navigation action based on the decision context with added navigation skill constraints. The final navigation action can be a candidate navigable viewpoint or a stop action.
[0080] After the environment executes the final navigation action, it returns new observation information. The system adds the new inference content, action selection, and returned environmental observations to the current task's historical navigation trajectory, and updates the structured task progress status, skill trigger status, and visited viewpoint information as needed.
[0081] The above steps are executed cyclically during online navigation until the second navigation model outputs a stop action, or the preset maximum number of navigation steps is reached. Example 2
[0082] This embodiment provides a long-range visual-language navigation intelligent agent system for performing the above-described method. The system includes an offline navigation skill library construction section and an online navigation enhancement decision-making section, and includes at least the following functional modules: Module M1: Historical Navigation Trajectory Processing Module, used to acquire the navigation trajectory generated by the first navigation model in historical visual language navigation tasks, and to filter successful or high-quality trajectories based on task completion and trajectory quality. The navigation trajectory includes at least natural language navigation instructions, viewpoint observation information for each navigation step, a set of candidate navigable viewpoints, inference content and actions generated by the first navigation model, environmental return observations after action execution, and the final navigation result.
[0083] Module M2: The structured navigation skill construction module is used to analyze the successful or high-quality trajectories, identify key navigation turning points by combining preset navigation skill templates, and extract applicable instruction patterns, visual observation conditions, historical state conditions, successful actions, and actions to be avoided from the key navigation turning points. This generates structured navigation skills with trigger conditions, candidate action constraints, stopping rules, and negative action rules. This module is also used to de-contextualize, deduplicate, merge, and filter the structured navigation skills, and write the processed structured navigation skills into an external navigation skill library.
[0084] Module M3: Dynamic History Reconstruction Module. This module dynamically reconstructs the continuously accumulated inference content, action selections, and environmental return observations when the second navigation model performs long-range visual-language navigation tasks. It organizes earlier original linear histories into structured task progress states while retaining the most recent few steps of the original trajectory. The structured task progress states include at least the current task stage, path progress, completed and incomplete sub-goals, visited or avoided areas, valid navigation cues, target confirmation status, and next decision bias. These states control the length of the long-range navigation decision context and provide a state basis for navigation skill triggering and action constraint judgment.
[0085] Module M4: Navigation Skill Triggering and Constraint Decision Module. This module determines whether the current navigation state meets the triggering conditions for one or more structured navigation skills in the external navigation skill library, based on the current natural language navigation instructions, current viewpoint observation information, a set of candidate navigable viewpoints, the progress status of the structured task, and, if necessary, the most recent original trajectory. When the triggering conditions are met, this module outputs the triggered navigation skill, the triggering reason, and its action constraints. It also adjusts the selection priority of candidate navigable viewpoints, constrains stopping actions, and verifies or re-determines preliminary actions that do not meet the skill constraints, enabling the navigation skill to participate in determining the current navigation action of the second navigation model.
[0086] Module M5: Joint Navigation Decision and State Update Module. This module constructs a joint navigation decision context by combining the current natural language navigation command, current viewpoint observation information, candidate navigable viewpoint set, structured task progress status, most recent original trajectory, and triggered navigation skill constraints, and inputs this context into the second navigation model. This module also receives candidate navigable viewpoints or stop actions output by the second navigation model, acquires new observation information after the environment executes the actions, and updates the historical navigation trajectory, structured task progress status, skill trigger status, and visited viewpoint information. Example 3
[0087] This embodiment uses an R2R indoor visual language navigation task as an example to illustrate a long-range visual language navigation method based on skill-enhanced decision-making and dynamic history reconstruction proposed in this invention. It should be noted that the following embodiments are only used to illustrate the technical solution of this invention and do not constitute a limitation on the scope of protection of this invention.
[0088] In this embodiment, the indoor environment is represented as a connected graph consisting of multiple reachable viewpoints. At each step, the agent receives visual observations from the current viewpoint, candidate navigable viewpoints, natural language navigation instructions, and historical navigation trajectories, and outputs either the next candidate navigable viewpoint or a stop action.
[0089] In this embodiment, the first navigation model is used to generate successful or high-quality trajectories offline, the second navigation model is used to execute new navigation tasks online, and the history reconstruction model is used to organize historical navigation trajectories and generate structured task progress status during the execution of the second navigation model. The first navigation model, the second navigation model, and the history reconstruction model can all be large language models, multimodal large models, or other intelligent agent models with language understanding, visual information understanding, and navigation-related reasoning capabilities.
[0090] In one implementation, the first navigation model can be a model with strong navigation capabilities, high task success rate, and good trajectory quality, used to generate reliable successful trajectories or high-quality trajectories; the second navigation model can be a navigation model to be enhanced, used to perform online navigation tasks, and receives assistance from an external navigation skill library and structured task progress status; the history reconstruction model can be a language model or multimodal model independent of the second navigation model, or it can share the same model with the second navigation model. The history reconstruction model does not directly select candidate navigable viewpoints or stop actions; its main function is to organize the current state, path progress, visited areas or areas to be avoided, effective navigation clues, and next decision biases based on the generated inference content, action selections, and environmental return observations.
[0091] For example, in a specific experimental setup, a multimodal model with strong navigation capabilities can be used as the first navigation model to generate successful or high-quality trajectories. A model with lower deployment costs or requiring enhancement can be used as the second navigation model to perform online navigation. An independent language model or multimodal model can be used as the history reconstruction model to periodically summarize the historical trajectories generated by the second navigation model. The above models are merely examples, and this invention does not limit the specific model names, parameter scales, or deployment methods of the first navigation model, second navigation model, and history reconstruction model.
[0092] The specific implementation process includes the following steps.
[0093] Step S1: Obtain and filter the historical navigation trajectories of the first navigation model. The purpose of step S1 is to provide a high-quality trajectory source for subsequent structured navigation skill extraction, rather than to make the second navigation model directly mimic the complete action sequence of the first navigation model.
[0094] S11: Obtain historical visual language navigation tasks.
[0095] The system acquires a batch of historical visual language navigation tasks. Each task includes natural language navigation instructions, an indoor environment connectivity map, a starting viewpoint, a set of candidate navigable viewpoints, and a target location or target area.
[0096] S12: Execute historical navigation tasks using the first navigation model.
[0097] The first navigation model receives the current viewpoint observation, candidate navigable viewpoints, natural language navigation instructions, and existing navigation history at each step, and outputs the next candidate navigable viewpoint or stops the action.
[0098] S13: Record the complete navigation trajectory of the first navigation model.
[0099] Each navigation trajectory includes at least the natural language navigation instructions, the current viewpoint observation information for each step, the set of candidate navigable viewpoints, the inference content generated by the first navigation model, the action selected by the first navigation model, the environmental return observation after the action is executed, and the final navigation result.
[0100] S14: Filter successful or high-quality trajectories.
[0101] The system filters trajectories based on preset success conditions. These success conditions may include: the final stopping position being less than a preset distance threshold from the target position; the trajectory passing through the target area; the model outputting a stopping action near the target area; or the trajectory passing through one or more intermediate landmarks in the order of natural language instructions.
[0102] S15: Filter high-quality local trajectories.
[0103] Even for trajectories where the final mission was not entirely successful but contained locally correct decisions, high-quality segments can still be extracted. For example, if a trajectory did not ultimately reach its goal but correctly entered the next space after passing an intermediate landmark, that segment can still be used for subsequent skill extraction.
[0104] Through the above steps, the system obtains a set of successful trajectories or a set of high-quality trajectory fragments for skill extraction.
[0105] Step S2: Identify key navigation turning points based on preset navigation skill templates. The purpose of step S2 is to locate local decision segments from the complete trajectory that can be abstracted into general navigation skills, rather than saving the entire path.
[0106] This embodiment pre-sets five types of navigation skill templates to guide the identification of key navigation turning points.
[0107] The first type is the landmark switching skill template, which is used to handle situations where "after observing or passing an intermediate landmark, you should switch to the next instruction phase".
[0108] The second category is the premature stop suppression skill template, which is used to deal with the situation where "the model is ready to stop, but the final target or target area has not yet been identified".
[0109] The third category is the repeated visit avoidance skill template, which is used to handle situations where "the candidate action will return to the recently visited area, and there is no clear need to go back at present".
[0110] The fourth category is the spatial transfer skill template, which is used to handle situations where "no target is found in the current area, but spatial transfer clues such as doorways, corridors, stairs, and open areas appear in the observation."
[0111] The fifth category is the directional constraint skill template, which is used to handle situations where "natural language navigation instructions contain directional constraints such as left turn, right turn, straight ahead, and return".
[0112] Furthermore, the identification of key navigation inflection points may include the following sub-steps.
[0113] S21: Segment the local trajectory.
[0114] The system divides successful or high-quality trajectories into multiple local trajectory segments based on navigation steps, command fragments, or local time windows. Each local trajectory segment can include observations from the current step and several steps before and after it, candidate navigable viewpoints, actions, and model inference content.
[0115] S22: Identify command elements and trajectory status changes.
[0116] The system identifies key landmarks, target areas, direction words, spatial transition words, and phased actions from natural language navigation instructions. It also combines current observations, candidate navigable viewpoints, and action changes in local trajectory segments to determine whether the segment contains key behaviors such as task phase switching, spatial transfer, direction execution, stop judgment, or repeated access avoidance.
[0117] S23: Match navigation skill templates and mark key navigation turning points.
[0118] The system will match the identified key behaviors with preset navigation skill templates to determine whether the local trajectory segment is a key navigation turning point, and label it with the corresponding skill template type.
[0119] For example, if the instruction is "enter the corridor after passing through the kitchen," and the first navigation model, after observing the kitchen, does not linger but instead selects a candidate navigable viewpoint leading to the corridor or a new space, this segment can be labeled as a landmark transition-type critical navigation inflection point. As another example, if the target has not yet appeared in the current or historical observations, and the first navigation model refuses to stop and continues exploring, this segment can be labeled as a premature cessation inhibition-type critical navigation inflection point.
[0120] Step S3: Generate structured navigation skill units The purpose of step S3 is to transform the local decision-making capabilities demonstrated by the first navigation model under critical navigation states into structured skill units that can be invoked by the second navigation model.
[0121] The structured navigation skill unit is neither a complete navigation trajectory nor ordinary text experience, but a triggerable, reusable, and executable skill unit used to describe the action decision-making method in a certain type of local navigation state.
[0122] S31: Determine the skill type.
[0123] Based on the skill template type marked in step S2, determine whether the current skill unit belongs to the landmark conversion type, premature stop suppression type, repeated access avoidance type, spatial transfer type, or directional constraint type.
[0124] S32: Extract trigger conditions.
[0125] Based on the state before the critical navigation inflection point occurs, extract the trigger conditions for the skill. For example, the trigger condition for a landmark conversion skill could be "the intermediate landmark has already appeared in the current or historical observations, and the subsequent target has not yet been completed"; the trigger condition for a premature stop suppression skill could be "the model has a tendency to stop, but the target has not yet been confirmed".
[0126] S33: Extract applicable instruction mode.
[0127] Based on landmarks, directional words, and phased actions in natural language navigation instructions, extract the instruction patterns applicable to this skill. For example, "Enter a certain area after passing a certain landmark," "Turn left near a certain object," "Continue to the target area," etc.
[0128] S34: Extract visual observation conditions.
[0129] Based on observational information before and after key navigational turning points, extract landmarks, objects, spatial structures, or directional clues that should or should not appear in the current observation. For example, spatial transfer skills can record "doorways, corridors, stairs, or open areas appearing in the current observation".
[0130] S35: Extract historical state conditions.
[0131] Based on the historical trajectory before and after key navigation turning points, extract historical state conditions such as completed sub-targets, current task stage, visited areas, repeated visit status, and target confirmation status.
[0132] S36: Generate action constraint rules.
[0133] Based on the successful and avoided actions of the first navigation model at key navigation inflection points, candidate action constraints, stopping rules, and negative action rules are generated. The candidate action constraints describe candidate navigable viewpoint types that should be prioritized or avoided; the stopping rules describe the conditions that allow or suppress stopping actions; and the negative action rules describe actions that should be avoided, such as backtracking, repeated visits, inconsistent directions, or premature stopping.
[0134] S37: Determine the intervention method.
[0135] The method of using the skill during the online phase is determined based on the skill type. The action intervention methods may include skill constraint injection, stopping action verification, repeated access avoidance prompts, phase switching prompts, spatial transfer prompts, directional constraint prompts, or re-decision prompts.
[0136] After the above processing, a structured navigation skill unit can include the following fields: skill type, triggering condition, applicable instruction mode, visual observation condition, historical state condition, candidate action constraint, stopping rule, negative action rule, action intervention method, and skill source information.
[0137] For example, landmark conversion skills can be represented as follows: when an intermediate landmark has appeared and the subsequent target has not yet been completed, the second navigation model should be prompted to switch to the next instruction stage, and priority should be given to selecting candidate navigable viewpoints leading to new spaces or subsequent targets, avoiding staying near completed landmarks, returning to completed areas, or stopping directly.
[0138] Step S4: De-contextualize the structured navigation skill units and build an external navigation skill library. The purpose of step S4 is to transform navigation skills from specific trajectory experience into local navigation capabilities that can be reused across tasks and environments.
[0139] S41: Delete specific environment and path identifiers.
[0140] Remove specific house numbers, scene numbers, dataset internal environment identifiers, specific candidate navigable viewpoint numbers, complete path answers, or complete action sequences from the structured navigation skill unit to prevent the external navigation skill library from degenerating into a path memory library that is only applicable to a single environment.
[0141] S42: Abstract concrete entity.
[0142] Specific landmarks, spaces, or objects are abstracted into general expressions such as intermediate landmarks, target areas, visited areas, new spaces, and directional clues.
[0143] S44: Preserve transferable decision conditions.
[0144] Retain transferable information such as applicable instruction patterns, visual observation conditions, historical state conditions, candidate action constraints, stopping rules, and negative action rules.
[0145] S45: Skill deduplication and merging.
[0146] Skill units with highly similar triggering conditions, applicable instruction patterns, candidate action constraints, and action intervention methods can be deduplicated or merged; skill units with unclear applicable conditions or weak correlation with success trajectories can have their priority reduced or not be written into the skill library.
[0147] S46: Write to the external navigation skill library.
[0148] The processed structured navigation skill units are written into an external navigation skill library and indexed according to skill type, triggering conditions, applicable command mode, visual observation conditions, and historical state conditions, so as to match and call them in the online stage.
[0149] The resulting external navigation skill library differs from a regular experience library. A regular experience library typically stores historical tasks, complete trajectories, or similar cases, primarily used to provide the model with references to past tasks. In contrast, the external navigation skill library of this invention stores local navigation decision-making capabilities extracted from key turning points of successful trajectories. It has clear triggering conditions, applicable states, and action constraints, and can participate as a decision constraint in the action selection of the second navigation model during online navigation.
[0150] Step S5: The second navigation model performs online navigation and dynamic history reconstruction. The purpose of step S5 is to maintain the task progress state that can be used for skill triggering and action decision-making when the second navigation model performs a long-range navigation task.
[0151] S51: Obtain online navigation task input.
[0152] When the second navigation model executes a new long-range visual language navigation task, the system acquires the current natural language navigation instructions, the current viewpoint observation information, the set of candidate navigable viewpoints, and the historical navigation trajectory in the current task.
[0153] S52: Records the online historical trajectory of the second navigation model.
[0154] The system continuously records the inference content, executed actions, action inputs, environmental return observations, and visited viewpoint information generated by the second navigation model in the current task.
[0155] S53: Determine whether dynamic history reconstruction is triggered.
[0156] The system determines whether to perform dynamic history reconstruction based on fixed step intervals or event triggering conditions. The event triggering conditions may include: the length of the historical record exceeds a preset threshold, no new key landmarks or spatial clues are observed for several consecutive steps, repeated visits or back-and-forth movement occur, the second navigation model is preparing to output a stop action, or the current candidate action has the risk of backtracking, inconsistent direction, mismatched stage, or premature stopping.
[0157] S54: Generate structured task progress status.
[0158] When the conditions for historical reconstruction are met, the system organizes the continuously accumulated historical navigation trajectory and generates a structured task progress status. The structured task progress status may include the current status, path progress, visited areas or areas to be avoided, valid navigation clues, and the next decision bias.
[0159] In one implementation, the structured task progress status is generated by a history reconstruction model. The history reconstruction model receives historical navigation trajectories generated by a second navigation model, the most recent original trajectory fragment, and current navigation task information as input, and outputs the structured task progress status, but does not directly output candidate navigable viewpoints or stop actions.
[0160] S55: Preserve the most recent original trajectory segment.
[0161] To avoid losing short-term continuity during structured history reconstruction, the system retains the original reasoning content, action selection, and environmental observation information from the most recent steps to assist the second navigation model in determining the current local action relationships.
[0162] The dynamic history reconstruction serves two purposes: firstly, it reduces contextual redundancy caused by excessively long historical records in long-range navigation; secondly, it provides a state basis for subsequent skill triggering, skill matching, and action enhancement. This structured task progress state is not a simple summary, but rather a state representation generated based on navigation task stages and skill triggering conditions.
[0163] Development Process Description: Initially, we attempted to have a second navigation model simultaneously handle online navigation action decisions and history reconstruction. While simple to implement, this approach proved unstable in long-range navigation tasks. When the historical trajectory was long, the instructions contained multiple phased objectives, or there was a large amount of landmark information in the current observation, the second navigation model tended to mix action decisions with history organization, leading to omissions of key phase information in the history summary and creating a historical reconstruction illusion. Later, we attempted to separate history reconstruction from the action decision-making task of the second navigation model, setting up a history reconstruction model or independent history reconstruction module specifically to generate a structured task progress status based on the generated inference content and the most recent original trajectory fragments. Through this functional separation, this invention reduces the burden on the second navigation model to simultaneously handle action decisions and history maintenance, and reduces the interference of unstable expressions and illusory information in ordinary history summaries or model self-summaries on subsequent action decisions.
[0164] Step S6: Trigger matching navigation skills based on the current navigation state The purpose of step S6 is to select the appropriate structured navigation skill from the external navigation skill library based on the current navigation status.
[0165] S61: Construct skill triggers input judgment.
[0166] The system uses the current natural language navigation command, current viewpoint observation information, candidate navigable viewpoint set, historical navigation trajectory and / or structured task progress status as inputs for skill triggering judgment.
[0167] S62: Matching skill trigger conditions.
[0168] The system matches the current navigation status with the trigger conditions, applicable command modes, visual observation conditions, and historical status conditions of the structured navigation skill units in the external navigation skill library.
[0169] S63: Triggers landmark transformation skills.
[0170] When the progress status of a structured task shows that the intermediate landmark has appeared but the subsequent objectives have not yet been completed, a landmark transformation skill is triggered.
[0171] S64: Triggers premature cessation of suppression skills.
[0172] When the second navigation model is ready to output a stop action, but there is insufficient evidence of stopping in the structured task progress state, a premature stop inhibition skill is triggered.
[0173] S65: Triggers repeated access avoidance skills.
[0174] When a candidate action points to a recently visited area and there is no clear need to revert to the previous state, a revisit avoidance skill is triggered.
[0175] S66: Triggers spatial teleportation skills.
[0176] When doorways, corridors, staircases, open areas, or other spatial movement clues appear in the current observation, and the target has not yet been confirmed, spatial movement skills are triggered.
[0177] S67: Triggers directional constraint skills.
[0178] When natural language navigation instructions contain directional words and there are relative directional differences between candidate navigable viewpoints, directional constraint skills are triggered.
[0179] S68: Determine how to use the skill.
[0180] When multiple navigation skills meet the triggering conditions at the same time, the system can determine the triggering order based on the current task stage, candidate risk type, skill source reliability, or skill type priority. Alternatively, multiple navigation skills can be used together as decision constraints for the current step.
[0181] The skill triggering in this step is not a simple search for similar historical cases, but rather an invocation based on whether the current navigation state meets the triggering conditions of the structured skill unit.
[0182] Step S7: Use the triggered navigation skills as decision constraints to enhance the action selection of the second navigation model. The purpose of step S7 is to transform the triggered navigation skill into a decision constraint that the second navigation model can use, thereby influencing its current action selection.
[0183] S71: Generate skill constraint content.
[0184] The system will convert the triggering reason, applicable state, candidate action constraints, stopping rules and negative action rules in the triggered skills into natural language constraints, structured field constraints or a combination of both.
[0185] S72: Skill Constraint Injection.
[0186] The system adds the skill constraint to the navigation decision context of the second navigation model, so that the second navigation model explicitly considers the skill constraint when selecting the next action.
[0187] S73: Stop action verification.
[0188] When the second navigation model shows a tendency to output a stopping action, the system checks the stopping evidence in the structured task progress status. If the final target or target area has not yet been identified, or if there are still targets to be completed, the stopping action is suppressed, and the second navigation model is asked to reselect a candidate navigable viewpoint.
[0189] S74: Repeated access avoidance.
[0190] When a candidate action points to a recently visited area and there is no clear need to backtrack, the system prompts the second navigation model in the navigation decision context to avoid the candidate action and prioritizes unvisited areas or candidate navigable viewpoints related to the goal to be completed.
[0191] S75: Phase switching boot.
[0192] When the structured task progress status shows that the current instruction phase has been completed but the subsequent objectives have not yet been completed, the system prompts the second navigation model to switch to the next phase to be completed in the navigation decision context.
[0193] S76: Spatial transfer guidance.
[0194] When spatial transfer clues appear in the current observations but the target has not yet been confirmed, the system prompts the second navigation model in the navigation decision context to give priority to candidate navigable viewpoints for entering new space.
[0195] S77: Enhanced directional constraints.
[0196] When natural language navigation instructions contain directional words, the system prompts the second navigation model in the navigation decision context to prioritize candidate navigable viewpoints that meet the directional constraints.
[0197] S78: Re-decision.
[0198] When the initial action of the second navigation model violates the candidate action constraints, stopping rules, or negative action rules in the triggered navigation skill, the system will re-input the reason for the violation and the relevant skill constraints into the second navigation model, requiring it to output the corrected action.
[0199] In this way, external navigation skills are not merely provided as textual experience for the model to reference, but rather participate in the action selection of the second navigation model as a triggerable decision constraint. The second navigation model uses the local problem-solving capabilities demonstrated by the first navigation model in key navigation states, rather than mechanically replicating the complete trajectory of the first navigation model in a specific environment.
[0200] Step S8: Construct the joint navigation decision context and output the final navigation action. The purpose of step S8 is to provide the current task information, historical state, and triggered navigation skill constraints to the second navigation model so that it can output the final navigation action.
[0201] S81: Construct the joint navigation decision context.
[0202] The system constructs a joint navigation decision context by combining the current natural language navigation instructions, current viewpoint observation information, candidate navigable viewpoint set, structured task progress status, most recent original trajectory fragment, and triggered navigation skill constraints.
[0203] Among them, the current natural language navigation command is used to provide the global task objective; the current viewpoint observation information is used to provide the current environmental state; the candidate navigable viewpoint set is used to limit the range of executable actions; the structured task progress status is used to provide long-term task progress information; the most recent original trajectory fragment is used to maintain the continuity of local actions; and the triggered navigation skill constraints are used to provide the basis for action enhancement under the current critical state.
[0204] S82: Output the final navigation action.
[0205] The second navigation model outputs a final navigation action based on the joint navigation decision context. The final navigation action can be a candidate navigable viewpoint or a stop action.
[0206] S83: Perform the action and receive new observations.
[0207] After the environment performs the final navigation action, it returns new observation information.
[0208] S84: Update navigation status.
[0209] The system adds new inference content, action selections, and environmental return observations to the current task's historical navigation trajectory, and updates the structured task progress status, skill trigger status, and visited viewpoint information as needed.
[0210] The above steps are executed cyclically during online navigation until the second navigation model outputs a stop action, or the preset maximum number of navigation steps is reached.
[0211] To verify the effectiveness of the dynamic history reconstruction mechanism in this invention for long-range visual-language navigation tasks, the applicant conducted preliminary experiments based on an R2R indoor visual-language navigation task. The R2R task requires an agent to progressively select the next candidate navigable viewpoint in an indoor environment composed of multiple reachable viewpoints, based on natural language navigation instructions, and to output a stop action when the target area is determined to have been reached. This task includes multi-step navigation, stage switching, intermediate landmark recognition, and stop determination processes, making it suitable as a verification scenario for long-range navigation methods.
[0212] The completed experiments primarily verify the dynamic history reconstruction mechanism in this invention. The experiments use the original ReAct-style visual language navigation method as the baseline. In each navigation step, natural language commands, current viewpoint observations, candidate navigable viewpoints, and the linear historical trajectory are input into the navigation decision model, which then outputs the next action. The comparative method adds a dynamic history reconstruction module to the baseline. An independent history reconstruction model organizes the generated inference content, action selections, and environmental return observations to generate a structured task progress state. This state, along with the most recent few steps of the original trajectory, is then input into the navigation decision model.
[0213] In this experiment, the navigation decision model is responsible for selecting candidate navigable viewpoints and determining stopping actions, while the history reconstruction model is responsible for summarizing and structuring the navigation history. This setup does not involve additional supervised fine-tuning of the navigation decision model and is mainly used to verify the role of the dynamic history reconstruction mechanism itself in long-term history utilization and task progress maintenance.
[0214] This experiment used the Qwen3VL-8b model on the R2R dataset, evaluating 2349 navigation prediction results. The experiment used metrics such as success rate (SR), SPL, navigation error (NE), nDTW, and average navigation steps (steps) for evaluation. SR measures whether the agent can stop correctly near the target area; SPL, the path efficiency success rate, comprehensively measures navigation success rate and path efficiency; navigation error measures the distance between the final stopping position and the target position, with lower values indicating closer proximity; nDTW, the normalized dynamic time warping score, measures the overall similarity between the predicted trajectory and the reference trajectory; and average navigation steps reflects the average number of steps the agent performs during navigation. The experimental results are shown in the table below.
[0215] Table 1 Experimental Results Data Table
[0216] The experimental results show that, compared to the baseline method without dynamic history reconstruction, the addition of dynamic history reconstruction improved the navigation success rate from 28.51% to 31.91%, an increase of 3.40 percentage points; the SPL increased from 20.61% to 21.89%, an increase of 1.28 percentage points; and the navigation error decreased from 7.27 meters to 6.82 meters, indicating that the agent's final stopping position is closer to the target area. The nDTW slightly improved from 40.02 to 40.10, indicating that the overall similarity between the predicted trajectory and the reference trajectory remained relatively stable and slightly improved after the addition of dynamic history reconstruction.
[0217] Meanwhile, with the addition of dynamic history reconstruction, the average number of navigation steps increased from 7.25 to 7.92. This phenomenon indicates that, after introducing structured task progress states, the agent tends to explore more in some long-term tasks rather than stopping prematurely before the target is fully confirmed. Combining the changes in success rate, SPL, and navigation error, it can be seen that although the dynamic history reconstruction mechanism results in a slightly longer navigation path, it improves overall task completion and the stability of finally reaching the target area.
[0218] To further illustrate the impact of the dynamic history reconstruction mechanism on navigation trajectory quality, Figure 5 A specific example of predicted trajectories in a navigation task is provided to demonstrate the differences between the baseline method's predicted trajectory, the predicted trajectory after incorporating dynamic historical reconstruction, and the reference trajectory under the same natural language navigation instructions. Figure 5 As shown, the green dashed line represents the reference trajectory, the blue solid line represents the predicted trajectory after incorporating dynamic historical reconstruction, the red solid line represents the trajectory predicted by the baseline method, and the gray line segment represents the preceding trajectory portion traversed by different methods.
[0219] Further observation of the navigation trajectory reveals that the dynamic history reconstruction mechanism is more helpful for long-range instructions involving multiple intermediate landmarks or room transitions. Structured task progress status helps the model distinguish between completed navigation phases and subsequent objectives, reducing the risk of the model missing key information, repeatedly visiting already explored areas, or stopping prematurely in long historical inputs.
[0220] It should be noted that the above experiments primarily verified the dynamic history reconstruction mechanism in this invention. The external navigation skill library construction, skill triggering, and skill enhancement decision-making will be further verified in subsequent experiments. Further analysis of the contribution of each module to long-range visual-language navigation performance can be conducted through ablation comparisons between the baseline method, methods with only dynamic history reconstruction, methods with only the external navigation skill library, and the complete method.
[0221] The core inventive points and protective points of this invention are as follows: This invention does not simply add a history summarization module or experience retrieval module to existing visual language navigation methods, but rather forms a holistic technical solution across the entire chain of "navigation skill extraction—skill-triggered decision enhancement—dynamic history reconstruction—joint context construction." The four core innovations of this invention work together to form a complete decision-making chain from skill extraction and state triggering to action constraints, which can be summarized as follows: (1) The greatest innovation lies in proposing a skill enhancement decision-making mechanism for long-range visual-language navigation. This invention designs navigation skills stored in an external navigation skill library as structured skill units composed of applicable conditions and action constraints. The applicable conditions are used to determine whether the current natural language instructions, visual observations, and task progress status meet the skill triggering requirements. The action constraints are used to limit the selection of candidate navigable viewpoints, the allowed conditions for stopping actions, and actions that should be avoided, such as repeated visits, erroneous backtracking, and inconsistent directions. When a skill is triggered, it is not merely used as textual experience for the second navigation model to refer to, but actually participates in the selection and verification of the current action. When the initial action generated by the second navigation model violates the corresponding skill constraints, the action decision is re-made based on the skill constraints, thereby enabling the navigation skill to form executable decision constraints for key navigation states.
[0222] (2) The second innovation lies in proposing a structured navigation skill extraction mechanism based on key turning points of the successful trajectory of the first navigation model. This invention does not store or retrieve the complete successful trajectory of the first navigation model, but instead identifies key navigation turning points in the trajectory that cause task phase switching, spatial transformation, direction execution, repeated access avoidance, continued exploration, or correct stopping. From these key navigation turning points, it extracts the applicable state before the action occurs, the successful action implemented by the first navigation model, and the corresponding avoided or negative action. After deleting the specific environment number, specific viewpoint number, and complete path information, the above local decision-making relationship is converted into a "state condition-action constraint" type navigation skill that can be used across tasks and environments, thereby realizing the transfer of the local decision-making ability of the first navigation model to the second navigation model, rather than making the second navigation model imitate the specific trajectory.
[0223] (3) The third innovation lies in proposing a dynamic history reconstruction mechanism that balances long-term context management and skill triggering. Addressing the problem that the continuous accumulation of reasoning content, action selection, and environmental observation during navigation leads to increased model input with each navigation step, context redundancy, and dilution of key information, this invention dynamically reconstructs the historical trajectory according to a fixed period or preset events. It organizes the earlier original linear history into a structured task progress state while retaining the most recent few steps of the original trajectory. The structured task progress state can include the current task stage, path progress, completed and uncompleted sub-goals, visited or avoided areas, valid navigation cues, target confirmation status, and next step decision bias. On the one hand, this mechanism replaces the continuously growing early original history with a structured state, controlling the length of the navigation decision context and retaining key long-term information. On the other hand, at least some fields in the structured task progress state can serve as the state basis for navigation skill triggering, matching, and action constraint judgment, thus simultaneously supporting long-term memory management and skill enhancement decision-making.
[0224] (4) The fourth innovation lies in proposing a joint decision context construction mechanism that coordinates "navigation skill constraints, structured task progress status, and short-term original trajectory". This invention does not continuously input the complete linear history into the second navigation model, nor does it completely replace the recent original trajectory with the historical summary. Instead, the structured task progress status carries the compressed and organized long-term task information, the recent few steps of the original trajectory retain local observation changes and action continuity, and the triggered navigation skills provide action constraints under the current critical state. Through the above division of labor, the joint decision context can retain long-term task progress and short-term local details while controlling the input length, and use the structured task progress status to support skill triggering and action constraints, enabling the second navigation model to determine candidate navigable viewpoints or stop actions accordingly.
[0225] While the invention has been described herein with reference to specific embodiments, it should be understood that these embodiments are merely examples of the principles and applications of the invention. Therefore, it should be understood that many modifications can be made to the exemplary embodiments, and other arrangements can be designed without departing from the spirit and scope of the invention as defined by the appended claims. It should be understood that different dependent claims and features described herein can be combined in ways different from those described in the original claims. It is also understood that features described in conjunction with individual embodiments can be used in other described embodiments.
Claims
1. A long-range visual-language navigation method, characterized in that, Includes the following steps: S1. Obtain the navigation trajectory generated by the first navigation model in the historical visual language navigation task, and filter the successful trajectory or high-quality trajectory from the historical navigation trajectory according to the preset task success conditions and trajectory quality conditions. S2. Identify key navigation turning points in the successful or high-quality trajectory based on the navigation skill template; S3. Based on the identified key navigation turning points, generate structured navigation skill units according to the corresponding navigation skill templates; S4. Perform de-contextualization processing on the structured navigation skill unit and construct an external navigation skill library; S5. The second navigation model performs online navigation and dynamic history reconstruction. S6. Trigger applicable navigation skills based on the current navigation status; S7. The triggered navigation skills are used as decision constraints to participate in the action selection of the second navigation model; S8. Output the final navigation action and update the navigation state.
2. The long-range visual-language navigation method according to claim 1, characterized in that, The first navigation model is a model with visual language navigation capability. The historical visual language navigation task can be an indoor visual language navigation task, a robot navigation task, a virtual environment navigation task, or other navigation tasks based on natural language instructions. The navigation trajectory includes at least natural language navigation instructions, viewpoint observation information for each navigation step, a set of candidate navigable viewpoints, inference content and actions generated by the first navigation model, environmental return observations after the actions are executed, and the final navigation result. The success conditions for the mission can be determined based on the final stopping position, the arrival status of the target area, and the completion status of intermediate landmarks. The trajectory quality conditions can be determined based on information such as repeated visits, invalid detours, and navigation path length.
3. The long-range visual-language navigation method according to claim 1, characterized in that, The navigation skill templates include at least the following five categories: The first category is the landmark switching skill template, which describes the local decision-making ability of an agent to switch from the current instruction stage to the next instruction stage after observing or passing an intermediate landmark. The second category is the premature stop inhibition skill template, which describes the agent's ability to avoid outputting a stop action and continue comparing candidate navigable viewpoints for local decision-making before the final goal is confirmed. The third category is the repeated access avoidance skill template, which describes the agent's local decision-making ability to avoid returning to an invalid region when the candidate action points to a recently visited region and there is no clear need to backtrack. The fourth category is the spatial transfer skill template, which describes the agent's local decision-making ability to choose to enter a new space when no target is found in the current area and spatial transfer clues such as doorways, corridors, stairs, and open areas appear in the observation. The fifth category is the direction constraint skill template, which describes the agent's local decision-making ability to select candidate navigable viewpoints consistent with the direction constraints when there are direction constraints such as left turn, right turn, straight ahead, and return in natural language navigation instructions.
4. The long-range visual-language navigation method according to claim 1, characterized in that, Step S2 specifically includes the following steps: S21. Divide the successful trajectory or high-quality trajectory into multiple local trajectory segments according to the number of navigation steps, instruction fragments, or local time windows; S22. Identify key landmarks, direction words, target areas, spatial transformation words, and phased actions in the natural language navigation instructions from the local trajectory segments; S23. Combining current viewpoint observation information, candidate navigable viewpoint set and historical trajectory changes, determine whether there are key behaviors such as phase switching, spatial transfer, direction execution, stop judgment or repeated access avoidance in local trajectory segments; S24. Match the identified key behaviors with preset navigation skill templates to obtain key navigation turning points and their corresponding skill template types.
5. A long-range visual-language navigation method according to claim 1, characterized in that, Step S3 specifically includes the following steps: S31. Determine the skill type of the structured navigation skill unit based on the skill template type corresponding to the key navigation inflection point; S32. Extract applicable instruction patterns based on natural language navigation instructions and local trajectory segments corresponding to key navigation turning points; S33. Extract visual observation conditions based on the observation information before and after the key navigation turning point; S34. Extract historical state conditions based on the historical trajectory changes before and after key navigation turning points; S35. Based on the successful actions and avoided actions of the first navigation model at key navigation inflection points, generate candidate action constraints, stopping rules, and negative action rules. S36. Determine the action intervention method based on the skill type.
6. The long-range visual-language navigation method according to claim 1, characterized in that, Step S4 specifically includes the following steps: S41. Delete the specific house number, scene number, dataset internal environment identifier, candidate navigable viewpoint number, specific path number, and complete path or action sequence in the structured navigation skill unit to avoid navigation skills relying on fixed navigation routes in a single environment. S42. Abstract the landmarks, objects, spaces and positional relationships in a specific environment into general state expressions such as intermediate landmarks, target areas, visited areas, new spaces, spatial transfer cues and directional cues, while retaining applicable instruction patterns, visual observation conditions, historical state conditions, candidate action constraints, stopping rules and negative action rules. S43. Based on skill type, triggering conditions, applicable status and action constraints, deduplicate, merge and filter the de-contextualized structured navigation skill units, and write the structured navigation skill units that meet the preset requirements into the external navigation skill library for skill triggering and calling when the second navigation model is navigating online. For skill units with unclear applicable conditions, low reliability of source trajectory or highly repetitive content, reduce their priority or do not write them into the external navigation skill library.
7. The long-range visual-language navigation method according to claim 1, characterized in that, The conditions for triggering the dynamic history reconstruction are at least one of the following: the length of the historical record exceeds a preset threshold, a fixed navigation step interval is reached, no new key landmarks or spatial clues are observed for several consecutive steps, repeated visits, back-and-forth movement or path stagnation occur, the second navigation model is ready to output a stop action, and the current candidate action has the risk of backtracking, inconsistent direction, mismatched stage or premature stop.
8. A long-range visual-language navigation method according to claim 1, characterized in that, The triggering rules for applicable navigation skills are as follows: When the structured task progress status indicates that an intermediate landmark has appeared or the corresponding sub-target has been completed, but the subsequent target has not yet been completed, a landmark conversion skill is triggered; when the second navigation model has a tendency to stop, but the structured task progress status lacks confirmation information of the final target or target area, a premature stop suppression skill is triggered; when there is a viewpoint among the candidate navigable viewpoints pointing to a recently visited area, and there is no clear need to backtrack in the current task stage, a revisit avoidance skill is triggered; when a doorway, corridor, staircase, open area, or other spatial transfer clue appears in the current viewpoint observation, and the target has not yet been confirmed, a spatial transfer skill is triggered; when the natural language navigation instruction contains directional requirements, and there are corresponding directional differences among the candidate navigable viewpoints, a directional constraint skill is triggered; when a navigation skill meets the triggering conditions, the system outputs the triggered navigation skill and its triggering basis, for step S7 to generate corresponding navigation decision constraints; When multiple navigation skills meet the triggering conditions at the same time, the order of use can be determined based on the current task stage, action risk type, skill source reliability, or preset skill priority. Multiple navigation skills can also be triggered simultaneously. When no navigation skill meets the triggering conditions, the current navigation step is executed according to the decision-making process without additional skill constraints.
9. A long-range visual-language navigation method according to claim 1, characterized in that, Step S7 specifically includes the following steps: S71. Based on the skill type of the triggered navigation skill and the current triggering reason, generate candidate action constraints, stopping rules and negative action rules that are adapted to the current navigation state, and add them to the decision context of the second navigation model. S72. Adjust the selection priority of candidate navigable viewpoints based on the candidate action constraints in the triggered navigation skill; S73. When the second navigation model tends to output a stop action, determine whether to allow the stop based on the target confirmation status, the sub-targets to be completed, and the stop evidence in the structured task progress status; if the final target or target area has not yet been confirmed, or there are still sub-targets that have not been completed, then suppress the stop action and require the second navigation model to continue to select the next action from the candidate navigable viewpoints. S74. After the second navigation model generates the initial navigation action, it determines whether the initial navigation action conforms to the candidate action constraints, stopping rules, and negative action rules in the triggered navigation skill. When the initial navigation action does not conform to the corresponding skill constraints, the violated constraint content and corresponding reasons are provided to the second navigation model again, requiring it to regenerate the corrected navigation action.
10. A long-range visual-language navigation system, applied in a long-range visual-language navigation method as described in any one of claims 1-9, characterized in that, Includes the following modules: The historical navigation trajectory processing module is used to obtain the navigation trajectory generated by the first navigation model in the historical visual language navigation task, and to filter successful or high-quality trajectories based on the task completion status and trajectory quality. The structured navigation skill construction module is used to analyze the successful or high-quality trajectory, identify key navigation turning points by combining preset navigation skill templates, and extract applicable instruction patterns, visual observation conditions, historical state conditions, successful actions and actions to be avoided from the key navigation turning points to generate structured navigation skills with triggering conditions, candidate action constraints, stopping rules and negative action rules. It is also used to decontextualize, deduplicatize, merge, and filter structured navigation skills, and write the processed structured navigation skills into an external navigation skill library; The dynamic history reconstruction module is used to dynamically reconstruct the continuously accumulated reasoning content, action selection and environmental return observations when the second navigation model performs long-range visual language navigation tasks. It organizes the earlier original linear history into a structured task progress state, while retaining the original trajectory of the most recent steps. The navigation skill triggering and constraint decision module is used to determine whether the current navigation state meets the triggering conditions of one or more structured navigation skills in the external navigation skill library based on the current natural language navigation instructions, current viewpoint observation information, candidate navigable viewpoint set, structured task progress status, and the most recent original trajectory when necessary. When the triggering condition is met, the triggered navigation skill, the triggering reason and its action constraints are output. The navigation skill is then used to participate in the determination of the current navigation action of the second navigation model by adjusting the selection priority of candidate navigable viewpoints, constraining the stopping action, and verifying or re-deciding the preliminary actions that do not meet the skill constraints. The joint navigation decision and state update module is used to construct a joint navigation decision context by combining the current natural language navigation command, current viewpoint observation information, candidate navigable viewpoint set, structured task progress status, most recent original trajectory, and triggered navigation skill constraints, and input it into the second navigation model; it is also used to receive candidate navigable viewpoints or stop actions output by the second navigation model, obtain new observation information after the environment executes the actions, and update the historical navigation trajectory, structured task progress status, skill trigger status, and visited viewpoint information.
