Visual language navigation method and system for continuous environment

Abstract

The invention provides a visual language navigation method and system for a continuous environment, and relates to the field of visual language navigation, based on a language instruction and an environment topological graph initialized to be null, an intelligent agent autonomously navigates to a specified position in an unknown continuous environment by adopting a step-by-step navigation mode, and the navigation efficiency is improved. The specific operation of each step-by-step navigation comprises the following steps: acquiring a local panoramic image and a current environment topological graph of the current position of the intelligent agent; deadlock detection is carried out based on meta-cognitive monitoring, candidate waypoints in the local panoramic image are identified if deadlock is not detected, node features of the candidate waypoints are generated and added to the current environment topological graph, and the optimal waypoint is selected from the current environment topological graph; generating a bottom action instruction of the current position of the intelligent agent through an active reasoning obstacle avoidance method based on a depth map; according to the invention, meta-cognitive monitoring, gating multi-modal fusion decision and inference-reaction active obstacle avoidance are organically combined, and a systematic solution is provided for visual language navigation in a continuous environment.

Description

Technical Field

[0001] This invention relates to the field of visual language navigation, and more specifically to a visual language navigation method and system for continuous environments. Background Technology

[0002] Vision-and-Language Navigation (VLN) is a crucial research task in multimodal artificial intelligence. Its goal is to enable intelligent agents to autonomously navigate complex 3D scenes by visually perceiving their environment and understanding natural language instructions that include spatial relationships, object attributes, and action sequences. VLN tasks not only require agents to possess cross-modal semantic understanding capabilities but also necessitate path planning, obstacle avoidance, and action control in dynamic, partially observable environments. This makes it an important platform for validating multimodal understanding, decision-making reasoning, and robot control technologies.

[0003] As research scenarios have gradually expanded from discrete space to the continuous three-dimensional world, namely VLN tasks in continuous environments (VLN-CE), the introduction of continuous environments has improved the realism of the tasks compared to discrete environments, but it has also brought many challenges, including: Lacking global path planning capabilities, it is prone to deadlock: waypoint predictors usually generate candidate path points based only on local visual information, lacking effective perception of the global topology, making it easy for agents to enter narrow or closed areas and become unable to escape.

[0004] Lack of effective path correction and backtracking capabilities: Once an agent selects an incorrect path, existing methods are usually unable to identify and replan the path in a timely manner, leading to task failure.

[0005] Multimodal feature fusion lacks dynamism: Most methods use static weighting or simple concatenation for feature fusion, which cannot flexibly adjust the importance of different modal features according to the task status, resulting in insufficient decision robustness.

[0006] Obstacle avoidance strategies lack proactive reasoning capabilities: Most existing methods adopt passive reaction-based obstacle avoidance, only adjusting the path after a collision, lacking the ability to proactively predict obstacles and avoid them in advance, resulting in a high navigation failure rate.

[0007] In summary, the shortcomings of existing visual-language navigation technologies in continuous environments significantly limit the navigation performance of intelligent agents in these aspects. Summary of the Invention

[0008] To address the aforementioned problems, this invention proposes a visual language navigation method and system for continuous environments, which organically combines metacognitive monitoring, dynamic cognitive map construction, gated multimodal fusion decision-making, and reasoning-response active obstacle avoidance, providing a systematic solution for visual language navigation in continuous environments.

[0009] According to some embodiments, the present invention adopts the following technical solution: A visual-language navigation method for continuous environments, based on pre-input language commands and an initialized empty environment topology, allows an agent to autonomously navigate to a designated location in an unknown continuous environment using a step-by-step navigation approach. The specific operations of each step-by-step navigation step include: Acquire a local panoramic image of the agent's current location and a topological map of the current environment; Based on metacognitive monitoring, deadlock detection is performed. If no deadlock is detected, candidate waypoints in the local panoramic image are identified, node features of the candidate waypoints are generated and added to the current environment topology map, and the best waypoint is selected from the current environment topology map. Based on the optimal waypoint, the low-level action instructions for the agent's current position are generated through an active reasoning obstacle avoidance method based on depth maps. The deadlock detection employs a dual-scale deadlock detection-backtracking mechanism based on metacognitive theory. Deadlock detection is performed on the local panoramic image. If a deadlock is detected, the optimal backtracking waypoint is determined. After backtracking, deadlock nodes are generated and deadlock features are used to update the node features in the current environment topology graph. The backtracking waypoint is then used as the current position for navigation.

[0010] According to some embodiments, the present invention adopts the following technical solution: A visual language navigation system for continuous environments, employing the aforementioned visual language navigation method for continuous environments, includes: The acquisition module is configured to acquire a local panoramic image of the agent's current location and a topological map of the current environment. The selection module is configured to: perform deadlock detection based on metacognitive monitoring; if no deadlock is detected, identify candidate waypoints in the local panoramic image, generate node features of the candidate waypoints, add them to the current environment topology map, and select the best waypoint from the current environment topology map. The generation module is configured to generate the low-level action instructions of the agent's current position based on the best waypoint and through an active reasoning obstacle avoidance method based on depth maps. The deadlock detection employs a dual-scale deadlock detection-backtracking mechanism based on metacognitive theory. Deadlock detection is performed on the local panoramic image. If a deadlock is detected, the optimal backtracking waypoint is determined. After backtracking, deadlock nodes are generated and deadlock features are used to update the node features in the current environment topology graph. The backtracking waypoint is then used as the current position for navigation.

[0011] According to some embodiments, the present invention adopts the following technical solution: A computer program product includes a computer program that, when executed by a processor, implements the visual language navigation method for a continuous environment.

[0012] According to some embodiments, the present invention adopts the following technical solution: A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the visual language navigation method for a continuous environment.

[0013] According to some embodiments, the present invention adopts the following technical solution: An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform a visual language navigation method for a continuous environment.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) A dual-scale metacognitive deadlock detection and backtracking mechanism is proposed. This mechanism innovatively combines global and local scales to monitor the navigation status in real time, enabling the agent to actively identify and correct abnormal states that occur during navigation, and significantly improving its ability to autonomously correct and escape from trouble. (2) A dynamic waypoint update mechanism based on DPSA attention mechanism was designed. This mechanism dynamically constructs and updates the node features in the environmental topology graph by integrating multimodal information such as visual information, attitude features, navigation step size and historical deadlock features, effectively enhancing the agent's long-term memory and spatial reasoning ability of the environment. (3) A gated multimodal fusion navigation decision unit is introduced. Through an adaptive gating mechanism, the importance of multimodal information in navigation decision is dynamically adjusted, which significantly improves the accuracy and robustness of navigation decision and enables the agent to cope with navigation tasks in different complex environments more flexibly and effectively. (4) A reasoning-response active obstacle avoidance module was constructed. This module innovatively combines active obstacle prediction with a real-time feedback mechanism, enabling the agent to predict potential collision risks in advance and adjust its action strategy in a timely manner, which greatly improves the safety and effectiveness of navigation in complex and continuous environments. Attached Figure Description

[0015] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0016] Figure 1 This is a diagram of the MAIEN framework for Example 1. Detailed Implementation The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0017] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0018] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0019] Example 1 One embodiment of the present invention provides a visual language navigation method for a continuous environment. Based on pre-input language commands and an initialized empty environment topology, the agent autonomously navigates to a specified location in an unknown continuous environment using a step-by-step navigation approach. The specific operations of each step-by-step navigation include: Step S1: Obtain a local panoramic image of the agent's current location and a topological map of the current environment; Step S2: Based on metacognitive monitoring, perform deadlock detection. If no deadlock is detected, identify candidate waypoints in the local panoramic image, generate node features of the candidate waypoints, and add them to the current environment topology map. Select the best waypoint from the current environment topology map. Step S3: Based on the optimal waypoint, generate the low-level action instructions for the agent's current position using a depth map-based active reasoning obstacle avoidance method; The deadlock detection employs a dual-scale deadlock detection-backtracking mechanism based on metacognitive theory. Deadlock detection is performed on the local panoramic image. If a deadlock is detected, the optimal backtracking waypoint is determined. After backtracking, deadlock nodes are generated and deadlock features are used to update the node features in the current environment topology graph. The backtracking waypoint is then used as the current position for navigation.

[0020] The specific implementation process will be explained below, starting with the relevant work.

[0021] I. Related Work 1. Visual-and-Language Navigation (VLN) Foundational research in the field of Virtual Level Navigation (VLN) has primarily focused on discrete environments, structuring these environments as predefined connected graphs. In this setting, the navigation problem is simplified to a sequential decision-making process, where the agent moves between discrete nodes through a graph-based action space. Anderson et al. (2018) first formalized this paradigm through the Room-to-Room (R2R) benchmark task, which requires an agent to move along connected graph nodes in a simulated environment based on natural language instructions. These nodes correspond to panoramic viewpoints and are connected by traversable edges. The agent needs to select a predefined action (e.g., "go to node X") based on a fused visual and textual representation.

[0022] This discretized modeling facilitates controlled experiments but also introduces significant limitations: the agent operates within an artificially constrained action space, bypassing key challenges in the real world such as continuous motion control and partial observability; this structured simplification fundamentally limits the ability of discrete VLN methods to be transferred to real-world physical navigation scenarios.

[0023] 2. Visual Language Navigation in Continuous Environments (VLN-CE) To bridge the gap between simulated environments and real-world scenarios, Krantz et al. (2020) proposed Vision-and-Language Navigation in Continuous Environments (VLN-CE), shifting the research paradigm from discrete navigation based on connected graphs to autonomous movement in continuous three-dimensional space. In VLN-CE, the agent needs to perform low-level action control (such as linear or angular velocity control) while processing egocentric visual flow, requiring it to possess multiple capabilities including environmental perception, localization, and dynamic path planning.

[0024] Subsequently, Hong et al. (2021) proposed a waypoint prediction method based on the Matterport3D dataset, which learns to generate navigable intermediate targets, thus compensating for the lack of predefined nodes in continuous environments. However, VLN-CE agents still have several shortcomings: such as a lack of global topology awareness and path correction capabilities, making the agent prone to deadlock; in addition, frequent collisions during exploration lead to task failure. The root cause of these problems lies in the agent's lack of forward-looking prediction capabilities for obstacle paths in path planning, as well as the lack of an effective mechanism for recovery from erroneous trajectories, highlighting the fundamental inadequacy of existing path recovery strategies.

[0025] 3. Metacognitive Theory This theory refers to an individual's ability to monitor and regulate their own cognitive processes. Metacognition comprises two core components: metacognitive knowledge and metacognitive regulation. Metacognitive knowledge includes an individual's understanding of cognitive tasks, cognitive strategies, and their own cognitive state; while metacognitive regulation involves the planning, monitoring, evaluation, and regulation of cognitive activities. In recent years, metacognitive theory has been increasingly introduced into the fields of artificial intelligence and robot navigation. In artificial intelligence, metacognition is applied to robot planning and decision-making processes. Intelligent agents use metacognitive mechanisms to continuously monitor their own state and task performance quality, and can proactively identify and correct potential problems, thereby significantly improving their adaptability and robustness in complex task environments.

[0026] 4. The Challenges of Deadlock in VLN-CE Deadlock is a critical failure mode in VLN-CE, where agents often become trapped in areas without accessible exits, preventing further progress. This is typically due to incomplete environmental perception or inappropriate action sequence selection. While existing methods employ reactive strategies such as backtracking or random exploration, these methods lack a systematic mechanism for detecting deadlock states and do not effectively utilize linguistic context for path recovery. This phenomenon prompted this invention to explore path replanning guided by metacognitive theory, thereby avoiding irreversible navigation decisions. The MAIEN framework proposed in this invention establishes a theoretical standard for metacognitive theory in visual-language navigation tasks, endowing agents with metacognitive capabilities, enabling them to assess their own navigation state in real time and identify and correct path decisions.

[0027] 5. Topology Maps for Navigation In VLN-CE, topology maps are a core component enabling agents to follow natural language instructions. Early methods focused on discretized abstraction, representing the environment as a graph where nodes represent key locations and edges define traversable paths. However, these methods face challenges in terms of geometric granularity and dynamic adaptability in continuous spaces, making them difficult to meet the requirements.

[0028] ETPNav (Continuous Environment Visual Language Navigation Based on Evolutionary Topology Programming) enhances navigation robustness by utilizing streaming sensor data to generate and update the topology map online and in real time. However, ETPNav also has a key limitation: insufficient handling of node loss caused by collisions and environmental unreachability. In complex environments, the agent may find that predefined nodes are unreachable, which disrupts the structural connectivity and information integrity of the topology map, thus affecting the reliability and accuracy of navigation.

[0029] To address this issue, the MAIEN framework proposed in this invention introduces a novel iterative online update mechanism that explicitly models the dynamic correspondence between target nodes and reachable locations of the agent. This mechanism can correct inconsistencies in the topology graph caused by collisions or inaccessibility in real time, ensuring the structural consistency and information integrity of the topology graph, thereby providing a strong guarantee for language-command-based navigation.

[0030] 6. Obstacle avoidance issues in VLN-CE In Visual Language Navigation-CE (VLN-CE) in continuous environments, low-level obstacle avoidance capabilities are crucial for ensuring the safe and reliable execution of high-level path planning by the agent. Traditional control strategies, such as random forest (RF) controllers, have significant limitations: (1) they lack proactive obstacle prediction capabilities, reacting only passively after a collision occurs; (2) their steering angle and step size adjustment strategies are usually coarse and difficult to precisely adapt to specific obstacle configurations. This often leads to agent deadlock, significantly reducing navigation efficiency.

[0031] ETPNav employs a trial-and-out mechanism. While this method seeks feasible paths by testing and adjusting the agent's actions, it incurs high computational overhead in complex scenarios and has limited adaptability to dynamic obstacle changes. To overcome these shortcomings, the MAIEN framework proposed in this invention designs a reasoning-response obstacle avoidance module.

[0032] II. This method As one embodiment, this invention provides a continuous environment visual language navigation method. From the perspective of meta-cognition and active inference, it proposes for the first time a visual language navigation theoretical framework—MAIEN (Meta-cognitive and Active-Inference driven Embodied Navigation)—that integrates meta-cognition monitoring, dynamic waypoint feature updating, gated multimodal fusion decision-making, and active inference obstacle avoidance mechanisms. Figure 1 As shown, the MAIEN framework systematically integrates a self-state monitoring module with a dual-scale deadlock detection-backtracking mechanism, a dynamic waypoint updater based on DPSA attention, a gated multimodal fusion navigation decision unit, and an active reasoning obstacle avoidance module. The following is a detailed description of these four core components with metacognitive regulation capabilities.

[0033] 1. Self-state monitoring module for dual-scale deadlock detection and backtracking mechanism The self-state monitoring module endows the intelligent agent with metacognitive capabilities. Dual-scale deadlock detection compares positional changes and semantic deviations in real time. Once an anomaly is detected, it backtracks to the optimal waypoint, enabling the agent to monitor its own state in real time and efficiently correct decisions when in trouble.

[0034] First, let's analyze the deadlock problem in the VLN-CE task: The navigation problem in the VLN-CE task involves an agent relying on a waypoint predictor to generate and select reachable candidate locations (i.e., candidate waypoints) from a local panoramic image. This predictor, as a probabilistic model, is defined on a finite action space (candidate waypoints) and generates a probability distribution based on the current local visual observations (panoramic image) and the embedding of the language command, where represents the candidate waypoint, visual input, observation state, and language command, respectively. Although some methods (e.g., ETPNav) introduce a progress metric reflecting geometric proximity to the target region and employ a weighted combination for decision-making, this locally-aware mechanism remains a major cause of navigation deadlocks frequently encountered by agents.

[0035] The root cause of this problem lies in the inherent limitations of local perception. Waypoint predictors can only perceive fragments of the environment within their current 360° field of view and cannot predict the environment beyond that field of view, especially areas obscured by walls, large furniture, or door frames. A candidate point that appears geometrically advantageous from the current perspective (e.g., approaching a target or aligned with directional indicators such as "turn left") may lead to a path later blocked by unforeseen obstacles (such as a closed door, an impassable narrow passage, or a deadlock outside the field of view). This forces the agent to make decisions only within a localized and uncertain field of view, unable to predict the feasibility of paths after traversing a room or around a corner.

[0036] Another deeper challenge lies in effectively fusing language instructions with visual scenes. While VLN-CE models typically employ cross-modal attention mechanisms (such as in architectures like HAMT or DUET) for multimodal fusion, this fusion often lacks robustness and accuracy in complex environments. Key instruction constraints (such as "avoid the study" or "turn right after passing the blue sofa") may not be effectively aligned in visually noisy scenes or may be obscured by geometric heuristics. This can cause the predictor to ignore key turning points or choose the wrong path. Semantic-spatial ambiguities (such as mapping "the second door on the left" to multiple visually possible candidate paths) can cause the agent to take the wrong path.

[0037] In addition, a key bottleneck is historical path dependence and the lack of an effective backtracking mechanism; most VLN-CE models lack explicit, structured environmental topology memory (although recent studies like FAST-MLP attempt to introduce path memory units); once an incorrect decision leads the agent into a poorly connected region (e.g., a trap-like storage room with no exit), it struggles to identify deadlock states and execute an effective retreat; since typical models only plan single-step actions, they cannot foresee the feasibility of multi-step paths; incorrect action sequences can lead to chain reactions: the agent may iterate repeatedly in a local subspace or continue moving towards a local deadlock, unable to autonomously return to critical path nodes to replan.

[0038] Based on the aforementioned deadlock problem, this embodiment designs a dual-scale deadlock detection-backtracking mechanism based on metacognitive theory during the agent's navigation process, endowing the agent with metacognitive capabilities so that it can promptly identify and escape from a deadlock state.

[0039] At a local scale, the agent continuously records its corresponding historical position sequence. and the distance sequence between each path node and its preceding path nodes ,in, They represent the first time. The specific location of the agent during the step and the Euclidean distance between the current waypoint and its previous waypoint. This represents the total number of navigation steps taken by the agent; the agent obtains a certain number of steps. Average position change This is used to characterize the magnitude of historical position changes, and its calculation formula is: (1) When an agent gets stuck in a deadlock state, it is often unable to navigate effectively, and its movement efficiency in local environments is low. When the value is small, it indicates that the agent may be in a deadlock state.

[0040] At the global scale, the agent uses a pre-trained CLIP model to calculate the matching degree between the current visual information and the language instructions in real time, thereby measuring whether the agent's overall navigation direction deviates from the target. The semantic matching degree formula is as follows: (2) in, Indicates current visual information (i.e., panoramic RGB images captured by the intelligent agent), and language commands. The degree of matching, This represents the feature map after encoding by the CLIP encoder.

[0041] To avoid the agent misjudging deadlock during the early exploration phase, the deadlock detection mechanism will only be activated after the agent has navigated a certain number of steps, and only after local position changes. and global semantic matching degree Below the set threshold If so, the agent is considered to be in a deadlock state.

[0042] Upon detecting a deadlock, the visited waypoints are traversed to determine the optimal backtracking waypoint; specifically, a waypoint that satisfies certain conditions is found. and Visited waypoints ,in, Indicates the first The waypoint where the agent is located during the pacing process. Compare the waypoints. To obtain the optimal backtracking waypoint That is, in order to ensure If the conditions are met, choose the larger number of steps. This enables deadlock backtracking and increases navigation efficiency.

[0043] The agent returns to the position corresponding to the backtracking point. The nodes traversed during the backtracking process are considered deadlock nodes, and their corresponding positions and the distances of their position changes are recorded. The number of node visits and visual information are embedded separately as deadlock features into each waypoint so that these deadlock paths can be avoided during subsequent navigation.

[0044] 2. Dynamic Waypoint Updater Based on DPSA Attention This embodiment designs a novel waypoint updater based on the DPSA (Deadlock Perception-Self-Attention) mechanism. As a metacognitive adjustment module of the agent, DPSA fuses multimodal information of waypoints, adding topological structure information, visual features, attitude features, navigation step size features, and deadlock features. It dynamically updates the node feature representation of waypoints, enhances the agent's long-term memory of the environment and spatial reasoning ability, and guides the agent to avoid deadlock paths.

[0045] The waypoint updater consists of two sub-modules: a feature generation module and a cross-modal interaction module. The feature generation module generates visual features, attitude features, navigation step size features, semantic embedding vectors, and deadlock features for waypoints. The cross-modal interaction module performs cross-modal interaction on the above features to obtain node features.

[0046] Semantic embedding vector generation: The encoding of language instructions is implemented based on a pre-trained BERT model. The core processing unit of BERT is a multi-layer Transformer encoder, each layer containing a multi-head self-attention mechanism and a feedforward neural network sublayer. Through layer stacking, the model abstracts and refines more generalized high-level semantic representations layer by layer, finally outputting semantic embedding vectors. The vocabulary, structure, and deep contextual information of the integrated instructions are used as input to the cross-modal interaction module and fused with features from other modalities.

[0047] Generation of visual features, pose features, navigation step size features, and deadlock features: In the topology graph node encoding part, for each node... (i.e., waypoints) Add visual features extracted from RGB and depth images by the deep learning model ResNet. Pose features representing the position and orientation information of an agent The historical access time of a node is used as a navigation step size feature for analyzing navigation inertia. Deadlock characteristics of nodes .

[0048] Cross-modal interaction: After encoding the nodes, cross-modal interaction is performed on them, combining the encoded nodes with semantic embedding vectors. By incorporating topological graph attention and deadlock feature attention mechanisms, a multi-layer Transformer network is used for training and alignment. The following attention weight formula is designed to enhance the guiding role of deadlock features: (3) in, It is based on visual features Posture characteristics Navigation step size characteristics Semantic embedding vectors Deadlock characteristics Concatenate the encoded vector, It is a spatial matrix representing topological relationships. It is the feature dimension. It is a learnable weight matrix.

[0049] Finally, based on the normalized attention weights Generate node feature representations Used for updating.

[0050] 3. Gated multimodal fusion navigation decision unit After the waypoint updater generates new node feature representations, these features, although incorporating multimodal context and deadlock information, may still contain redundant or misleading content. Especially in complex environments, some features may affect the accuracy of navigation decisions due to local perception errors or historical biases. To address this, referencing the dynamic adjustment and pre-screening mechanism for information in metacognitive theory, this embodiment proposes a gating refinement mechanism applied to the newly generated node feature representations. By adaptively adjusting the weights of each dimension of the node features, irrelevant or low-confidence information is suppressed, and feature representations highly correlated with the current command and environmental state are strengthened.

[0051] Specifically, the generated node features Input a lightweight feedforward network (MLP) to generate dimension-wise gated weights. The formula is as follows: (4) in, and These are learnable parameters, and the weights reflect the importance of each dimension of the feature to the current navigation decision.

[0052] By applying gated weights to the original node features through element-wise multiplication, noise suppression and key information enhancement are achieved. The corresponding formula is as follows: (5) This operation can be viewed as an action on node features. The "soft mask" ensures that high-confidence feature dimensions participate in subsequent path planning.

[0053] Characteristics after refining Normalization is performed, and residual processing is applied using relatively stable original visual features to enhance the image. The expressive power and stability of these features are used to form the final fusion feature output: (6) in, Presentation layer normalization operation, The original visual features were extracted from the panoramic RGB image using a Transformer. Finally, the following method is used as the waypoint scoring function to score the waypoints: (7) in, It is a feedforward network. It is the waypoint score.

[0054] Select the waypoint with the highest rating as the best waypoint for navigation to the current location.

[0055] In previous VLN-CE studies, the optimal waypoint selection often employed a strategy of masking visited points and the current position to avoid repeatedly visiting already visited nodes. This could lead to a deadlock situation where the agent has no better unvisited waypoints to choose from to escape the deadlock, and the waypoint scores of other unvisited waypoints within the deadlock state are still higher than the "stop" score, forcing the agent to remain in the deadlock state. To prevent this, during backtracking, the agent selects the optimal waypoint that satisfies the backtracking strategy through the global topology graph. After backtracking, the agent masks visited points and the current position, and each waypoint has been updated with deadlock features. During path planning, deadlock paths can be effectively avoided, thus achieving path correction.

[0056] In previous VLN-CE models, a student-forcing strategy was used to train the agent so that it could infer the optimal waypoint, thereby inferring the unvisited waypoint closest to the target location. This training logic is consistent with the work presented in this paper, and its loss function is as follows: (8) in, Represents a topology graph and network parameters The optimal action to reach the goal.

[0057] Furthermore, in the VLN-CE task, the maintenance and updating of the topology graph is crucial for path planning. However, traditional methods often fail to reach the target point due to collisions, action execution errors, and other factors when dealing with topology graph updates in continuous environments. This results in the loss of target point information, which seriously affects the accuracy and reliability of path planning.

[0058] This embodiment designs a novel online topology map update mechanism. This mechanism can solve the problem of information loss caused by collisions or unreachability, ensuring the structural consistency and information integrity of the topology map. Specifically, the structure of the topology map is dynamically adjusted by monitoring the navigation status of the agent in real time. When the agent finds that a predefined target point (i.e., the selected optimal waypoint) is unreachable, it decides whether to delete the target point by evaluating the distance to the target point and the number of times the target point has been selected. When the distance between the agent and the target point is less than a preset threshold or the target point has been selected multiple times but is unreachable, a deletion operation is performed, ensuring that the corresponding target point is only removed when the agent actually approaches the target location or the target point is indeed unreachable, avoiding information loss due to misjudgment.

[0059] 4. Active reasoning obstacle avoidance module The core difference between continuous and discrete environment visual-language navigation tasks lies in the fact that in continuous environments, agents need to translate high-level path planning results into specific, continuous low-level action commands (such as forward and turn) and execute them in the simulation environment. Traditional control strategies (such as RF controllers) use predefined "turn-forward" action patterns, which have two significant limitations: 1. Lack of obstacle perception: the passive turning strategy is only triggered after a collision, lacking active detection and avoidance of potential obstacles ahead; 2. Low action execution efficiency: the turning angle and step size adjustment strategies are inefficient, easily leading to trajectory deviation or local deadlock.

[0060] To address this, this embodiment proposes a reasoning-response obstacle avoidance framework based on an active reasoning mechanism. The agent can predict obstacles in advance and adjust its actions accordingly, thereby significantly improving the safety and effectiveness of navigation.

[0061] The depth map-driven environmental perception mechanism forms the core technological foundation for obstacle detection in the VLN-CE task, enabling proactive reasoning to achieve obstacle avoidance. As an information carrier of the spatial geometry of the environment, the depth map provides crucial information for real-time construction of the scene's 3D projection through its pixel-level distance information (recording the distance from each point in the scene to the sensor). By setting a distance threshold (e.g., 1 meter), pixels in the near-field region of the depth map can be identified as potential obstacles. Based on this pixel-level distance information, the spatial distribution characteristics of the obstacles (such as their centroid) and their azimuth angles relative to the agent (such as their orientation angle) can be further calculated, providing computationally achievable perceptual support for obstacle avoidance decisions.

[0062] Specifically, during navigation, the agent uses depth maps to detect the current distribution of obstacles, actively infers feasible paths, and calculates the collision risk of each feasible path to obtain the optimal feasible obstacle avoidance path. For example, if the currently detected obstacles are mainly distributed on the left side of the perception area (manifested as the average distance in the left half of the depth map is significantly smaller than that in the right half), the agent generates an obstacle avoidance command to turn right, completes the obstacle avoidance, and then returns to the target point.

[0063] The reactive obstacle avoidance mechanism constructs a direct response logic based on collision feedback: once a collision occurs, the agent immediately triggers a pre-set emergency turning behavior, forming an efficient "collision-response" closed loop. The key improvement of this mechanism lies in the introduction of a collision event-driven dynamic step size adjustment strategy. Specifically, after consecutive collisions, the system dynamically shortens the compensation step size, thereby reducing the exploration range of a single action, significantly reducing the risk of subsequent collisions, and improving robustness in complex or crowded environments.

[0064] The reasoning-response obstacle avoidance framework proposed in this embodiment constructs a hierarchical collaborative decision-making mechanism. The upper-level decision (reasoning avoidance) uses depth map information to assess and avoid obstacles. The lower-level decision (collision emergency response) serves as a supplement and redundancy guarantee mechanism for the upper-level strategy. It is activated only when active reasoning fails or a dynamic change in the environment causes an actual collision to occur, and then performs obstacle avoidance behavior.

[0065] III. Experiment This embodiment compares the MAIEN method with other state-of-the-art algorithms on the R2R-CE dataset and designs entity experiments to verify the feasibility of the method for transferring entities.

[0066] 1. Task Settings The VLN-CE task requires an agent to perform fine-grained navigation in a continuous 3D indoor environment based on paths described by natural language instructions. Unlike traditional navigation methods based on discrete grid representations, the VLN-CE task takes place in a continuous space simulated by a physics engine.

[0067] Simulation Environment Construction: This task utilizes the Habitat platform for environment construction, which provides accurate spatial representation through high-fidelity environment rendering using the Matterport3D scene dataset. The spatial representation employs a 3D mesh map, incorporating precise geometric topology and material properties.

[0068] Panoramic perception: To achieve panoramic perception, VLN-CE captures corresponding RGB images and depth observation sequences (covering 0°, 30°, ..., 330°) from 12 isoangular horizontal viewpoints (intervals of 30 degrees) under each agent's pose.

[0069] Action Space: The action space consists of three basic primitives: forward (fixed displacement of 0.25 meters), left turn (+15°), right turn (-15°), and stop. When performing these actions, the agent is constrained by continuous physical kinematics—it cannot pass through obstacles or penetrate solid structures—thus ensuring the realism of the action simulation.

[0070] 2. Dataset The R2R-CE dataset, proposed by Krantz et al. in 2020, is a visual language navigation benchmark designed to extend the original discrete graph-based R2R task to continuous spatial navigation settings that more closely resemble real-world robot deployment scenarios. This dataset uses 90 high-fidelity indoor 3D scenes from the Matterport3D dataset. Low-level control is achieved in the Habitat simulation platform with a step size of 0.25m and a rotation angle of 15°, outputting 640×480 RGB-D observations per step. R2R-CE contains 5,611 shortest path trajectories, each corresponding to an average of 3 English instructions (average 32 words), with an average trajectory length of 9.89m. Officially, it is divided into a training set of 10,819 trajectories, a validation set of 778 seen trajectories, a validation set of 1,839 unseen trajectories, and a test set of 3,408 unseen trajectories.

[0071] 3. Evaluation Indicators Under a unified continuous environment VLN setting, the evaluation metric for R2R-CE is based on navigation success rate. R2R-CE adopts the complete metric system of R2R: NE records the Euclidean distance (m) between the final position and the target, while TL calculates the total actual path length (m); the success rate SR requires the agent to trigger STOP within 3m of the target; SPL further weights SR with the shortest path length to penalize redundant roaming; OSR records the proportion of times the agent's NE is less than 3 meters at any point along the entire trajectory. In addition, the CR (Collision Rate) metric from Habitat is introduced, which is the proportion of steps in the trajectory where collisions occur out of the total number of steps, to measure the obstacle avoidance performance of the MAIEN method.

[0072] 4. Training Setup The experiments were conducted on an NVIDIA RTX 3090. During the pre-training phase, a joint optimization planning module of Masked Language Modeling (MLM) and Semantic Action Prediction (SAP) was employed: MLM enhances instruction semantic understanding through masking and reconstruction; SAP predicts high-level actions with partial observations, introducing action priors to promote deep alignment between language and action, significantly improving cross-task generalization. The method comprises two trainable parts: a waypoint updater for learning deadlock features and a navigator using a gating mechanism to achieve multimodal fusion. Training time was approximately 73 hours, with 45,000 iterations. A pre-trained CLIP-ViT-B / 32 model was used to align instruction and visual information in the dual-scale deadlock detection-backtracking mechanism.

[0073] 5. Comparison with state-of-the-art methods Table I compares the MAIEN method of this embodiment with other state-of-the-art methods on the R2R-CE dataset, and the results are shown in Table 1. Experimental results show that the method achieves state-of-the-art performance in NE, OSR, SR, and SPL. In the Val Seen (validation set scenarios), Val Unseen (validation machine scenarios), and Test Unseen (test set scenarios), the method of this embodiment achieves significant performance improvements, even achieving the best performance across all metrics in the Val Seen part, with OSR, SR, and SPL exceeding the second-place ETPNav method by 7%, 7%, and 4%, respectively. In the Val Unseen part, the method of this embodiment outperforms the second-place ETPNav method by 6% and 5% in OSR and SR, respectively. It should also be noted that the backtracking mechanism is an important component of the MAIEN method. When the agent is stuck in a deadlock state during navigation tasks, it can successfully escape and complete the navigation task. While improving the success rate, it inevitably sacrifices some SPL. However, as shown in Table 1, the SPL performance of the MAIEN method is still comparable to other state-of-the-art VLN methods.

[0074] Table 1. Comparison results with state-of-the-art methods on the R2R-CE dataset.

[0075] IV. Conclusion This embodiment proposes the MAIEN framework, which organically combines metacognitive monitoring, dynamic cognitive map construction, gated multimodal fusion decision-making, and reasoning-response proactive obstacle avoidance, providing a systematic solution for visual-language navigation in continuous environments. Experimental results show that MAIEN significantly outperforms existing methods in navigation success rate, path efficiency, and obstacle avoidance performance on the R2R-CE benchmark, and has been effectively validated on a real robot platform. This research not only theoretically expands the cognitive architecture of embodied intelligence but also provides a feasible paradigm for safe, robust, and efficient navigation in complex dynamic environments.

[0076] Example 2 One embodiment of the present invention provides a visual language navigation system for a continuous environment, employing a visual language navigation method for a continuous environment provided in Embodiment 1, comprising: The acquisition module is configured to acquire a local panoramic image of the agent's current location and a topological map of the current environment. The selection module is configured to: perform deadlock detection based on metacognitive monitoring; if no deadlock is detected, identify candidate waypoints in the local panoramic image, generate node features of the candidate waypoints, add them to the current environment topology map, and select the best waypoint from the current environment topology map. The generation module is configured to generate the low-level action instructions of the agent's current position based on the best waypoint and through an active reasoning obstacle avoidance method based on depth maps. The deadlock detection employs a dual-scale deadlock detection-backtracking mechanism based on metacognitive theory. Deadlock detection is performed on the local panoramic image. If a deadlock is detected, the optimal backtracking waypoint is determined. After backtracking, deadlock nodes are generated and deadlock features are used to update the node features in the current environment topology graph. The backtracking waypoint is then used as the current position for navigation.

[0077] Example 3 One embodiment of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the visual language navigation method for a continuous environment.

[0078] Example 4 In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the visual language navigation method for a continuous environment.

[0079] Example 5 One embodiment of the present invention provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform a visual language navigation method for a continuous environment.

[0080] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0081] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0082] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A visual-language navigation method for a continuous environment, characterized in that, Based on pre-input language commands and an initialized empty environment topology, the agent autonomously navigates to a designated location in an unknown, continuous environment using a step-by-step navigation approach. The specific operations of each step-by-step navigation step include: Obtain a local panoramic image of the agent's current location and a topological map of the current environment; Based on metacognitive monitoring, deadlock detection is performed. If no deadlock is detected, candidate waypoints in the local panoramic image are identified, node features of the candidate waypoints are generated and added to the current environment topology map, and the best waypoint is selected from the current environment topology map. Based on the optimal waypoint, the low-level action instructions for the agent's current position are generated through an active reasoning obstacle avoidance method based on depth maps. The deadlock detection employs a dual-scale deadlock detection-backtracking mechanism based on metacognitive theory. Deadlock detection is performed on the local panoramic image. If a deadlock is detected, the optimal backtracking waypoint is determined. After backtracking, deadlock nodes are generated and deadlock features are used to update the node features in the current environment topology graph. The backtracking waypoint is then used as the current position for navigation.

2. The visual-language navigation method for a continuous environment as described in claim 1, characterized in that, The local panoramic image is a sequence of several RGB image-depth map pairs captured continuously at preset angle intervals.

3. The visual language navigation method for a continuous environment as described in claim 1, characterized in that, The environmental topology graph uses waypoints as nodes and traversable paths between two waypoints as edges. The node features are generated by encoding the visual features, attitude features, navigation step size features, semantic embedding vectors, and deadlock features of the waypoints.

4. The visual-language navigation method for a continuous environment as described in claim 1, characterized in that, The dual-scale deadlock detection combines global and local scales to evaluate the navigation status in real time. At the global scale, it calculates the semantic matching degree between the current visual information and the language command, and at the local scale, it calculates the historical position change amplitude. Using the deadlock threshold of semantic matching degree and historical position change amplitude, it determines whether a deadlock has occurred.

5. The visual language navigation method for a continuous environment as described in claim 4, characterized in that, The backtracking process involves the following steps: Based on semantic matching degree and historical position change magnitude, the optimal backtracking waypoint is determined by weighted fusion. The agent returns to the position corresponding to the backtracking waypoint and treats the nodes traversed during the backtracking process as deadlock nodes. The corresponding location of the deadlock node, the distance of the node's location change, the number of times the node was visited, and visual information were used as deadlock features to re-encode and generate node features in the current environment topology graph.

6. The visual-language navigation method for a continuous environment as described in claim 1, characterized in that, The encoding of the node features is generated using a waypoint update method based on the DPSA mechanism, specifically as follows: Language instructions are encoded using a pre-trained BERT model to generate semantic embedding vectors; Visual features are extracted from RGB images and depth maps using the deep learning model ResNet. The attitude feature is composed of the agent's position and orientation information; Use the historical access time of a node as a navigation step size feature for analyzing navigation inertia; Under the topological graph attention mechanism and the deadlock feature attention mechanism, cross-modal interaction is performed on visual features, pose features, navigation step size features, semantic embedding vectors and deadlock features to obtain node features.

7. The visual language navigation method for a continuous environment as described in claim 1, characterized in that, The selection of the optimal waypoint from the current environmental topology map employs a navigation method based on a gating mechanism, specifically as follows: By using a gating refinement mechanism, the weights of each dimension of node features are adaptively adjusted to strengthen the feature representation that is highly correlated with the current instruction and the environmental state, thus obtaining the final node fusion features. Calculate waypoint scores based on node fusion features; The best waypoint is selected based on the waypoint score.

8. A visual language navigation system for a continuous environment, characterized in that, A visual language navigation method for a continuous environment as described in any one of claims 1-7, comprising: The acquisition module is configured to acquire a local panoramic image of the agent's current location and a topological map of the current environment. The selection module is configured to: perform deadlock detection based on metacognitive monitoring; if no deadlock is detected, identify candidate waypoints in the local panoramic image, generate node features of the candidate waypoints, add them to the current environment topology map, and select the best waypoint from the current environment topology map. The generation module is configured to generate the low-level action instructions of the agent's current position based on the best waypoint and through an active reasoning obstacle avoidance method based on depth maps. The deadlock detection employs a dual-scale deadlock detection-backtracking mechanism based on metacognitive theory. Deadlock detection is performed on the local panoramic image. If a deadlock is detected, the optimal backtracking waypoint is determined. After backtracking, deadlock nodes are generated and deadlock features are used to update the node features in the current environment topology graph. The backtracking waypoint is then used as the current position for navigation.

9. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement a visual language navigation method for a continuous environment as described in any one of claims 1-7.

10. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to perform a visual language navigation method for a continuous environment as described in any one of claims 1-7.

Images