A visual language navigation method and system based on three-dimensional geometric perception and shape-level representation

Abstract

A visual language navigation method and system based on three-dimensional geometric perception and shape level representation, the method comprising: constructing a Floor-to-Floor multi-layer navigation data set containing viewpoint level three-dimensional observation data and shape perception instructions; the viewpoint level three-dimensional observation data comprises multi-view RGB images, point graphs aligned with the multi-view RGB images pixel by pixel, point graph confidence and relative orientation encoding; through geometric block extraction and mask optimal transmission, the RGB images and the point graphs are aligned in cross-modal geometric perception features to generate geometric enhanced visual features; through a geometric-language alignment module based on a cross-attention mechanism, the geometric enhanced visual features and the shape perception instructions are fused to generate multi-modal features; an incremental topological graph is dynamically constructed and maintained during navigation, and global action decision is made based on the multi-modal features and the incremental topological graph; the navigation strategy is jointly optimized and trained through behavior cloning and data aggregation strategy.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of computer vision, natural language processing, and robot navigation, and particularly to a visual language navigation method and system based on three-dimensional geometric perception and shape-level representation. Background Technology

[0002] Vision-and-Language Navigation (VLN) is a core task in the field of embodied intelligence, requiring agents to simultaneously understand natural language instructions, perceive visual information from the environment, and make continuous movement decisions. In recent years, with the emergence of large-scale Vision-Language Models (VLMs), researchers have begun to leverage their powerful cross-modal understanding and reasoning capabilities to address VLN problems. Existing methods have evolved primarily in two directions: First, pixel-level methods directly process the raw pixels of RGB images or raw points in point clouds to capture fine spatial geometric cues (such as obstacle boundaries and room layouts), which is crucial for accurate localization and obstacle avoidance, but is typically computationally expensive and requires domain-specific adjustments. Second, tag-level methods encode visual inputs (such as voxels or point clouds) into discrete sequences of semantic tags, aligned with language instructions in a shared space. This approach achieves efficient and robust cross-modal reasoning and is easily scalable across different environments. Recently, hybrid designs fusing these two paradigms have also emerged, aiming to simultaneously utilize the efficiency of tag-level methods and the geometric fidelity of pixel-level methods.

[0003] Currently, the most representative VLN methods are navigation agents built directly on pre-trained large-scale vision-language models (such as InstructBLIP, Qwen2.5-VL) or large-scale language models (such as FlanT5, Vicuna), such as NavGPT-2. These methods typically encode visual input (RGB images) as features, align them with language instructions in the model's latent space, and then select the next navigation action using a policy network or directly within the inference framework of an LLM. These methods benefit from the powerful semantic understanding capabilities of VLMs / LLMs, achieving excellent performance in standard single-layer environments (such as the Room-to-Room dataset). They represent the current state-of-the-art in utilizing general-purpose large models for VLN tasks.

[0004] Despite the progress made by the aforementioned large-model-based methods, two fundamental drawbacks remain when dealing with complex multi-layered scene navigation tasks, which are precisely the technical problems that this invention aims to solve:

[0005] (1) Lack of explicit 3D geometry perception: Existing methods mainly rely on visual features extracted from 2D RGB images. These features, whether pixel-level or marker-level, are difficult to effectively capture and understand the 3D geometry of a scene (such as the 3D shape of objects, surface curvature and normals, and spatial topological relationships). At key decision points with significant geometric structures, such as stairs, door frames, and multi-level connections, the lack of 3D contextual information can lead to path ambiguity and increase the risk of navigation failure. Marker-level representation abstracts observations from a fixed viewpoint into a sequence of markers, often ignoring the relationship between shape and texture and discarding key geometric clues such as spatial occlusion boundaries and height changes.

[0006] (2) Domain mismatch between 2D pre-training and 3D navigation requirements: Existing VLMs are typically pre-trained on a large number of 2D image-text pairs, and their visual encoders are adapted to 2D image understanding. When directly applied to 3D navigation tasks, a domain shift occurs. The model struggles to transfer from pre-training and effectively utilize 3D geometric features (such as the overall layout of a room and the volume of obstacles) that are crucial for navigation. This modal mismatch limits the model's ability to uncover geometric cues, resulting in limited performance in complex environments requiring precise spatial reasoning. Summary of the Invention

[0007] To achieve the above objectives, this invention provides a visual-language navigation method and system based on 3D geometric perception and shape-level representation. By introducing a shape-level representation between pixel-level details and marker-level semantics, and utilizing an innovative cross-modal alignment and feature fusion mechanism, 3D geometric priors are seamlessly injected into a frozen large-scale model. This overcomes the performance degradation of existing navigation agents based on large-scale visual-language models in multi-layered complex scenes due to the lack of explicit 3D geometric perception capabilities, significantly improving the spatial understanding and decision-making capabilities of the navigation agent.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] Firstly, a visual language navigation method based on three-dimensional geometric perception and shape-level representation includes the following steps:

[0010] S1. Construct a floor-to-floor multi-layer navigation dataset containing viewpoint-level 3D observation data and shape-aware commands; the viewpoint-level 3D observation data includes multi-view RGB images, point maps aligned pixel-by-pixel with the multi-view RGB images, point map confidence, and relative orientation codes;

[0011] S2. By extracting geometric blocks and transmitting masks optimally, the RGB image and the point map are aligned with cross-modal geometric perception features to generate geometrically enhanced visual features.

[0012] S3. Through a geometry-language alignment module based on a cross-attention mechanism, geometrically enhanced visual features are fused with shape-aware instructions to generate multimodal features.

[0013] S4. Dynamically construct and maintain an incremental topology map during navigation, and make global action decisions based on multimodal features and the incremental topology map;

[0014] S5. The navigation strategy is jointly optimized and trained through behavior cloning and data aggregation strategies.

[0015] Preferably, S1 includes:

[0016] Multi-view RGB images and camera pose information are obtained based on indoor 3D scanning data; a 3D reconstruction network is used to generate a point map that is pixel-by-pixel aligned with the multi-view RGB images, realizing the mapping from pixels to 3D coordinates;

[0017] Shape-aware commands are generated based on multi-view RGB images, and geometric attributes are enhanced on the original navigation commands. Consistency verification is performed on the shape-aware commands. Viewpoint-level 3D observation data and shape-aware commands are stored in a floor-to-floor multi-layer navigation dataset.

[0018] Preferably, the geometric block extraction in S2 includes:

[0019] The dot plot and RGB image are divided into geometric blocks and image blocks of fixed size; geometric features are extracted from the geometric blocks; and visual features aligned with the geometric blocks are extracted from the RGB image.

[0020] As a preferred option, the optimal mask transmission in S2 includes:

[0021] A mask for filtering reliable regions is generated based on the point map confidence level. The optimal transmission problem with mask constraints is constructed based on geometric and visual features and solved iteratively to generate the optimal transmission matrix. The optimal transmission matrix is ​​then fused with the visual features and mapped back to the visual feature space to generate geometrically enhanced visual features.

[0022] Preferably, in S2, generating the optimal transfer matrix includes:

[0023] Computing the relationship between geometric features and visual features Distance, generating cross-modal cost matrix ;

[0024] Establish the optimal transmission problem with mask constraints:

[0025]

[0026] The constraints are:

[0027]

[0028] Where N is the number of geometric blocks;

[0029] The Sinkhorn iterative algorithm is used to solve the regularized over time (OT) problem to obtain the optimal transfer matrix. .

[0030] Preferably, the generated geometrically enhanced visual features include:

[0031] Based on the optimal transmission matrix and visual features Perform feature fusion:

[0032]

[0033] via decoder Aligned geometric features Mapping back to the visual feature space generates geometrically enhanced visual features:

[0034] .

[0035] Preferably, S3 includes:

[0036] The geometrically enhanced visual features are input into the visual encoder to obtain a high-level visual representation; a learnable query vector is introduced, and key geometric information related to navigation is filtered from the high-level visual representation through cross-attention; shape-aware instructions are encoded and fused with the query features to generate multimodal features; the multimodal features are mapped to the language model input, and the frozen large language model is used for reasoning and decision-making.

[0037] Preferably, S4 includes:

[0038] During navigation, an incremental topology graph is dynamically constructed, with nodes representing viewpoints and edges representing connectivity.

[0039] The system integrates geometrically enhanced visual features, directional information, and temporal information to generate node feature embeddings; it models spatial relationships between nodes through a graph-aware self-attention mechanism; it selects target nodes and performs navigation based on global graph reasoning; and it imposes constraints on visited nodes to avoid repeated exploration.

[0040] As a preferred embodiment, S5 includes: supervising the training of the current navigation strategy using a behavior cloning method; generating pseudo-labels by combining a data aggregation strategy to improve the robustness of the strategy; constructing a joint loss function including behavior cloning loss and data aggregation loss to optimize the navigation strategy; freezing the parameters of the pre-trained visual-language model and language model, and optimizing only the geometric block extraction, optimal mask transmission, geometric-language alignment module and graph policy network; and using a phased training strategy to gradually optimize the model.

[0041] Secondly, a visual language navigation system based on three-dimensional geometric perception and shape-level representation includes:

[0042] The data construction module is used to acquire multi-view RGB images and camera pose information, generate a dot map that is aligned pixel by pixel with the image, and construct a floor-to-floor multi-layer navigation dataset containing shape-aware instructions.

[0043] The geometry perception and alignment module is used to perform cross-modal geometry perception feature alignment between RGB images and point maps through geometry block extraction and optimal mask transmission, generating geometrically enhanced visual features;

[0044] The geometry-language alignment module is used to fuse geometrically enhanced visual features with shape-aware instructions through a cross-attention mechanism to generate multimodal features;

[0045] The topology graph modeling and action selection module is used to dynamically build and maintain an incremental topology graph during navigation, and to perform global reasoning and select the optimal navigation action through a graph attention mechanism.

[0046] The strategy training and optimization module is used to jointly train the navigation model based on behavior cloning and data aggregation strategies, construct a joint loss function of supervision and pseudo-labels, and optimize the parameters of each functional module under the condition of freezing the parameters of the pre-trained model.

[0047] The navigation execution module is used to control the agent to perform navigation operations step by step in the environment based on the action selection results, and to update the topology map and observation information in real time;

[0048] The aforementioned visual language navigation system based on three-dimensional geometric perception and shape-level representation is used to implement the visual language navigation method and its steps based on three-dimensional geometric perception and shape-level representation as described in the first aspect.

[0049] Compared with the prior art, the beneficial effects of the present invention are reflected in:

[0050] 1. Compared to existing visual language navigation methods, this method has a significant advantage in 3D geometric perception. Traditional methods often rely on 2D image features or discrete semantic tokens for modeling, making it difficult to accurately depict spatial structural relationships in complex scenes, especially at key decision-making locations such as stairs and door frames, which can easily lead to ambiguity. This method introduces point map representation to achieve accurate mapping from pixels to 3D space, and combines geometric block extraction and optimal mask transmission mechanisms to achieve high-precision alignment between image and geometric information at the feature level. This significantly improves the model's ability to understand spatial structure, shape attributes, and hierarchical relationships, making navigation decisions more reliable.

[0051] 2. Secondly, this method significantly improves cross-modal semantic alignment and reasoning capabilities. Existing methods typically only perform visual and language fusion at a shallow level, making it difficult to fully utilize geometric information to guide language understanding. This method constructs a geometry-language alignment module and introduces a cross-attention mechanism to deeply fuse shape-aware geometric features with natural language instructions. This fusion is then input into a large-scale language model for reasoning. While maintaining the capabilities of the pre-trained model, it effectively injects 3D geometric priors, thereby enhancing the model's ability to parse complex instructions and improving semantic consistency and decision accuracy in multi-layered scenarios.

[0052] 3. Finally, this method demonstrates superior performance in long-term planning and navigation robustness. Traditional navigation methods often rely on local observations for incremental decision-making, lacking the ability to model the global spatial structure and prone to path redundancy or getting trapped in local optima. This method introduces an incremental topological graph memory mechanism to model historical observations and unexplored areas in a unified manner, and combines this with a graph attention mechanism for global reasoning, achieving continuous modeling of the environment and path planning. Simultaneously, a training strategy combining behavior cloning and data aggregation effectively mitigates the distribution offset problem, improving the model's generalization ability and decision stability in unknown environments. Attached Figure Description

[0053] Figure 1 This is a schematic diagram of the method flow of Embodiment 1 of the present invention. Detailed Implementation

[0054] To make the technical means, inventive features, objectives, and effects of the invention readily understandable, the invention is further described below with reference to specific illustrations. However, the invention is not limited to the embodiments described below.

[0055] It should be noted that the structures, proportions, sizes, etc., illustrated in the accompanying drawings of this specification are only used to complement the content disclosed in the specification for those skilled in the art to understand and read, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0056] This invention provides a shape-instruction-guided 3D perceptive visual-language navigation method and system. This method innovatively constructs an end-to-end geometry-aware navigation framework. By introducing explicit 3D scene geometric priors (point graphs) and designing a cross-modal feature alignment module of "geometric block extraction - optimal mask transmission," it solves the problem of limited navigation performance in complex structured environments (such as scenes containing multi-level staircases and porches) caused by the lack of 3D geometric understanding in existing visual-language navigation methods. The framework also combines a learnable query transformer and a graph action selector, achieving efficient understanding of geometric description instructions and robust long-term action planning. This invention not only significantly improves performance on standard visual navigation datasets, but its core mechanism—utilizing generative 3D geometry to assist 2D visual-language models for spatial reasoning—provides a new technical path for solving a wider range of embodied artificial intelligence tasks (such as object manipulation and scene question answering).

[0057] Example 1:

[0058] This embodiment uses target-driven 3D geometry-aware navigation as a specific application scenario to demonstrate the typical application of this invention in the field of embodied artificial intelligence. The environment used is a typical multi-layered, complex, structured indoor scene, such as an office building or residence containing stairs, corridors, and multiple rooms; the input is a continuous video stream and natural language navigation commands from the first-person perspective of the navigation agent. The challenges of this scenario are as follows: the pose of the monocular RGB camera on the agent changes continuously and irregularly during movement; the ambient lighting is uneven, with reflections, shadows, and dynamic lighting changes (such as switching lights on and off); the scene geometry is complex, containing a large number of visual occlusions, height changes (stairs, thresholds), and areas with similar geometric shapes (such as multiple rectangular door frames, L-shaped corridors); the navigation commands are shape-enhanced geometric descriptive language, such as "Please go to the living room with the arched doorway on the second floor," rather than conventional object- or location-based commands (such as "Go to the kitchen on the left"). This scenario does not provide depth maps, point clouds, scene mesh models, or precise robot localization (SLAM) information, but relies solely on monocular visual flow and language commands. It is a complex decision-making task with high degree of underdeterminacy, strong geometric dependence, and long field of view, and is a typical challenging scenario for verifying the effectiveness of the method of this invention.

[0059] like Figure 1 The visual language navigation method shown includes the following steps: (The method is based on 3D geometric perception and shape-level representation.)

[0060] S1. Construct a Floor-to-Floor multi-layer navigation dataset containing 3D geometric information and shape-aware instructions;

[0061] This step forms the data foundation of the method of this invention, aiming to provide training samples rich in 3D structural information and spatial language descriptions for subsequent training of the geometric perception model. Specifically, based on existing indoor environment datasets (such as the Room-to-Room dataset, whose underlying layer is a real indoor environment scanned by Matterport3D), this invention extends and constructs a Floor-to-Floor (F2F) dataset specifically for multi-level navigation tasks. The construction of this dataset includes two core sub-tasks:

[0062] This step obtains pixel-level 3D coordinate mapping through multi-view geometric reconstruction and uses a large language model to generate shape-aware instructions containing geometric attributes, solving the problem of traditional VLN (Visual Language Navigation) data lacking explicit 3D structural information.

[0063] S1-1, Viewpoint-level multimodal data acquisition and point map reconstruction

[0064] Acquire multi-view RGB image sequences in the Matterport3D simulation environment Record the corresponding camera pose parameters ,in Let be a rotation matrix. It is a translation vector.

[0065] A VGGT network based on the Transformer architecture is used for efficient single-stage inference. This network can directly reconstruct high-fidelity 3D scenes from multi-view RGB images and predict point maps that are pixel-wise aligned with the input multi-view RGB images.

[0066]

[0067] in Represents image pixels The corresponding 3D spatial coordinates. Compared to backprojection based on depth maps, VGGT globally optimizes multi-view geometric consistency through the Transformer architecture, generating a more robust 3D representation.

[0068] Simultaneously generate a confidence map. This is used to quantify the reconstruction reliability of each 3D point. Low-confidence areas typically correspond to occlusion, reflection, or areas lacking texture (such as under a stair railing or a reflective area in a glass door).

[0069] Build Time Viewpoint-level 3D observation data:

[0070]

[0071] in The number of candidate directions. This is the relative orientation encoding compared to the agent's current orientation. In an office building scenario, when the agent is at the top of the stairs, Each discrete orientation covers a 360-degree field of view, including key decision-making perspectives such as the downward staircase entrance and the direction of the corridor extension.

[0072] S1-2 Automatic Generation and Verification of Shape-Aware Commands

[0073] Based on multi-view visual context, a visual-language large model (such as Qwen2.5-VL) is used to analyze key view images (such as front, back, left, and right) of each scene. Structured prompts guide the model to extract geometric attributes and generate refined instructions containing shape descriptors (such as 'rectangle', 'L-shape', and 'arch').

[0074] Original instructions Extended to shape-aware instructions ,in Includes explicit geometric description:

[0075] Original instructions: "Start from the lobby, go down the stairs, through the doorway, and wait at the meeting room door."

[0076] Shape-aware instruction: "Start from the lobby, descend the L-shaped staircase, pass through the arched doorway, and wait at the entrance of the rectangular conference room."

[0077] To ensure the reliability of the generated instructions, this invention employs multi-model cross-validation (e.g., using large language models such as GPT-4 and Gemini-1.5 for validation) and performs manual sampling checks on the results to guarantee the overall quality of visual-language alignment.

[0078] S2. Achieve cross-modal geometric perception feature alignment through geometric block extraction and optimal mask transmission;

[0079] This method precisely aligns the appearance information of the RGB image with the geometric structure information of the point map at the feature level to generate geometrically enhanced visual features. The process is as follows: This step includes two core sub-modules: GeometryPatch Extraction (GPE) and Masked Optimal Transport (MOT). Addressing the domain offset problem between 2D pre-trained VLMs and 3D navigation requirements, this step proposes a cross-modal alignment strategy combining GPE and MOT. In staircase scenarios, this method effectively distinguishes the geometric boundaries and texture-similar regions at "L-shaped" corners, preventing the agent from misjudging the direction of travel at stair landings.

[0080] S21, Geometric Blocks and Visual Feature Extraction

[0081] Point map and RGB images Divided into non-overlapping Local blocks (in practice) (aligned with the subsequent ViT patch size) to obtain a set of geometry blocks. and image patch set ,in .

[0082] Extracting geometric features using a lightweight MLP encoder:

[0083]

[0084] For RGB images, a ConvNeXt-based lightweight network is used. (Kernel size $7 times 7, aligned with the geometric block size) Extracting visual features:

[0085]

[0086] In a staircase scene, this alignment ensures that the 3D geometric blocks at the "L-shaped" corners have the same spatial index as the corresponding 2D image blocks (such as the edges of the steps and the outlines of the handrails), which facilitates the subsequent establishment of accurate cross-modal correspondences.

[0087] S22, Confidence-Aware Mask Construction

[0088] Based on point plot confidence Construct a binary mask matrix First, the confidence plot is divided into... Blocks are used to calculate the average confidence level for each block, and then a threshold is applied. (Typically set to 0.7) Generate mask:

[0089]

[0090] mask Used to filter geometric regions with low confidence. In a downhill staircase scene, the distant area below the stairs has low confidence due to viewpoint occlusion. The masking mechanism can effectively suppress the interference of these unreliable areas on the alignment process.

[0091] S23. Solving for optimal transmission with mask constraints

[0092] Construct a cross-modal cost matrix and adopt... Distance measures the difference between geometric and visual features:

[0093]

[0094] Establish the optimal transmission problem with mask constraints:

[0095]

[0096] The constraints are:

[0097]

[0098] The Sinkhorn iterative algorithm is used to solve the regularized over time (OT) problem to obtain the optimal transfer matrix. This process automatically establishes a soft correspondence between 3D step geometry and 2D visual texture in a staircase scene, without the need for explicit 3D pre-training.

[0099] S24, Geometrically Enhanced Visual Feature Fusion

[0100] Based on the transmission matrix Perform feature fusion:

[0101]

[0102] via decoder Mapping aligned geometric features back to the visual feature space:

[0103]

[0104] This fusion feature retains the semantic understanding capabilities of the pre-trained VLM while explicitly injecting 3D geometric information. In the arched portico scene, the fusion feature can simultaneously encode the semantic concept of "portico" and the geometric curvature of "arch", enabling the agent to accurately identify key landmarks in the instructions.

[0105] S3. Through a geometry-language alignment module based on a cross-attention mechanism, geometrically enhanced visual features are fused with shape-aware instructions;

[0106] In order to perform high-level alignment between the geometrically enhanced visual features rich in 3D geometric information obtained in step S2 and the semantics of navigation instructions, and input them into the downstream large language model for inference, this invention adopts a geometry-language alignment module based on a cross-attention mechanism (whose design concept is similar to Q-Former).

[0107] This step establishes a bridge between low-level geometric features and high-level semantic instructions through the Q-Former architecture, addressing the issue of insufficient granularity in visual-language modal alignment in traditional methods. For landmarks in office building scenarios that require precise geometric understanding, such as "L-shaped staircases" and "arched doorways," this module utilizes learnable query vectors to extract key spatial information related to instructions from geometrically enhanced visual features, achieving fine-grained cross-modal alignment.

[0108] S31. High-level semantic encoding of visual features

[0109] Geometrically enhanced visual features generated in stage S2 Input a pre-trained VisionTransformer (ViT), and capture global contextual relationships and long-range spatial dependencies through a multi-layer self-attention mechanism:

[0110]

[0111] in This is the output dimension of ViT. This process aggregates local geometric block features into a high-level visual representation that includes the overall layout of the scene. In a staircase scene, this encoding process can integrate scattered step geometry information to form a unified representation of the overall structure of the "L-shaped" corner, rather than isolated single-step features.

[0112] S32. Visual Feature Extraction Guided by Query (Q-Former Phase 1)

[0113] Introduce a set of learnable query vectors (where $d_q$ is the query dimension and $d_q$ is the query quantity) serves as a medium for information extraction. A cross-attention mechanism is used to filter navigation-related geometric-semantic information (key geometric information) from the high-level visual representation:

[0114]

[0115] in , , For a learnable projection matrix, For the attention head dimension. Query vector. By training the system end-to-end to focus on key navigation elements (such as porch boundaries and stair corners), specific queries in arched porch scenarios will automatically focus on the geometric features of the arched curves rather than the background wall textures.

[0116] S33. Semantic Encoding and Deep Fusion of Shape Instructions (Q-Former Phase 2)

[0117] Shape-aware instructions (e.g., "through the arched doorway") Input the frozen Large Language Model (LLM) encoder (e.g., Flan-T5) to obtain context-sensitive text features:

[0118]

[0119] in The length of the instruction token. This represents the dimension of the LLM hidden layer.

[0120] By establishing a deep interaction between query representation and linguistic semantics through a second cross-attention approach, multimodal features are generated:

[0121]

[0122] in The projection parameters for the second layer of cross-attention. This mechanism achieves fine-grained alignment between geometric features and language commands: when the command mentions "rectangular meeting room", the corresponding query will strengthen the response to geometric features such as right-angled boundaries and parallel lines, and suppress the activation of circular or irregularly shaped regions.

[0123] S34, Multimodal representation projection and LLM input adaptation

[0124] The fused multimodal features Projected into the input space of the LLM through two layers of MLP:

[0125]

[0126] in The embedding dimension of the target LLM. The projected sequence. As soft visual cues, these are concatenated with navigation instruction text and input into a frozen LLM for inference and decision-making. This design allows a pre-trained LLM to understand visual information containing 3D geometric semantics without altering its own parameters.

[0127] S4. Dynamically construct and maintain an incremental topology map during navigation to make global action decisions;

[0128] To model long-term spatial experience and achieve efficient global path planning, this invention dynamically constructs and maintains a topology graph during navigation. ,in Represents visited and unexplored navigation nodes. This represents the walkable connections between nodes.

[0129] To address the needs of accumulating historical experience and global planning in long-range navigation, this step constructs an incremental topology graph memory structure. In a multi-level office building environment, the graph structure explicitly encodes the spatial relationships of explored areas (such as the connection between the lobby and the stairwell, and the topological relationship between the staircase and the porch). Through the graph-aware self-attention mechanism (GASA), it integrates geometric, directional, and historical information to achieve an improvement from local observation to global path planning.

[0130] S41, Dynamic Topology Graph Memory Construction

[0131] Dynamically maintain the topology map during navigation. :

[0132] 1. Node set Each node For each visited or observed viewpoint, store the geometrically enhanced visual features of that location. 3D coordinates And access timestamp.

[0133] 2. Edge set Represents the physical reachability between two viewpoints, weight The distance is Euclidean, and the relative orientation difference between the two nodes is also recorded.

[0134] When an agent moves from the lobby to the stairwell in an office building, the system adds new viewpoint nodes in real time and establishes edge connections with the previous node based on odometer information, forming an incremental map.

[0135] S42, Multi-dimensional Node Feature Embedding

[0136] Each node The feature representation integrates three types of complementary information:

[0137]

[0138] in:

[0139] 1. Geometrically enhanced visual features : A visual-geometric fusion representation from S3, encoding 3D structural information of the location (such as the number of stair steps, the width of the porch).

[0140] 2. Orientation Embedding Based on the azimuth angle of the node relative to the agent's current orientation. Calculated sinusoidal position code:

[0141]

[0142] In a downhill staircase scenario, this embedding helps distinguish between nodes "facing down the stairs" and "facing the staircase platform".

[0143] 3. Step Embedding Based on node access order The sinusoidal encoding imparts temporal information, enabling the model to distinguish between visits to the same visual location (such as symmetrical doorways in a corridor).

[0144] S43. Graph-Aware Self-Attention Mechanism (GASA)

[0145] The node representation is updated using the GASA mechanism, explicitly modeling the topology and spatial proximity:

[0146]

[0147] Attention weight The calculation is as follows:

[0148]

[0149] This is a geometric bias term based on node Euclidean distance, which enhances the attention weight of physically neighboring nodes. In multi-story office building scenarios, this mechanism ensures that information flow between nodes on the same floor (closely related) is stronger than that between nodes across floors, avoiding confusion between upper and lower floors at stairwell decision points.

[0150] S44, Global Action Selection and Path Planning

[0151] Integrating graph node features with multimodal features (From S3) Fusion, predicting the action value of each candidate node:

[0152]

[0153] in Global average pooling for instruction characteristics. This indicates feature splicing.

[0154] Select the node with the highest score as the temporary target:

[0155]

[0156] in This is the current set of candidate nodes (including currently visible unvisited adjacent nodes and key nodes in the explored graph). The agent moves along the topology graph. The shortest path (Dijkstra's algorithm) is used to move towards the target node.

[0157] Explore and utilize the equilibrium mechanism:

[0158] 1. Visited node mask: Apply attention masks to nodes that have been fully explored to avoid loops.

[0159] 2. Stop Node Judgment: When the agent is located at the entrance of the target room (rectangular conference room) and the instruction completion rate meets the threshold, the stop action is activated.

[0160] S5. The navigation strategy is jointly optimized and trained through behavior cloning and data aggregation strategies.

[0161] A hybrid supervision strategy combining Behavior Cloning and Data Aggregation (DAgger) is employed to address the compound error problem in imitation learning. In the training of the office building navigation task, only the newly added geometry alignment modules (Geometry Block Extraction (GPE), Mask Optimal Transmission (MOT), and Geometry-Language Alignment Module (Q-Former)) and the graph policy network are optimized, while the pre-trained VLM and LLM parameters are kept frozen, achieving efficient multi-stage training.

[0162] S51, Behavioral Cloning Supervised Learning

[0163] Based on expert demonstration trajectory Supervised learning is performed to minimize the negative log-likelihood between the predicted action distribution and the expert actions:

[0164]

[0165] in For the current policy network, The loss is based on historical observation sequences. In the expert trajectory of the staircase scene, this loss ensures that the model learns the correct turning decision at L-shaped corners (such as "turn left down the stairs" instead of "go straight into the corridor").

[0166] S52, Data Aggregation and Online Iteration (DAgger)

[0167] To mitigate the training-test distribution bias (the agent enters intermediate states not seen during training while autonomously exploring the office building), the DAgger algorithm is used to iteratively collect data.

[0168] 1. Policy rollout: Use the current policy. Execute within the environment to generate interaction trajectories. ,in Predict actions for the model.

[0169] 2. Pseudo-label generation: In the partially constructed graph Calculate the shortest path from the current node to the target (meeting room), and use the next node on the path as a pseudo-label action. .

[0170] 3. DAgger loss calculation:

[0171] This mechanism enables the model to recover from erroneous states during autonomous exploration, such as learning how to return to the stairwell and re-determine if it mistakenly goes to the second floor instead of the first.

[0172] S53, Joint Optimization Strategy

[0173] The total loss function is a weighted combination of behavioral cloning and DAgger loss:

[0174]

[0175] in A balance coefficient (set to 1.0 in practice) is used to ensure a balance between expert supervision and autonomous exploratory learning. An additional confidence regularization term is introduced to constrain the MOT module:

[0176]

[0177] Encourage transmission matrix In low confidence regions (mask) Assign lower weights.

[0178] S54. Phased Training and Parameter Freezing Strategy

[0179] Two-stage training is used to achieve efficient parameter fine-tuning:

[0180] Phase 1: Geometric Alignment Pre-training (200K steps)

[0181] Frozen parameters: Pre-trained ViT, LLM encoder / decoder

[0182] Trainable parameters: GPE module ( , ), MOT transfer solver, Q-Former projection matrix ( wait)

[0183] Optimizer settings: AdamW optimizer Weight decay of 0.05

[0184] Learning rate scheduling: linear warmup from to (First 1K steps), subsequent cosine decays to 0.

[0185] Batch size: 8, trained using a mix of R2R and PREVALENT synthetic data.

[0186] Phase Two: Strategy Fine-tuning (50K steps)

[0187] Frozen parameters: ViT, LLM, Q-Former (pre-trained)

[0188] Trainable parameters: Graph action selector (GASA layer, MLP scoring network)

[0189] Learning rate: fixed

[0190] Batch size: 2, using a mix of online DAgger data and expert data (ratio 1:1).

[0191] Data efficiency optimization:

[0192] Experiments show that in multi-story office building scenarios, using only 50% of the F2F dataset (approximately 10,000 trajectories), this training strategy can achieve the performance of the full-data training baseline (SR 74% vs 72%), verifying the effect of geometric prior injection on improving data efficiency.

[0193] Example 2:

[0194] A visual language navigation method and system based on three-dimensional geometric perception and shape-level representation, comprising:

[0195] 1. Data Construction Module

[0196] This module is responsible for providing high-quality training data rich in 3D geometric priors for subsequent model training and validation during the offline phase before system deployment. Its innovation and key feature lies in the fact that it is not merely a simple data collector, but an automated, geometry-aware augmentation-oriented data generation and annotation pipeline. Its workflow begins by utilizing existing large-scale real-world indoor 3D scanning datasets (such as Matterport3D). By invoking an efficient, Transformer-based multi-view geometry reconstruction network (such as VGGT), it automatically generates a point map precisely aligned pixel-by-pixel with each frame of the acquired RGB image. This means that each pixel in the image not only contains color information but is also assigned its coordinates in the 3D world, providing a dense and accurate geometric foundation for subsequent processing. Simultaneously, this process outputs a confidence map, quantifying the reliability of each 3D point reconstruction, which is crucial for subsequent noise filtering. Building upon this, this module further introduces the semantic generation capabilities of a large-scale vision-language model, automatically "understanding" scene images and generating "shape-aware instructions" that include shape attributes and geometrically augment the original navigation commands. For example, "go through that door" can be enhanced to "go through that arched doorway." To ensure the accuracy and consistency of command generation, this module also integrates multi-model cross-validation and manual sampling checks. Ultimately, this module outputs a structured, multimodal floor-to-floor dataset. Each data sample contains a complete information pair: "RGB image - dot plot - confidence plot - camera pose - shape-aware command," providing a solid data foundation for training a navigation model with 3D geometric understanding capabilities.

[0197] 2. Geometric Awareness and Alignment Module

[0198] This module is the core of the system's visual processing during real-time inference in online navigation. Its design aims to address the "domain gap" problem faced by existing 2D pre-trained visual models in navigation tasks. Specifically, this module is activated when the agent moves in the environment and the camera captures the current multi-view RGB image and corresponding dot plot. Its internal operation follows a sophisticated three-step process, with the core being the innovative alignment mechanism of "geometric block extraction - optimal mask transmission." First, the geometric block extraction submodule synchronously divides the high-dimensional, dense dot plot and RGB image into spatially aligned local blocks. Then, it efficiently extracts deep features representing the local 3D structure and 2D appearance using a lightweight multilayer perceptron and a lightweight convolutional neural network, respectively. Second, the optimal mask transmission submodule introduces a crucial adaptive mechanism: it automatically constructs a binary mask using the confidence information inherent in the dot plot to "cover" areas where 3D reconstruction is unreliable due to occlusion, reflection, or missing texture. Under the constraint of this mask, the module solves a regularized optimal transport problem to accurately calculate the optimal soft correspondence between geometric feature blocks and image feature blocks. This is equivalent to "finding" the most likely corresponding 3D geometric structure description for each image texture at the feature level. In the third step, the feature fusion submodule, based on the optimal correspondence calculated in the previous step, "injects" the 3D geometric information into visual features extracted from RGB images and pre-trained from large-scale 2D images in a learnable and differentiable manner, ultimately outputting a "geometrically enhanced visual feature". At key decision points such as stairs and door frames, this module can effectively distinguish visually similar but geometrically different regions (e.g., distinguishing between a downward-extending L-shaped staircase and a forward-extending flat corridor), thus providing accurate geometric cues for subsequent decisions.

[0199] 3. Geometry-Language Fusion Module

[0200] This module acts as a "cross-modal semantic translator," its core task being to deeply fuse and align concrete spatial information from low-level visual-geometric features with abstract semantic intent from high-level natural language instructions within the same semantic space. This module employs a learnable query transformer architecture similar to Q-Former, but its focus is on achieving fine-grained visual-language alignment guided by geometric priors. The module operates in two phases: In the first phase, a set of learnable query vectors actively "question" and "extract" the key geometric semantic information most relevant to the current navigation task from the geometrically enhanced visual features generated by the previous layer through a cross-attention mechanism. For example, when multiple doorways exist in a scene, the query automatically focuses on the doorway boundary with an "arched" curve feature. In the second phase, these pre-selected query vectors, rich in geometric information, undergo a second cross-attention interaction with "shape-aware instructions" encoded by a large language model, which also contain geometric descriptions. This interactive process essentially matches and verifies geometric "shape" evidence from the vision with geometric "description" concepts from the language, thereby generating a deeply fused, highly usable multimodal feature representation for downstream decision-making. Finally, this module adapts this fused representation into an input vector sequence that can be directly "understood" by a large downstream language model through a simple projection layer, essentially preparing a "visual briefing" with detailed geometric annotations for the frozen language model.

[0201] 4. Topology Graph Modeling and Action Selection Module

[0202] This module serves as the system's "global planning and memory hub," overcoming the shortcomings of traditional navigation strategies based on single-frame decision-making, such as limited vision and susceptibility to local loops. Throughout the agent's navigation process, this module dynamically constructs and maintains an incremental, lightweight topology graph. The nodes of this graph represent the various spatial viewpoints the agent has reached or observed. Each node stores not only its visual features and 3D coordinates but also its access sequence. The edges of the graph represent the physical connectivity between these nodes. The core innovation of this module lies in its "graph-aware self-attention mechanism." When updating the representation of each node, this mechanism considers not only the node's own features but also explicitly aggregates information from its spatial neighbors (i.e., physically adjacent locations) through the graph structure, assigning higher attention weights to nodes that are physically closer. This allows the agent to "perceive" the contextual relationships of its current location within the global spatial layout when making decisions. For example, at the bottom of a staircase, a node can "know" that it is connected to steps below and a corridor ahead. The action selection submodule makes decisions based on this constantly evolving topology: it jointly analyzes the features of all current candidate nodes (including new nodes that are directly visible and key nodes that have been explored in the graph) with the instruction features from the language fusion module, and predicts the "value" of each node as the next target through a scoring network. This design realizes a leap from local reactive navigation of "seeing where you are" to human-like intelligent navigation of "having a map in mind and planning the whole picture", enabling the agent to plan efficient long-distance paths that avoid repeated exploration.

[0203] 5. Strategy Training and Optimization Module

[0204] This module serves as the system's "offline learning engine," responsible for optimizing the performance of all learnable modules (such as geometric alignment networks, query vectors, and graph attention network parameters) in a data-driven manner, while keeping the parameters of the pre-trained visual and language foundational models from massive amounts of internet data frozen. This module employs an imitation learning paradigm, the core of which is a "hybrid supervised strategy combining behavior cloning and data aggregation." In the early stages of training, the system learns its basic policy by imitating expert demonstrations (i.e., correct navigation trajectories provided by humans or optimal planners), minimizing the difference between predicted actions and expert actions. However, simple imitation learning can lead to rapid failure due to "composite errors" when the agent deviates from states seen by the expert trajectory during autonomous exploration. Therefore, this module introduces a data aggregation algorithm: allowing the current policy to explore the environment independently, when it goes astray, the system (or a planner) generates a corrective "pseudo-label" action for this new state (i.e., "how to get back on track"), and these new "state-correction action" pairs are added to the training dataset for iterative optimization. This "learning from mistakes" mechanism greatly enhances the robustness of the strategy and its generalization ability in unknown environments. The entire training process is parameter-efficient, requiring only optimization of a small number of new parameters, yet enabling the rapid injection of 3D geometric perception capabilities.

[0205] 6. Navigation Execution Module

[0206] This module serves as the "execution and feedback interface" for the system's interaction with the physical world or simulation environment, achieving a closed loop from abstract decision-making to concrete actions. It receives the decision output (typically a target node ID) from the action selection module and then converts this abstract decision into concrete, executable low-level control commands. For example, if the target node is 3 meters in front of the current agent, this module generates a "move forward 3 meters" linear and angular velocity command and sends it to the actuator (such as a mobile chassis) through the robot's operating system or simulator's control interface. After the agent executes actions, moves, and acquires new visual observations, this module packages these new observations (new RGB images and point maps) and sends them to the geometry perception and alignment module, initiating the next round of the perception-decision loop. Simultaneously, this module also updates the topology graph modeling module in real time, adding newly arrived viewpoints as new nodes to the graph and establishing connections between them and previous nodes, thereby continuously accumulating environmental awareness and dynamically expanding the map.

Claims

1. A visual language navigation method based on three-dimensional geometric perception and shape-level representation, characterized in that, Includes the following steps: S1. Construct a Floor-to-Floor multi-layer navigation dataset containing viewpoint-level 3D observation data and shape-aware commands; Viewpoint-level 3D observation data includes multi-view RGB images, pixel-by-pixel pixel-aligned dot plots with the multi-view RGB images, dot plot confidence, and relative orientation coding; S2. By extracting geometric blocks and transmitting masks optimally, the RGB image and the point map are aligned with cross-modal geometric perception features to generate geometrically enhanced visual features. S3. Through a geometry-language alignment module based on a cross-attention mechanism, geometrically enhanced visual features are fused with shape-aware instructions to generate multimodal features. S4. Dynamically construct and maintain an incremental topology map during navigation, and make global action decisions based on multimodal features and the incremental topology map; S5. The navigation strategy is jointly optimized and trained through behavior cloning and data aggregation strategies.

2. The visual language navigation method based on three-dimensional geometric perception and shape-level representation according to claim 1, characterized in that, S1 includes: Multi-view RGB images and camera pose information are obtained based on indoor 3D scanning data; a 3D reconstruction network is used to generate a point map that is pixel-by-pixel aligned with the multi-view RGB images, realizing the mapping from pixels to 3D coordinates; Shape-aware commands are generated based on multi-view RGB images, and geometric attributes are enhanced on the original navigation commands. Consistency verification is performed on the shape-aware commands. Viewpoint-level 3D observation data and shape-aware commands are stored in a floor-to-floor multi-layer navigation dataset.

3. The visual language navigation method based on three-dimensional geometric perception and shape-level representation according to claim 1, characterized in that, Geometric block extraction in S2 includes: The dot plot and RGB image are divided into geometric blocks and image blocks of fixed size; geometric features are extracted from the geometric blocks; and visual features aligned with the geometric blocks are extracted from the RGB image.

4. The visual language navigation method based on three-dimensional geometric perception and shape-level representation according to claim 1, characterized in that, Optimal mask transmission in S2 includes: A mask for filtering reliable regions is generated based on the point map confidence level. The optimal transmission problem with mask constraints is constructed based on geometric and visual features and solved iteratively to generate the optimal transmission matrix. The optimal transmission matrix is ​​then fused with the visual features and mapped back to the visual feature space to generate geometrically enhanced visual features.

5. A visual language navigation method based on three-dimensional geometric perception and shape-level representation according to claim 4, characterized in that, In S2, generating the optimal transfer matrix includes: Computing the relationship between geometric features and visual features Distance, generating cross-modal cost matrix ; Establish the optimal transmission problem with mask constraints: ； The constraints are: ； Where N is the number of geometric blocks; The Sinkhorn iterative algorithm is used to solve the regularized over time (OT) problem to obtain the optimal transfer matrix. .

6. The visual language navigation method based on three-dimensional geometric perception and shape-level representation according to claim 5, characterized in that, Generate geometrically enhanced visual features including: Based on the optimal transmission matrix and visual features Perform feature fusion: ； via decoder Aligned geometric features Mapping back to the visual feature space generates geometrically enhanced visual features: 。 7. A visual language navigation method based on three-dimensional geometric perception and shape-level representation according to claim 1, characterized in that, S3 includes: The geometrically enhanced visual features are input into the visual encoder to obtain a high-level visual representation; a learnable query vector is introduced, and key geometric information related to navigation is filtered from the high-level visual representation through cross-attention; shape-aware instructions are encoded and fused with the query features to generate multimodal features; the multimodal features are mapped to the language model input, and the frozen large language model is used for reasoning and decision-making.

8. A visual language navigation method based on three-dimensional geometric perception and shape-level representation according to claim 1, characterized in that, S4 include: During navigation, an incremental topology graph is dynamically constructed, with nodes representing viewpoints and edges representing connectivity. The system integrates geometrically enhanced visual features, directional information, and temporal information to generate node feature embeddings; it models spatial relationships between nodes through a graph-aware self-attention mechanism; it selects target nodes and performs navigation based on global graph reasoning; and it imposes constraints on visited nodes to avoid repeated exploration.

9. A visual language navigation method based on three-dimensional geometric perception and shape-level representation according to claim 1, characterized in that, S5 includes: supervising the training of the current navigation strategy using the behavior cloning method; generating pseudo-labels by combining data aggregation strategy to improve the robustness of the strategy; constructing a joint loss function including behavior cloning loss and data aggregation loss to optimize the navigation strategy; freezing the parameters of the pre-trained visual-language model and language model, and optimizing only the geometric block extraction, mask optimal transmission, geometric-language alignment module and graph policy network; and using a phased training strategy to gradually optimize the model.

10. A visual language navigation system based on three-dimensional geometric perception and shape-level representation, characterized in that, include: The data construction module is used to acquire multi-view RGB images and camera pose information, generate a dot map that is aligned pixel by pixel with the image, and construct a floor-to-floor multi-layer navigation dataset containing shape-aware instructions. The geometry perception and alignment module is used to perform cross-modal geometry perception feature alignment between RGB images and point maps through geometry block extraction and optimal mask transmission, generating geometrically enhanced visual features; The geometry-language alignment module is used to fuse geometrically enhanced visual features with shape-aware instructions through a cross-attention mechanism to generate multimodal features; The topology graph modeling and action selection module is used to dynamically build and maintain an incremental topology graph during navigation, and to perform global reasoning and select the optimal navigation action through a graph attention mechanism. The strategy training and optimization module is used to jointly train the navigation model based on behavior cloning and data aggregation strategies, construct a joint loss function of supervision and pseudo-labels, and optimize the parameters of each functional module under the condition of freezing the parameters of the pre-trained model. The navigation execution module is used to control the agent to perform navigation operations step by step in the environment based on the action selection results, and to update the topology map and observation information in real time; The aforementioned visual language navigation system based on three-dimensional geometric perception and shape-level representation is used to implement the visual language navigation method and its steps based on three-dimensional geometric perception and shape-level representation as described in the first aspect.

Images