Large model navigation method guided by historical topological graph based on manifold perception

Abstract

The invention discloses a manifold perception-based large model navigation method guided by a historical topological graph, and relates to a computer vision technology. The method aims at solving the challenges that in the navigation process, long-distance reasoning experience is insufficient, instruction fragments and dynamic visual observation are difficult to align, and large model reasoning is prone to illusion interference. Firstly, a large model based on an encoder-decoder structure is used for supplementing historical information coding for a visual observation sequence, and therefore global topological information guidance is provided for long-distance reasoning. And secondly, in order to effectively solve the problem that large model reasoning is subjected to illusion interference, significant space-time differences in a visual observation sequence are mined by using a multi-curvature manifold, so that the large model can accurately describe the current environment and make a decision according to a visual reference object in thinking. Besides, in order to strengthen the perception capability of the large model to the space structure and establish a graph self-attention mechanism, the node distance embedded in the constructed historical topological graph is combined with the visual similarity so as to model the space relationship between the nodes.

Description

Technical Field

[0001] This invention relates to computer vision technology, specifically to a large model navigation method guided by a historical topology map based on manifold perception. Background Technology

[0002] Current research has made progress in using large language models for visual-language navigation. Large language models can describe the movement process of an agent in a navigation task and its progress in executing target instructions. To endow language models with visual capabilities, existing methods typically convert the agent's stepwise observations of the environment into text descriptions using image captioning models, and summarize its past observations and decisions into plain text to track its own experience and provide a basis for subsequent planning. Finally, the large language model integrates the above information and infers the actions that guide the agent to the target.

[0003] While this workflow design fully leverages the massive knowledge that large models can generalize from, reducing the over-reliance on scarce dedicated training data for visual-language navigation, it still faces numerous challenges. For example, the existing workflow relies on complex and fragile prompt engineering, requiring significant computational resources for progressive prompting, and the captions and summaries may contain noise that interferes with the model, potentially leading to information loss. Furthermore, the ability of large language models to accurately understand the spatial structure of the environment and the causal relationships of current movement decisions remains questionable. For instance, in the Room-to-Room (R2R) task, there is a significant success rate gap between zero-shot navigation methods based on large language models and dedicated visual-language navigation models. Therefore, directly utilizing large models that rely solely on text representations is insufficient for achieving reliable navigation.

[0004] On the other hand, some methods attempt to fine-tune large models for visual-language navigation tasks. Their workflow typically involves encoding visual observations as features or text inputs to the large model and converting navigation actions into structured text to suit the large model's autoregressive prediction training. However, despite attempting to leverage the powerful pre-training capabilities of large models, these methods still significantly underperform dedicated visual-language navigation models. This may stem from insufficient training data to fully unleash the potential of the large model, or from a discrepancy between the pre-training objectives of the large model and the core objectives of the visual-language navigation task—for example, how to align multi-view images with target command fragments during dynamic navigation. Furthermore, directly fine-tuning large models may even weaken their original general language understanding capabilities. This loss of capability is one of the key issues limiting the application of large models to embodied intelligence, potentially leading to uncontrollable black-box behavior. Summary of the Invention

[0005] The purpose of this invention is to address the problems of insufficient long-distance reasoning experience, difficulty in aligning instruction fragments with dynamic visual observations, and susceptibility of large-model reasoning to hallucination interference in existing large-model-based visual language navigation methods. It provides a robust and efficient large-model navigation method guided by historical topology maps based on manifold perception, which can accurately capture the spatiotemporal features and multimodal associations of the environment and effectively improve navigation accuracy and generalization.

[0006] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0007] A large-model navigation method based on manifold-aware historical topology graph guidance includes the following steps:

[0008] 1) Prepare a visual language navigation dataset, which includes panoramic views, navigation instructions, and corresponding navigation trajectories;

[0009] 2) Construct an overall navigation framework based on a visual language model. The overall navigation framework consists of a large visual language model, a large model based on an encoder-decoder, and a navigation strategy network. A dual-branch network design is used for feature encoding to realize feature encoding of multimodal visual observation and command information.

[0010] 3) Design a visual reference reasoning method based on the spatiotemporal differences of multi-curvature manifolds, and generate high-quality navigation instructions by combining multi-curvature manifold embedding and spatiotemporal difference filtering;

[0011] 4) Construct a navigation strategy network based on the historical representation of topology to provide long-term navigation memory and backtracking capabilities;

[0012] 5) Employ a multi-stage training method that combines imitation learning and data aggregation to optimize the model end-to-end;

[0013] 6) Utilize the trained navigation model to perform navigation reasoning in the indoor environment and generate executable action commands to guide the agent to the target.

[0014] In step 1), the specific method for preparing the visual language navigation dataset is as follows: A Room-to-Room (R2R) dataset is used, collected from the Matterport3D simulator, containing 90 real-world building-scale indoor scenes. Specifically, this dataset contains 10,800 panoramic views, 21,567 navigation commands, and 7,189 corresponding navigation trajectories, where each navigation trajectory can correspond to 3 navigation commands. The R2R dataset is divided into four parts: a training set, a validation set in the seen environment, a validation set in the unseen environment, and a test set. When evaluating the performance of the navigation method of this invention, the navigation metrics used include: trajectory length (TL), used to measure the average path length; navigation error (NE), calculated as the average distance between the final position and the target position; success rate (SR), calculated as the percentage of paths with a navigation error of less than 3 meters; success rate under the ideal stopping strategy (OSR); and path length-weighted success rate (SPL), which comprehensively considers success rate and efficiency.

[0015] In step 2), the specific method for constructing the overall navigation framework based on the visual language model and realizing feature encoding of multimodal visual observation and command information is as follows:

[0016] Construct an overall navigation framework consisting of a large visual language model, a large encoder-decoder-based model, and a navigation policy network;

[0017] Feature encoding is performed using a dual-branch network design:

[0018] The first branch: uses the Q-former network in InstructBLIP to extract image tags from visual observations and instructions, and uses the image tags as visual input to the large language model;

[0019] The second branch: By matching object detection and target segmentation results with the intersection-union ratio (IoU≥0.5), the agent's perception of the distance to surrounding objects is enhanced;

[0020] The visual observation information obtained from the two branches is input into the encoder of the large model as a feature vector. At the same time, the direction information is injected into the navigation prompts in a structured input format, and special markers are introduced to insert image markers and instructions into the prompts.

[0021] Navigation inference data is generated based on the training set, and the Q-former and projection layers on the prediction labels are adjusted using the autoregressive training objective to construct a complete visual language model.

[0022] In step 3), the design is based on a visual reference reasoning method that considers the spatiotemporal differences of multi-curvature manifolds. The specific method for generating high-quality navigation instructions is as follows:

[0023] 3.1) Construct a multi-curvature manifold space, including Euclidean space mapping, hyperbolic space mapping, and spherical space mapping;

[0024] Euclidean space mapping: For the feature vector X of a multi-view visual observation sequence, through an encoder... Directly extract features to obtain Euclidean space features. :

[0025]

[0026] Hyperbolic space mapping: To map features to a hyperbolic space with constant negative curvature, an exponential mapping is employed; specifically, given an encoder... The extracted features are first calculated by their modulus, and then mapped using the hyperbolic tangent function and an appropriate scaling factor to obtain the hyperbolic space features. :

[0027]

[0028] in, It is the hyperbolic tangent function. This is a scaling factor based on the curvature parameter; Let be the curvature parameter of hyperbolic space, derived from the curvature of hyperbolic space. The derivation is usually set to -1; Represents the norm; For feature encoder Feature extraction results for input X;

[0029] Spherical Space Mapping: To map features to a spherical space with constant positive curvature, ensuring that the feature vectors lie on a unit sphere, a normalized encoder is used. Extracted features, and processed by a spherical projection function. Processing yields spherical spatial features.

[0030]

[0031] in, For feature encoder Feature extraction results for input X Represents the norm; For spherical projection functions; spherical projection functions The specific implementation is as shown in the formula. and As shown, by using Cartesian coordinates ( , , Convert to spherical coordinates ( accomplish:

[0032] Spherical projection function The specific implementation is as shown in the formula. and As shown, by using Cartesian coordinates ( , , Convert to spherical coordinates ( accomplish:

[0033]

[0034]

[0035] in, The distance from the origin in Cartesian coordinates; It is the inverse cosine function; It is a two-parameter arctangent function; , , These are the x, y, and z components of a three-dimensional Cartesian coordinate system, respectively.

[0036] 3.2) Filtering visual references with spatiotemporal differences:

[0037] From the instruction and path annotation pairs in the training set, we first extract the nouns in the instructions as a set of original language visual reference objects. Subsequently, the visual reference set was supplemented using the following methods:

[0038] Calculate temporal importance score: Construct a sequence including mean pooled features of the panoramic view. and through and The importance score of calculating the cosine distance between them. :

[0039]

[0040] Where τ represents the time dimension, t represents the time step, and K represents the observed view number; For the mean pooled feature of the panoramic view at time step t, the superscript... This indicates the result after mean pooling; The mean pooled feature of the panoramic view at time step t+1; Represents the norm; The features of the k-th view at time step t;

[0041] Calculate spatial importance score: at time step t, extract action perspective. All objects are used as a candidate visual reference set, and spatial importance scores are assigned based on the object's presence in other candidate views. :

[0042]

[0043] Where t represents the time step, n represents the reference object number, and a represents the spatial dimension; Let be the cosine similarity between the action viewpoint at time step t and the first candidate viewpoint. Let be the cosine similarity between the action viewpoint at time step t and the second candidate viewpoint. Let be the cosine similarity between the action viewpoint at time step t and the third candidate viewpoint;

[0044] Calculate the final spatiotemporal dissimilarity score: For a candidate visual reference object, its final spatiotemporal dissimilarity score. :

[0045]

[0046] in, This represents the spatial importance score of the nth candidate visual reference at time step t. The time importance score at time step t;

[0047] Obtaining the comprehensive spatiotemporal dissimilarity score under multi-curvature manifolds: The visual mapping features under Euclidean, hyperbolic, and spherical spaces are processed according to the above-mentioned spatiotemporal dissimilarity visual reference selection process to obtain the comprehensive spatiotemporal dissimilarity score under multi-curvature manifolds. :

[0048]

[0049] in, For the first Spatial importance score of the nth candidate visual reference at time step; For the first Time importance score for time step; For hyperbolic space Spatial importance score of the nth candidate visual reference at time step; For hyperbolic space Time importance score for time step; In spherical space, the first Spatial importance score of the nth candidate visual reference at time step; In spherical space, the first The time importance score of the time step Represents hyperbolic space. Represents a spherical space. τ represents the time step, and τ represents the time dimension;

[0050] reserve Greater than the set threshold The visual reference objects serve as a new supplementary set of visual reference objects. The complete set of landmarks for the trajectory is constructed as follows: ;in, A set of visual reference objects for the original language. Indicate linguistic features; For the newly added set of visual references, Indicates visual features; This is the set merging operator;

[0051] By combining multi-curvature manifold embedding and spatiotemporal difference filtering, large models can more accurately identify and utilize key visual references in the environment, thereby generating more precise and executable navigation instructions.

[0052] In step 4), the specific method for constructing a navigation strategy network based on the historical representation of topology to provide long-term navigation memory and backtracking capabilities is as follows:

[0053] 4.1) Construct and maintain a dynamic environment topology graph ,in This is the dynamic environment topology diagram at time step t; For encapsulating symbols of sets; Let be the set of nodes at time step t; Let be the set of edges at time step t, where t represents the time step; this topological graph contains The set of nodes at time step t. There are three types of nodes: visited nodes, navigable nodes, and the current node. The agent can access a panoramic view of visited and current nodes, while navigable nodes are unexplored and only observed from the visited portion. At each time step... Add the current node and its adjacent unvisited nodes to the topology graph and update the set of reachable edges;

[0054] 4.2) Construct a historical memory topology graph memory and node representation. The memory consists of visited nodes along the trajectory and adjacent unexplored nodes, where:

[0055] Each visited node is represented by the average pooling feature of all its candidate views;

[0056] Each unexplored node is represented by partial pooling of the corresponding views of all its adjacent visited nodes in the trajectory;

[0057] The representation of each view is determined by its visual features. The sum, the directional embedding representing the node position Embedded steps representing the traversal order and action markers Together they form;

[0058] Add a "stop" node to the history memory topology graph memory to represent a stop action, and connect it to all other nodes to encode the visited nodes through different navigation histories, thereby improving alignment with instructions;

[0059] 4.3) A multi-layered Transformer is used to model the spatial relationships between the nodes, with its input node representation... for:

[0060]

[0061] in, This indicates that self-attention encoding is performed on node features; This represents the sum of visual features of the i-th view; Embed the direction of the i-th view; Embed the steps for the i-th view; The action tag for the i-th view is given, where i represents the view number and M represents the number of views for a node.

[0062] The nodes are embedded and input into a multi-layer cross-modal transformer (Transformer) to model the relationship between instructions and nodes; this process includes:

[0063] Cross-attention is applied between node embeddings and instructions encoded by the large language model;

[0064] 4.4) Semantic information fusion through graph-aware self-attention; the graph-aware self-attention algorithm considers the distance and visual similarity between nodes during the fusion process to enhance contextual understanding. Its calculation formula is as follows:

[0065]

[0066] in, , From The obtained pairwise distance matrix, and These are two learnable parameters; For activation functions; To query the projection matrix; The key projection matrix; The projection matrix is ​​a value. This is a matrix transpose operation; Scaling factor For node feature dimensions; It is a pairwise distance matrix; These are the learnable weight parameters of the distance matrix; These are learnable bias parameters;

[0067] 4.5) A two-layer feedforward network is used to process the output node representation of graph-aware self-attention and generate action scores; the agent selects the node with the highest score as the target and moves according to the shortest path in the graph memory, and hides the scores of the visited nodes.

[0068] The specific method for end-to-end optimization of the model using a multi-stage training approach combining imitation learning and data aggregation, as described in step 5), is as follows:

[0069] 5.1) Conduct two-stage training:

[0070] Phase 1: Initialize from the InstructBLIP model, using a frozen large language model and visual encoder, fine-tuning the Q-former only on navigation inference data, and using a preset prompt format for model inference;

[0071] The second stage involves connecting the pre-trained visual language model with the downstream navigation strategy, fine-tuning only the policy network, and keeping the visual language model frozen.

[0072] 5.2) Optimize by combining imitation learning loss and data aggregation;

[0073] Imitation learning loss for:

[0074]

[0075] in, It is the natural logarithm function; For navigation strategy functions; The decision is the one listed in the true decision label at time step t; Represents conditional probability; These are the textual features of the encoded instructions; This is the dynamic topology graph at time step t; This represents the total time step of the navigation trajectory;

[0076] Data aggregation loss for:

[0077]

[0078] in, For navigation strategy functions; It refers to the textual characteristics of the instructions after they have been encoded. This represents a partial graph generated by the agent through policy upsampling. The pseudo-decision label determined by the shortest path to the destination;

[0079] Total training loss function Loss due to imitation learning and data aggregation loss Common components:

[0080]

[0081] in, It is a balance factor used to balance the two loss terms;

[0082] Through the above two-stage training and joint loss function optimization, the model can effectively learn and improve its navigation ability and robustness in complex indoor environments.

[0083] In step 6), the process of using a trained navigation model to perform navigation reasoning in the indoor environment and generating executable action commands to guide the agent to the target specifically involves:

[0084] By using a navigation model trained through multiple stages, and by integrating observations of the current environment, past navigation experience, and decision summaries, the system infers action instructions that can guide the agent to accurately reach the target and generates an executable action sequence.

[0085] Compared with the prior art, the present invention has the following advantages:

[0086] 1. This invention proposes a large model navigation method based on manifold perception and historical topology graph guidance. It provides a more robust and efficient solution to address various challenges faced by existing large models in visual language navigation tasks, such as dependence on fragile cue engineering, high computational resource consumption, information loss due to noise interference, and insufficient understanding of spatial structure and causal relationships of actions.

[0087] 2. This invention introduces the spatiotemporal differences of multi-curvature manifolds to regulate the instructions generated by large models during inference, which enables the model to generate reliable and executable inference instructions, effectively reducing navigation failures caused by illusions that are prone to occur in existing large models, and significantly improving the reliability and accuracy of navigation.

[0088] 3. The multi-curvature manifold spatial feature mapping method proposed in this invention can effectively represent different curvature spaces such as Euclidean, hyperbolic, and spherical spaces based on the distribution of objects of different shapes in the environment and their influence on visual observation changes, thereby enhancing the model's ability to understand the spatial structure of complex environments.

[0089] 4. The historical memory network navigation strategy based on topological graph representation constructed in this invention enables intelligent agents to effectively track long-term navigation history and achieve backtracking, providing necessary fault tolerance for long-distance navigation, and providing inspiration for current planning by observing historical environment and aligning action fragments.

[0090] 5. By combining imitation learning and data aggregation in a multi-stage learning method, this invention not only ensures the optimized performance of the model on specific navigation tasks, but also successfully maintains the original general language understanding ability of the large language model. This avoids the weakening of general ability and uncontrollable black box behavior that may be caused by traditional fine-tuning, thereby improving the generalization and practicality of the model as a whole. Attached Figure Description

[0091] Figure 1 This is a schematic diagram of the overall framework of the present invention.

[0092] Figure 2 This is a schematic diagram of image feature processing in the visual observation sequence of the present invention. (a) shows the target detection result; (b) shows the instance segmentation result; and (c) shows the fusion depth result.

[0093] Figure 3 This is a schematic diagram of the visual reference reasoning method based on the spatiotemporal differences of multi-curvature manifolds in this invention.

[0094] Figure 4 This is a schematic diagram of the navigation strategy network module based on topological history representation in this invention.

[0095] Figure 5 This is one of the visualization processes for large model inference (steps 0-2).

[0096] Figure 6 The second step in visualizing the reasoning process for large models (steps 3-6). Detailed Implementation

[0097] To make the above-mentioned objectives, features and advantages of the present invention more apparent and understandable, the method of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. These embodiments are implemented based on the technical solution of the present invention, and provide implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0098] This invention, based on differential geometry and deep learning theory, constructs a large-model navigation method guided by a historical topology map based on manifold perception. Building upon the InstructBLIP encoder-decoder visual language model architecture, it utilizes multi-image perception to better adapt to visual language navigation tasks. Specifically, this invention employs a thought chain reasoning process based on spatiotemporal differences to standardize step-by-step navigation reasoning, enabling the visual language model to generate reliable and executable reasoning instructions, minimizing navigation failures caused by large-model illusions. To enhance the model's ability to distinguish changes in overall visual observation caused by the distribution of objects of different shapes in the environment, this invention introduces a multi-curvature manifold space feature mapping method. By continuously deforming the average visual feature vector of objects, and leveraging the geometric applicability of different curvature spaces to hierarchical structures, it collects the geometric topological features of objects of different shapes at different levels under multi-curvature conditions. Furthermore, this invention combines language decoding and action decoding of the visual language model's potential composite visual language representation with the topology map, enabling the agent to track long-term navigation history and effectively backtrack, while maintaining the general language capabilities of the large-scale language model.

[0099] The embodiments of the present invention include the following steps:

[0100] 1. Prepare a visual language navigation dataset.

[0101] The dataset used in this embodiment is the Room-to-Room (R2R) dataset, collected from the Matterport3D simulator, containing 90 real-world building-scale indoor scenes. Specifically, this dataset contains 10,800 panoramic views, 21,567 navigation commands, and 7,189 corresponding navigation trajectories, with each navigation trajectory corresponding to 3 navigation commands. The R2R dataset is divided into four parts: a training set, a validation set in seen environments, a validation set in unseen environments, and a test set. When evaluating the performance of the navigation method of this invention, the navigation metrics used include: trajectory length (TL), used to measure the average path length; navigation error (NE), calculated as the average distance between the final position and the target position; success rate (SR), calculated as the percentage of paths with a navigation error of less than 3 meters; success rate under the ideal stopping strategy (OSR); and path length-weighted success rate (SPL), which comprehensively considers success rate and efficiency.

[0102] 2. Construct an overall navigation framework based on a visual language model and realize feature encoding of multimodal visual observation and command information.

[0103] Figure 1This is a schematic diagram of the overall framework of the present invention, showing the composition of the historical topology visual language navigation model based on the multi-curvature spatiotemporal difference thinking chain. The navigation method proposed in this invention mainly consists of three core components: a large visual language model, an encoder-decoder-based instruction processing model, and a navigation strategy network, which work together to complete the navigation task.

[0104] Specifically, for the data received by the system from... Navigation instructions composed of word embeddings The agent is required to follow the instruction in a predefined undirected graph. Navigation is performed within this context, where... This represents the set of discrete nodes that the agent can navigate to, and These represent the connectivity edges between these nodes.

[0105] at any time step The agent perceives a set of multi-view navigable node candidates by observing the surrounding environment of its current location. .in, This represents the total number of candidate nodes within the currently visible range, for each unique view. A visual observation at a navigable node is represented by its angle relative to the agent's current heading. .

[0106] The core task of an intelligent agent is to select relative to the current visual observation. The right angle This allows the agent to predict and execute subsequent navigation actions. The agent needs to learn network parameters. To master navigation strategies .

[0107] Figure 2 This is a schematic diagram of image feature processing in the visual observation sequence of the present invention, illustrating the dual-branch network design of multi-view image encoding and object distance perception. Figure 2 (a) in the image represents the target detection result. The bounding box is used to accurately locate objects in the scene (such as chairs, sofas, windows, etc.) and establish spatial anchor points for visual feature extraction. Figure 2 (b) in the figure represents the instance segmentation result, which further distinguishes different object instances (such as chair and sofa areas of different colors), refines the semantic granularity of visual features, and clarifies the instance boundaries of objects. Figure 2(c) in the diagram represents the fusion depth result, where detected objects (such as chairs) are assigned specific distance information (2.15m), quantifying the spatial positional relationships of objects. The three elements work synergistically to achieve progressive spatial perception enhancement—object detection, instance segmentation, and deep fusion—in the second branch of the dual-branch network. This enables the agent to upgrade from recognizing objects to understanding their semantic categories, instance boundaries, and spatial distances, injecting precise spatial observation details into the large language model and thus improving its ability to understand the spatial structure of the environment during navigation reasoning.

[0108] To effectively encode multi-view images in the environment and build spatial awareness capabilities for large language models, this embodiment employs a dual-branch network design:

[0109] On one hand, the Q-former network in the InstructBLIP multimodal large model, based on contrastive learning, is used to encode each environment view as a fixed-length visual token. Specifically, for candidate view images, visual features are first extracted using the visual encoder in the large model. Subsequently, queries can be learned. Embedded instruction text Self-attention queries are then performed, and cross-attention is applied with the aforementioned visual features to obtain the recognized image queries. These queries are then linearly projected and input as image tags into the large language model, allowing it to preserve image details during inference.

[0110] On the other hand, by combining the intersection-union ratio (IoU) with the results of object detection and target segmentation in a specific implementation, the agent's perception of distance to surrounding objects is enhanced. Specifically, the IoU is used to match the bounding boxes of the object detector DETR. Masks for the target segmentation model Mask2Former Then through the bounding box and mask Get the first RGB image Each object has a set of pixels, which are then mapped to a depth map. The average value of the pixels in the depth map is calculated as the first pixel. The distance between each object and the agent. The visual observation information in this branch is input as a feature vector into the encoder of the large language model to preserve detailed spatial observation information.

[0111] In addition, to pass directional information to the large language model, the directional information is in a structured input format. Injected into navigation prompts. Additionally, to facilitate the flexible insertion of image markers and instructions into the prompts, special markers were introduced, including those for marking the start and end of images. and the marking of the start and end of instructions. <inst> 、< / inst> .

[0112] Finally, navigation inference data is generated from the R2R training set, and the Q-former and projection layers on the prediction labels are tuned using an autoregressive training objective to optimize their performance. Through the above process, a complete visual language model is constructed. The decoder of this visual language model can generate a detailed description of the navigation inference process, providing input features for subsequent action prediction.

[0113] 3. Design a visual reference reasoning method based on the spatiotemporal differences of multi-curvature manifolds to generate high-quality navigation instructions.

[0114] Figure 3 This diagram illustrates the visual reference reasoning method based on the spatiotemporal differences of multi-curvature manifolds in this invention, showcasing the combination of multi-curvature manifold spaces and their application in visual reference selection. Specifically, on one hand, it selects sparse objects (such as treadmills and projectors) as visual references from a spatial dimension, leveraging their uniqueness within the scene to enhance the recognition of the environmental spatial structure; on the other hand, it selects action decisions (such as continuous actions) that lead to scene changes from a temporal dimension. , , As visual references, it captures the causal relationship between actions and scene changes. Based on this, through multi-curvature manifold spatiotemporal difference thought chain reasoning, combined with target instructions and a set of reference objects (<treadmill, meeting room, projector>), the model can generate a thought process highly aligned with the instructions, thereby accurately selecting key visual references and providing reliable spatiotemporal information support for navigation decisions.

[0115] This embodiment introduces spatiotemporal variability based on multi-curvature manifolds to guide large models in generating high-quality navigation commands using key visual references in the navigation trajectory. This method is implemented through the following steps:

[0116] First, construct a multi-curvature manifold space:

[0117] Introducing combinatorial manifold spaces to enhance the effectiveness of visual observation is based on the principle of utilizing the different geodesic distances provided by manifolds with different curvatures, allowing features to acquire more comprehensive embedding relationships. Specifically, this includes:

[0118] Euclidean space mapping: For the feature vector X of a multi-view visual observation sequence, through an encoder... Directly extract features to obtain Euclidean space features. :

[0119]

[0120] Hyperbolic space mapping: To map features to a hyperbolic space with constant negative curvature, an exponential mapping is employed. Specifically, given an encoder... The extracted features are first calculated by their modulus, and then mapped using the hyperbolic tangent function and an appropriate scaling factor to obtain the hyperbolic space features. :

[0121]

[0122] in, It is the hyperbolic tangent function. This is a scaling factor based on the curvature parameter; Let be the curvature parameter of hyperbolic space, derived from the curvature of hyperbolic space. Derivation, It is usually set to -1; Represents the norm; For feature encoder Feature extraction results for input X.

[0123] Spherical Space Mapping: To map features to a spherical space with constant positive curvature, ensuring that the feature vectors lie on a unit sphere, a normalized encoder is used. Extracted features, and processed by a spherical projection function. Processing yields spherical spatial features.

[0124]

[0125] in, For feature encoder Feature extraction results for input X Represents the norm; For spherical projection functions; spherical projection functions The specific implementation is as shown in the formula. and As shown, by using Cartesian coordinates ( , , Convert to spherical coordinates ( accomplish:

[0126]

[0127]

[0128] in, The distance from the origin in Cartesian coordinates; It is the inverse cosine function; It is a two-parameter arctangent function; , , These are the x, y, and z components of a three-dimensional Cartesian coordinate system.

[0129] Secondly, select visual references with spatiotemporal differences: from the instruction and path annotation pairs in the training set, first extract the nouns in the instructions as the original language visual reference set. Subsequently, the visual reference set was supplemented using the following methods:

[0130] Calculate temporal importance score: Construct a sequence including mean pooled features of the panoramic view. and through and The importance score of calculating the cosine distance between them. :

[0131]

[0132] Where τ represents the time dimension, t represents the time step, and K represents the observed view number; For the mean pooled feature of the panoramic view at time step t, the superscript... This indicates the result after mean pooling; The mean pooled feature of the panoramic view at time step t+1; Represents the norm; Let be the feature of the k-th view at time step t.

[0133] Calculate spatial importance score: at time step t, extract action perspective. All objects are used as a candidate visual reference set, and spatial importance scores are assigned based on the object's presence in other candidate views. :

[0134]

[0135] Where t represents the time step, n represents the reference object number, and a represents the spatial dimension; Let be the cosine similarity between the action viewpoint at time step t and the first candidate viewpoint. Let be the cosine similarity between the action viewpoint at time step t and the second candidate viewpoint. Let be the cosine similarity between the action viewpoint at time step t and the third candidate viewpoint.

[0136] Calculate the final spatiotemporal dissimilarity score: For a candidate visual reference object, its final spatiotemporal dissimilarity score. :

[0137]

[0138] in, This represents the spatial importance score of the nth candidate visual reference at time step t. This represents the time importance score at time step t.

[0139] Obtaining the comprehensive spatiotemporal dissimilarity score under multi-curvature manifolds: The visual mapping features under Euclidean, hyperbolic, and spherical spaces are processed according to the above-mentioned spatiotemporal dissimilarity visual reference selection process to obtain the comprehensive spatiotemporal dissimilarity score under multi-curvature manifolds. :

[0140]

[0141] in, For the first Spatial importance score of the nth candidate visual reference at time step; For the first Time importance score for time step; For hyperbolic space Spatial importance score of the nth candidate visual reference at time step; For hyperbolic space Time importance score for time step; In spherical space, the first Spatial importance score of the nth candidate visual reference at time step; In spherical space, the first The time importance score of the time step Represents hyperbolic space. Represents a spherical space. τ represents the time step, and τ represents the time dimension;

[0142] Ultimately, retain Greater than the set threshold The visual reference objects serve as a new supplementary set of visual reference objects. The complete set of landmarks for the trajectory is constructed as follows: .

[0143] This method combines multi-curvature manifold embedding and spatiotemporal difference filtering, enabling large models to more accurately identify and utilize key visual references in the environment, thereby generating more precise and executable navigation instructions.

[0144] 4. Construct a navigation strategy network based on the historical representation of topology to provide long-term navigation memory and backtracking capabilities.

[0145] Figure 4This diagram illustrates the navigation strategy network module based on historical topology representation in this invention, showcasing topology graph maintenance, node representation, and relationship modeling. Specifically, it involves dynamically updating the topology graph to continuously model the environment. At time T-1, the topology graph includes visited nodes (point A), the current node (node ​​D), and accessible nodes (nodes B, C, and E). By integrating new visual observations, including actions (node ​​D → E) and new nodes (nodes F and G), the topology graph at time T is updated, clearly distinguishing the states of visited (nodes A and D), the current (node ​​E), and accessible (nodes B, C, F, and G) nodes. Simultaneously, node types and interaction logic are clearly defined, enabling node representation and relationship modeling. This mechanism allows the agent to track historical navigation trajectories (visited nodes), locate the current position (current node), and plan accessible subsequent paths (accessible nodes), providing memory support and decision-making basis for long-distance navigation. This ensures precise alignment between the navigation process and the instructions "walk past the counter, out of the kitchen, and wait next to the landscape painting on the wall."

[0146] This embodiment addresses the shortcomings of large language models in visual language navigation tasks, namely insufficient understanding of spatial structure and lack of long-range reasoning experience. It proposes and constructs a navigation strategy network based on a topology graph. This network dynamically maintains the topology graph during agent movement and uses it as a memory mechanism for tracking navigation experience. This allows the large language model to select the next action from the constructed topology graph and effectively plan and backtrack to unvisited nodes when the path selection is incorrect. The specific implementation process is as follows:

[0147] First, construct and maintain a dynamic environment topology graph: Environment Graph Initially, the intelligent agent was unknown.

[0148] The method model of this invention gradually builds its own map by observing along a path. The navigation map is represented as... , which includes Nodes. Node set. There are three types of nodes: visited nodes, navigable nodes, and the current node. The agent can access a panoramic view of visited and current nodes, while navigable nodes are unexplored and can only be observed from the visited portion of the view.

[0149] At every step The system will add the current node and its adjacent unvisited nodes to the topology graph and update the set of reachable edges accordingly. When When new observations are made, the visual representations of the current node and navigable nodes are also updated.

[0150] Secondly, a historical memory topology graph storage and node representation are constructed. The historical memory topology graph storage consists of visited nodes along the trajectory and adjacent unexplored nodes. In this invention:

[0151] Each visited node is represented by the average pooling feature of all its candidate views.

[0152] Each unexplored node is represented by partial pooling of the corresponding views of all its adjacent visited nodes in the trajectory.

[0153] The representation of each view is determined by its visual features. The sum, the directional embedding representing the node position (Egocentric view embedding relative to the direction and distance of the current node), step embedding representing the traversal order (The latest access time step of the embedded access node, 0 for unvisited nodes) and action flags (This indicates whether the view was selected as the target during the previous movement) together constitutes the whole.

[0154] In addition, a "stop" node is added to the history memory topology graph memory to represent a stop action, and it is connected to all other nodes, thereby encoding the visited nodes through different navigation histories to improve alignment with instructions.

[0155] Next, the spatial and instruction relationships between nodes are modeled. To model the spatial relationships between nodes, a multi-layer Transformer is used, whose input nodes are represented as follows:

[0156]

[0157] in, This indicates that self-attention encoding is performed on node features; This represents the sum of visual features of the i-th view; Embed the direction of the i-th view; Embed the steps for the i-th view; The action tag for the i-th view is given, where i represents the view number and M represents the number of views for a node.

[0158] The navigation graph is constructed at step t and represented as follows: Node embedding The node embeddings are then fed into a multi-layer cross-modal transformer (Transformer) to model the relationships between instructions and nodes. This process includes:

[0159] First, node embeddings undergo cross-attention with instructions encoded by the large language model.

[0160] Then, semantic information is fused using graph-aware self-attention. The graph-aware self-attention algorithm considers the distance and visual similarity between nodes during the fusion process to enhance contextual understanding; its calculation formula is as follows:

[0161]

[0162] in, This represents the set of discrete nodes that the agent can navigate to. From The obtained pairwise distance matrix, and These are two learnable parameters; For activation functions; To query the projection matrix; The key projection matrix; The projection matrix is ​​a value. This is a matrix transpose operation; Scaling factor For node feature dimensions; It is a pairwise distance matrix; These are the learnable weight parameters of the distance matrix; These are learnable bias parameters.

[0163] Finally, a two-layer feedforward network is employed to process the output node representation of the graph-aware self-attention mechanism to generate an action score. Based on this action score, the agent selects the node with the highest score as its target and controls its movement to the selected node by following the shortest path in the graph memory. To encourage exploration, this invention also incorporates a mechanism to mask the scores of already visited nodes. Furthermore, the fusion module for different branches is also a two-layer feedforward network.

[0164] 5. A multi-stage training method combining imitation learning and data aggregation is adopted to optimize the model end-to-end.

[0165] First, the first stage: fine-tuning of Q-former.

[0166] In this phase, the model is initialized from the pre-trained InstructBLIP model. Based on this, following common fine-tuning methods for large multimodal models, the Q-former module is fine-tuned on the collected navigation inference data using only the frozen large language model and visual encoder. The goal of this phase is to enable the Q-former to better extract image labels relevant to the navigation task.

[0167] In addition, the model's reasoning uses a pre-defined prompt format: "As an AI navigating an indoor environment, you have been given a task {instruction}. You find yourself at a specific moment in executing this command. Based on your current observations of the surrounding environment, including obstacles, paths, and relevant landmarks, determine the next step to complete this task. Your response should briefly describe the environment you are facing and explain the direction or action you will take. Summarize this in a concise paragraph, integrating your observations and decision-making process."

[0168] Secondly, the second stage: connecting and fine-tuning the pre-trained visual language model with the downstream navigation strategy:

[0169] After the first stage is completed, the pre-trained visual language model is connected to the downstream navigation policy network. In this stage, only the policy network is fine-tuned, while the visual language model remains frozen. When fine-tuning the navigation policy network, this invention combines two learning methods: imitation learning loss and data aggregation loss, to address the error accumulation problem that may occur during behavior cloning.

[0170] Behavioral cloning, as an imitation learning method, treats the expert's decision-making process as a supervised learning problem. The agent predicts actions by learning from expert demonstration data, with the training objective of making the predicted actions as close as possible to the expert's actions. In the visual-language navigation task, cross-entropy loss is used as the loss function for imitation learning:

[0171]

[0172] in, It is the natural logarithm function; For navigation strategy functions; The decision is the one listed in the true decision label at time step t; Represents conditional probability; These are the textual features of the encoded instructions; This is the dynamic topology graph at time step t; This represents the total time step of the navigation trajectory;

[0173] To address the error accumulation problem in behavior cloning, a data aggregation strategy is introduced. This strategy allows the agent to execute its own policy and collect additional expert data after deviating from the expert trajectory, continuously improving the model and thus constructing a more complete state-action mapping. The loss function for data aggregation is:

[0174]

[0175] in, For navigation strategy functions; It refers to the textual characteristics of the instructions after they have been encoded. This represents a partial graph generated by the agent through policy upsampling. The pseudo-decision label determined by the shortest path to the destination;

[0176] Finally, the overall training loss function It consists of both imitation learning loss and data aggregation loss:

[0177]

[0178] in, It is a balancing factor used to balance the two loss terms.

[0179] Through the above two-stage training and joint loss function optimization, the model can effectively learn and improve its navigation ability and robustness in complex indoor environments.

[0180] 6. Utilize the trained navigation model to perform navigation reasoning in the indoor environment and generate executable action commands to guide the agent to the target.

[0181] Figure 5 and Figure 6 This document demonstrates the visualization process of a large-scale model inference. This example utilizes a navigation model trained through multiple stages to perform navigation inference in an indoor environment. Specifically:

[0182] Using a navigation model trained through multiple stages, the agent begins with an initial scene (bathroom) and, in conjunction with the instructions "Turn right until you are facing a bedroom and a bed. Head towards the bed, then turn right and walk towards the bedroom door. Turn right into the hallway, then turn left. Go to the top of the stairs," proceeds with navigation reasoning step by step.

[0183] Step 0: The agent observes that it is currently in a bathroom, with a bathtub on the right, a shower area on the left, and a door directly in front. The model, considering the instructions, decides to move to the bed visible on the right, laying the groundwork for subsequent actions.

[0184] Step 1: After moving into the bedroom, the agent observes that there is a bed on the right, a bathroom with a bathtub and sink on the left, and a door leading to other areas directly in front. Following the instructions, the model determines that its next step is to move towards the bed, then turns right and moves towards the bedroom door.

[0185] Step 2: Observe the bedroom further. The door directly in front leads to the outer room. There is a chair with a footstool on the left and a bedside table with a lamp on the right. Continue to align the model with the instructions, clearly indicating that the action of "walking towards the bed and turning right towards the bedroom door" still needs to be performed.

[0186] Step 3: Upon approaching the bedroom door, the agent observes an artwork hanging on the curved wall to the right, and a door directly in front leading to the outer room. Combining this with the remaining instructions, the model determines that after reaching the bedroom door, it needs to turn right to enter the hall, then turn left to reach the top of the stairs.

[0187] Step 4: After entering the corridor, the agent observes the curved wall on the right and the staircase leading to the top on the left. Following the instructions, the model executes the action of "entering the corridor and turning left," and proceeds towards the top of the staircase.

[0188] Step 5: Upon reaching the staircase entrance, the agent observes an upward-curving staircase on the left and a corridor leading to another part of the building on the right. The model executes a "turn left" maneuver, moving towards the top of the staircase.

[0189] Step 6: Upon finally reaching the top of the stairs, the agent observes that there is an upward staircase on the right and a passage leading to another part of the corridor on the left, confirming that all the actions required by the instruction have been completed.

[0190] from Figure 5 and 6 As can be seen, the model of this invention can comprehensively describe the movement process of the intelligent agent in the navigation task and its progress in executing target instructions. By integrating the observation of the current environment, past navigation experience and decision summary, it can finally deduce the action instructions that can guide the intelligent agent to accurately reach the target and generate an executable action sequence.

[0191] Table 1 compares the results of this invention with navigation methods based on large models on the R2R dataset.

[0192] Table 1

[0193]

[0194] As shown in Table 1, in unseen validation set scenarios, the navigation performance of the proposed model (versions 1.5B and 5B) is comprehensively superior to existing large-model-based navigation methods (such as NavGPT, MapGPT, and DiscussNav). Specifically, version 5B of this invention achieves the highest success rate (SR) of 74%, the highest success rate (OSR) under the ideal stopping strategy (OSR) of 82%, and the highest path length weighted success rate (SPL) of 63% among all compared methods. Simultaneously, the navigation error (NE) is as low as 2.92, significantly lower than other methods, indicating that the proposed model leads in generalization ability, navigation accuracy, and efficiency in unknown environments.

[0195] Table 2 compares the results of this invention with those based on large-scale pre-training on the R2R dataset.

[0196] Table 2

[0197]

[0198] As shown in Table 2, compared with traditional navigation methods based on large-scale pre-training (such as PREVALENT), the present invention maintains similar trajectory length (TL) and navigation error (NE) in the seen validation set scenarios, and improves the success rate (SR), ideal stopping strategy success rate (OSR), and path length weighted success rate (SPL). In the unseen validation set scenarios, the advantages of the present invention are more obvious, especially the significant improvement in navigation error (NE) and weighted success rate (SPL). This indicates that the present invention effectively enhances the model's understanding of the spatial structure of complex environments and the navigation robustness in unexplored environments through multi-curvature manifold feature mapping and historical topology map navigation strategies.

[0199] Other methods are as follows, as shown in Tables 1 and 2:

[0200] NavGPT corresponds to the method proposed by Zhou et al. (Zhou G, Hong Y, Wu Q. Navgpt: Explicitreasoning in vision-and-language navigation with large language models [C] / / Proceedings of the AAAI Conference on Artificial Intelligence: volume 38.AAAI, 2024: 7641-7649).

[0201] MapGPT corresponds to the method proposed by Chen et al. (Chen J, Lin B, Xu R, et al. Mapgpt: Map-guided prompting for unified vision-and-language navigation[C] / / NorthAmerican Chapter of the Association for Computational Linguistics.Association for Computational Linguistics, 2024: 9796-9810).

[0202] DiscussNav corresponds to the method proposed by Long et al. (Long Y, Li X, Cai W, et al. Discussbefore moving: Visual language navigation via multiexpert discussions[C] / / IEEE International Conference on Robotics and Automation. IEEE, 2024: 17380-17387).

[0203] NavCoT corresponds to the method proposed by Lin et al. (Lin B, Nie Y, Wei Z, et al. Navcot: Boosting llm-based vision-and-language navigation via learning disentangledreasoning[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025).

[0204] LangNav corresponds to the method proposed by Pan et al. (Pan B, Panda R, Jin S, et al. Langnav: Language as a perceptual representation for navigation [C] / / North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics, 2024: 950-974).

[0205] NaviLLM corresponds to the method proposed by Zheng et al. (Zheng D, Huang S, Zhao L, et al. Towards learning a generalist model for embodied navigation [C] / / Proceedingsof the IEEE / CVF Conference on Computer Vision and Pattern Recognition. IEEE,2024: 13624-13634).

[0206] PREVALENT corresponds to the method proposed by Hao et al. (Hao W, Li C, Li X, et al. Towards learning a generic agent for vision-and-language navigation via pre-training[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2020: 13134-13143).

[0207] AirBERT corresponds to the method proposed by Guhur et al. (Guhur PL, Tapaswi M, Chen S, et al. Airbert: In-domain pretraining for vision-and-language navigation[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision.IEEE, 2021: 1614-1623).

[0208] VLN BERT corresponds to the method proposed by Hong et al. (Hong Y, Wu Q, Qi Y, et al. A recurrentvision-and-language BERT for navigation[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. IEEE, 2021: 1643-1653).

[0209] MARVAL corresponds to the method proposed by Kamath et al. (Kamath A, Anderson P, Wang S, et al. Anew path: Scaling vision-and-language navigation with synthetic instructions and imitation learning[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition. IEEE, 2023: 10813-10823).

[0210] HAMT corresponds to the method proposed by Chen et al. (Chen S, Guhur P, Schmid C, et al. Historyaware multimodal transformer for vision-and language navigation[C] / / Conference on Neural Information Processing Systems. MIT Press, 2021: 5834-5847).

[0211] DUET corresponds to the method proposed by Chen et al. (Chen S, Guhur PL, Tapaswi M, et al. Thinkglobal, act local: Dual-scale graph transformer for vision-and-languagenavigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. IEEE, 2022: 16537-16547).

[0212] HOP+ corresponds to the method proposed by Qiao et al. (Qiao Y, Qi Y, Hong Y, et al. Hop+: History-enhanced and order-aware pre-training for visionand-languagenavigation[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(7): 8524-8537).

[0213] BEVBert corresponds to the method proposed by Liu et al. (Liu R, Wang X, Wang W, et al. Bird's-eye-view scene graph for vision-language navigation [C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. IEEE, 2023: 10968-10980).

[0214] DUET+ScaleVLN corresponds to the method proposed by Wang et al. (Wang Z, Li J, Hong Y, et al. Scaling data generation in vision-and-language navigation[C] / / Proceedings of the IEEE / CVF International Conference on Computer Vision. IEEE, 2023: 12009-12020).

[0215] GOAT corresponds to the method proposed by Wang et al. (Wang L, He Z, Dang R, et al. Vision-and-language navigation via causal learning[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. IEEE, 2024: 13139-13150).

[0216] This invention first constructs an overall navigation framework based on a visual language model and employs a dual-branch network to achieve effective feature encoding of multimodal visual observation and command information. Second, it designs and introduces a visual reference reasoning method based on the spatiotemporal differences of multi-curvature manifolds to guide the model in generating high-quality, executable navigation commands and effectively reduce illusion effects during the reasoning process. Next, it constructs a navigation policy network based on topological history representation, providing the model with long-term navigation memory and backtracking capabilities, enhancing its robustness in complex environments. Finally, it employs a multi-stage training method combining imitation learning and data aggregation to optimize the entire model end-to-end. Through the above technical solutions, this invention aims to improve the spatial understanding, remote reasoning, and navigation capabilities of large-scale visual language models in complex indoor environments. Experiments demonstrate the effectiveness of this invention and verify the contribution of each module of the proposed method.

[0217] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A manifold-aware history topology map guided large model navigation method, characterized in that The method comprises the following steps: 1) preparing a visual language navigation data set; 2) constructing an overall navigation framework based on a visual language model, and realizing feature coding of multi-modal visual observation and instruction information; 3) designing a visual reference reasoning method based on the spatiotemporal difference of multi-curvature manifold, for generating high-quality navigation instructions; 4) constructing a navigation strategy network based on topological structure history expression, to provide long-term navigation memory and backtracking capability; 5) adopting a multi-stage training method combining imitation learning and data aggregation to optimize the model in an end-to-end manner; 6) using the trained navigation model to perform indoor environment navigation reasoning, and generating executable action instructions to guide the agent to reach the target.

2. The manifold-aware history topology map-guided large model navigation method of claim 1, wherein In step 1), the specific method for preparing the visual language navigation data set is as follows: the R2R data set is adopted, which is collected from a Matterport 3D simulator and contains 90 indoor scenes of real-world building scale; specifically, the data set contains 10,800 panoramic views, 21,567 navigation instructions and 7,189 navigation trajectories corresponding thereto, wherein each navigation trajectory can correspond to 3 navigation instructions; the R2R data set is divided into four parts: a training set, a validation set in a seen environment, a validation set in an unseen environment and a test set; when evaluating the performance, the navigation indicators adopted include: trajectory length, used to measure the average path length; navigation error, calculated as the average distance between the final position and the target position; success rate, calculated as the percentage of paths with a navigation error less than 3 meters; success rate under ideal stop strategy; and path length weighted success rate, which considers both success rate and efficiency.

3. The manifold-aware history topology map-guided large model navigation method of claim 1, wherein In step 2), the specific method for constructing the overall navigation framework based on the visual language model and realizing feature coding of multi-modal visual observation and instruction information is as follows: constructing an overall navigation framework composed of a large visual language model, an encoder-decoder-based large model and a navigation strategy network; adopting a double-branch network design for feature coding: first branch: using the Q-former network in InstructBLIP to extract image tags from visual observation and instructions, and taking the image tags as the visual input of the large language model; second branch: matching object detection and target segmentation results through intersection over union, to enhance the agent's perception of the distance of surrounding objects; inputting the visual observation information obtained by the two branches into the encoder of the large model in the form of a feature vector, while injecting directional information into the navigation prompt in a structured input format, and introducing special markers to insert image tags and instructions into the prompt words; generating navigation reasoning data based on the training set, and adjusting the Q-former and projection layer on the predicted markers using autoregressive training targets, to construct a complete visual language model.

4. The manifold-aware history topology map-guided large model navigation method of claim 1, wherein In step 3), the specific method for designing a visual reference reasoning method based on the spatiotemporal difference of multi-curvature manifold, for generating high-quality navigation instructions is as follows: constructing a multi-curvature manifold space, including Euclidean space mapping, hyperbolic space mapping and spherical space mapping; Euclidean space mapping: for the feature vector X of the multi-view visual observation sequence, through the encoder directly extract features to obtain Euclidean space features : Hyperbolic space mapping: To map the features to a hyperbolic space with constant negative curvature, an exponential mapping is employed; specifically, given an encoder The extracted features, first their length is computed, then a hyperbolic tangent function and an appropriate scaling factor are applied to map them to hyperbolic space features : wherein, is a hyperbolic tangent function, is a scaling factor based on the curvature parameter; is a hyperbolic space curvature parameter, derived from the curvature derivation, is typically set to -1; denotes a norm; is a feature encoder a feature extraction result for the input X; Spherical space mapping: To map the features to a spherical space with constant positive curvature, ensuring that the feature vectors lie on the unit sphere, the encoder is normalized extracted features, and passed through a spherical projection function processing, resulting in spherical space features : wherein, is a feature encoder a feature extraction result of the input X, denotes a norm; is a spherical projection function; the spherical projection function is implemented as shown in formulas and by converting the Cartesian coordinates , , into the spherical coordinates ; wherein the spherical projection function is implemented as shown in formulas and by converting the Cartesian coordinates , , into the spherical coordinates ; wherein is the distance of the Cartesian coordinate from the origin; is the inverse cosine function; is the two-argument inverse tangent function; , , are the x, y, z components of the three-dimensional Cartesian coordinate, respectively; From the instruction and path annotation pairs of the training set, first extract the nouns in the instructions as the original language set of visual referents ; subsequently, the set of visual referents is supplemented by the following methods: Computing the temporal importance score: constructing a sequence of panorama view mean-pooled features and by and the cosine distance between : where τ denotes the time dimension, t denotes the time step; K represents the view number of observation; is the panoramic view mean-pooling feature for the t-th time step, and the superscript denotes after mean-pooling; is the panoramic view mean-pooling feature for the t+1-th time step; denotes the norm; is the feature of the k-th view for the t-th time step; Compute spatial importance scores: At time step t, extract all objects in the action perspective as a set of candidate visual references, and assign a spatial importance score to each object based on how often it appears in other candidate views : wherein t represents a time step, n represents a reference number, and a represents a spatial dimension; is a cosine similarity between the action view of the t-th time step and the first candidate view, is a cosine similarity between the action view of the t-th time step and the second candidate view, is a cosine similarity between the action view of the t-th time step and the third candidate view. computing a final spatiotemporal distinctiveness score: for a candidate visual reference object, its final spatiotemporal distinctiveness score : wherein, is the spatial importance score for the nth candidate visual reference at the tth time step; is the temporal importance score at the tth time step. Obtaining a comprehensive spatiotemporal difference score under a multi-curvature manifold: the visual mapping features under the Euclidean, hyperbolic and spherical spaces are processed according to the above spatiotemporal difference visual reference screening process to obtain a comprehensive spatiotemporal difference score under a multi-curvature manifold : wherein, is the spatial importance score of the n-th candidate visual reference at the time step is the temporal importance score at the time step is the spatial importance score of the n-th candidate visual reference at the time step is the temporal importance score at the time step is the spatial importance score of the n-th candidate visual reference at the time step is the temporal importance score at the time step is the spatial importance score of the n-th candidate visual reference at the time step is the temporal importance score at the time step is the spatial importance score of the n-th candidate visual reference at the time step is the temporal importance score at the time step denotes a hyperbolic space, denotes a spherical space, denotes a time step, and denotes a time step, and denotes a time step, and By combining multi-curvature manifold embedding and spatiotemporal discrepancy filtering, the large model can more accurately identify and utilize key visual references in the environment, thereby generating more accurate and executable navigation instructions.

5. The manifold-aware history topology map-guided large model navigation method of claim 1, wherein In step 4), the navigation strategy network based on topological structure history expression is constructed, and the specific method for providing long-term navigation memory and backtracking capability is: Constructing and maintaining dynamic environment topologies wherein the topologies include nodes ; a set of nodes including three types of nodes: visited nodes, navigable nodes, and the current node; the agent has access to a panoramic view of the visited nodes and the current node, while the navigable nodes are unexplored and only partially observed from the visited locations; at each step the current node and its neighboring unvisited nodes are added to the topology, and the set of reachable edges is updated; A history memory topological graph memory and node representation are constructed, the memory is composed of visited nodes and adjacent unexplored nodes along the trajectory, wherein: Each visited node is represented by the average pooling features of all candidate views thereof; Each unexplored node is represented by the partial pooling of the corresponding views of all adjacent visited nodes in the trajectory thereof; the representation of each view is composed of a sum of its visual features directional embeddings representing node positions step embeddings representing traversal order and action labels together; A "stop" node is added to the history memory topological graph memory to represent the stop action, and is connected with all other nodes, so as to encode the visited nodes through different navigation histories to improve the alignment with the instructions; A multi-layer Transformer is used to model the spatial relationship between the nodes, and the input node representation is: wherein, represents self-attention encoding on node features; is a visual feature sum for the i-th view; is a direction embedding for the i-th view; is a step embedding for the i-th view; is an action label for the i-th view, i denotes a view number; M is a number of views representing nodes. The node embedding is input into a multi-layer cross-modal Transformer to model the relationship between the instructions and the nodes; the process includes: Cross-attention between the node embedding and the instruction encoded by the large language model; Semantic information fusion through graph-aware self-attention; the graph-aware self-attention algorithm considers the distance and visual similarity between nodes in the fusion process to enhance the understanding of the context, and the calculation formula is as follows: wherein, represents a set of discrete nodes that the agent can navigate to, is a pairwise distance matrix obtained from and are two learnable parameters; is an activation function; is a query projection matrix; is a key projection matrix; is a value projection matrix; is a matrix transpose operation; is a scaling factor, is a node feature dimension; is a pairwise distance matrix; is a learnable weight parameter for the distance matrix; is a learnable bias parameter;​ A two-layer feedforward network is used to process the output node representation of the graph-aware self-attention to generate action scores; the agent selects the node with the highest score as the target and moves along the shortest path in the graph memory, and the scores of the visited nodes are masked.

6. The manifold-aware history topology map-guided large model navigation method of claim 1, wherein In step 5), the multi-stage training method combining imitation learning and data aggregation is used to optimize the model end-to-end, and the specific method is: Two-stage training is performed: First stage: initialized from the InstructBLIP model, using a frozen large language model and a visual encoder, only fine-tuning the Q-former on navigation reasoning data, and the model reasoning uses a preset prompt format; Second stage: connecting the pre-trained visual language model with the downstream navigation strategy, only fine-tuning the strategy network, and the visual language model remains frozen; Optimization combined with imitation learning loss and data aggregation; The imitation learning loss is: wherein, is a natural logarithm function; is a navigation policy function; is a decision in the true decision label of the t-th time step; denotes a conditional probability; is a text feature after encoding of the instruction; is a dynamic topology graph of the t-th time step; is a total time step length of the navigation trajectory; The data aggregation loss is: wherein, is a navigation policy function; is a text feature after instruction encoding, denotes a partial graph generated by the agent through policy upsampling is a pseudo decision label determined by the shortest path to the destination Finally, the total training loss function is composed of the imitation learning loss and the data aggregation loss: wherein is a balancing factor for balancing the two loss terms; Through the above two-stage training and joint loss function optimization, the model can effectively learn and improve the navigation ability and robustness in complex indoor environments.

7. The manifold-aware history topology map-guided large model navigation method of claim 1, wherein In step 6), the trained navigation model is used to perform indoor environment navigation reasoning and generate executable action instructions to guide the agent to reach the target, specifically: using the multi-stage trained navigation model, by integrating observations of the current environment, past navigation experience and decision summaries, the action instructions that can guide the agent to accurately reach the target are inferred, and executable action sequences are generated.

Images