Visual language navigation method based on graph memory and hierarchical cross-modal alignment

Abstract

The application discloses a visual language navigation method based on graph memory and hierarchical cross-modal alignment, and relates to the technical field of visual language navigation.The application proposes a dynamic graph memory mechanism for visual language navigation, which explicitly models the visited viewpoints, candidate viewpoints and their connectivity in the navigation process, so that the historical information participates in subsequent action decision-making in a structured topological form, thereby relieving the problem that the implicit cycle state is difficult to express long-term spatial dependence; a hierarchical cross-modal alignment mechanism is designed, including two levels of global graph-language alignment and local visual-language alignment, which respectively constrain the semantic consistency between complete instructions and historical trajectories, and the fine-grained matching relationship between language word elements and the current candidate observation, so that the application can make the historical spatial structure, complete instruction semantics and current visual evidence cooperatively participate in action decision-making, thereby improving the navigation stability and action selection accuracy in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of visual language navigation technology, specifically a visual language navigation method based on graph memory and hierarchical cross-modal alignment. Background Technology

[0002] Vision-and-Language Navigation (VLN) is an important research area in embodied artificial intelligence, aiming to enable intelligent agents to complete complex navigation tasks based on natural language instructions in real or simulated environments. This task requires the agent to possess cross-modal understanding capabilities, that is, to simultaneously model linguistic semantic information and visual perception information, and to make sequential decisions in dynamic environments. This problem has broad application prospects in service robots, human-computer interaction, and smart homes, and has therefore received widespread attention in recent years.

[0003] With the development of deep learning, especially the successful application of Transformer-based models in multimodal tasks, VLN research has made significant progress. Methods such as VLN-BERT, by introducing pre-trained language models, achieve deep fusion of language and visual features, effectively improving navigation performance.

[0004] However, existing methods still have two key limitations.

[0005] First, most methods rely primarily on current observations and implicit historical states (such as sequence encoding in recurrent neural networks or Transformers) for decision-making. This implicit memory mechanism struggles to effectively model the global spatial structure of the environment. In long-path or complex topological environments, agents are prone to path repetition and directional deviations, limiting their long-term planning capabilities.

[0006] Secondly, in terms of cross-modal alignment, existing methods mostly focus on matching local visual observations with linguistic representations, lacking consistent modeling of the entire navigation path or global semantics. This single-layer alignment mechanism is prone to "local optimum" decisions, meaning that the agent can correctly understand local instruction fragments but cannot grasp the complete instruction as a whole, thus affecting the final navigation performance.

[0007] The limitations of existing technologies necessitate a new solution to these problems. Summary of the Invention

[0008] The purpose of this invention is to provide a visual language navigation method based on graph memory and hierarchical cross-modal alignment to solve the technical problems mentioned in the background art.

[0009] To achieve the above objectives, the present invention provides the following technical solution: a visual language navigation method based on graph memory and hierarchical cross-modal alignment, comprising the following steps:

[0010] S1: Multimodal feature extraction: Encode the input natural language navigation instructions to obtain language lexical features and sentence-level global language features; Encode the panoramic visual observations collected by the agent at the current time step to obtain the visual features of each candidate reachable viewpoint;

[0011] S2: Dynamic topology graph memory online update and representation learning, using visited viewpoints and candidate reachable viewpoints as graph nodes, and spatial reachability relationships between viewpoints and historical navigation trajectory relationships as graph edges, updates the dynamic topology graph online as the agent moves, and performs feature encoding and graph attention message passing on graph nodes to generate graph-level spatial memory context;

[0012] S3: Hierarchical cross-modal alignment and feature fusion. Global graph-language alignment is performed based on graph-level spatial memory context and sentence-level global language features. Local visual-language alignment is performed based on candidate viewpoint visual features and language lexical features. Global alignment information and local alignment information are fused through a gating mechanism to obtain hierarchical alignment fusion features.

[0013] S4: Navigation Action Prediction: Input the hierarchical alignment and fusion features into the cross-modal Transformer encoder to complete multimodal deep interaction, update the temporal navigation state in combination with the recurrent unit, match and score all candidate viewpoints based on the fusion features, and output the optimal navigation action.

[0014] S5: Multi-objective joint training: The model employs imitation learning loss, reinforcement learning loss, local alignment auxiliary loss, global alignment auxiliary loss, and parameter regularization loss to jointly optimize all network parameters.

[0015] Further, S2 includes the following steps:

[0016] S21: The topology graph is incrementally constructed and updated in real time as the agent moves, with the visited viewpoints and candidate unvisited viewpoints as graph nodes and the spatially reachable connectivity between viewpoints as graph edges.

[0017] S22: For each graph node, fuse viewpoint visual features, position orientation encoding, and node access status encoding to generate node features;

[0018] S23: Aggregate node neighborhood information through graph attention message passing to update node representations, and then generate a graph-level global spatial memory context that summarizes the entire explored environment through attention pooling.

[0019] Furthermore, the node feature calculation formula in S22 is as follows:

[0020]

[0021] in, For the first Features of each viewpoint node; Viewpoint visual features; Embedded relative position and orientation of viewpoint; Encode the node status to distinguish between visited nodes and candidate unvisited nodes; , , It is a learnable linear projection matrix.

[0022] Furthermore, in S21, when the agent reaches the current viewpoint... At that time, the incremental update formulas for the node set and edge set of the topological graph are:

[0023]

[0024]

[0025] in, For time steps The set of nodes; For time steps The set of edges; index Indicates the previous time step; From the current viewpoint The set of first-order reachable neighbor viewpoints; Forward viewpoint Any first-order reachable neighbor viewpoint node.

[0026] Only the visited nodes and their first-order candidate neighbors are retained to control the graph computation scale, and feature decay weights are applied to low-confidence candidate nodes.

[0027] Further, S23 includes the following steps:

[0028] S231: Graph attention message passing, updating node representations by aggregating neighborhood node features based on inter-node attention weights, using the following formula:

[0029]

[0030] in, For nodes The neighborhood group, For nodes , Attention weights between them It is a non-linear activation function. , The weight matrix is ​​a learnable weight matrix;

[0031] S232: Attention pooling generates a graph memory context, and a weighted summation of all updated node representations is obtained to obtain the graph-level global spatial memory context. The weighting coefficients are calculated from the learnable vector softmax.

[0032] Further, S3 includes the following steps:

[0033] S31: Perform local visual-language alignment, establish fine-grained semantic matching between visual features of the current candidate viewpoint and language lexical units based on cross-modal attention mechanism, and output local alignment context;

[0034] S32: Perform global graph-language alignment, calculate the semantic similarity between the graph-level global spatial memory context and the sentence-level global language features, and obtain the trajectory global semantic consistency constraint;

[0035] S33: Gated adaptive fusion, dynamically generates learnable gating weights based on temporal navigation state, local alignment context, and graph-level global spatial memory context, fuses local fine-grained visual semantic information and global trajectory semantic information, and outputs hierarchical alignment fusion features.

[0036] Furthermore, the local alignment context in S31 The calculation method is as follows:

[0037]

[0038]

[0039] in, This represents the current panoramic visual representation; This is the lexical feature matrix; , , For learnable projection matrix; For feature dimensions; Match attention weights to visual-language lexical units;

[0040] Furthermore, the formula for calculating the global semantic consistency score in S32 is as follows:

[0041]

[0042] in, For graph-level global spatial memory context; Sentence-level global language features for navigation instructions; and For learnable projection matrix; This is the function for calculating cosine similarity.

[0043] Furthermore, the gating weights and hierarchical alignment fusion features in S33 are calculated as follows:

[0044]

[0045]

[0046] in, For dynamic gating coefficients; The timing navigation state output by the loop unit; For local alignment context; For graph-level global spatial memory context; For the final layered alignment and fusion features; and It is a learnable weight matrix; the gating coefficients are dynamically adjusted according to the navigation phase, taking into account the weight ratio of local visual cues and global spatial memory.

[0047] Furthermore, in S4, for each candidate viewpoint The action score is calculated as follows:

[0048]

[0049]

[0050] in, For layered alignment and fusion features; Candidate viewpoints Visual features; This is an element-wise multiplication operation; For learnable linear layer weights; The probability distribution of each navigation action, Indicates the first The candidate actions are scored, and the action with the highest probability is selected as the output navigation action.

[0051] Furthermore, the total loss function of the model in S5 is:

[0052]

[0053] in, To mimic the learning cross-entropy loss, the expert's shortest path action is used as the supervision label; To reinforce the learning loss, the reward value is calculated based on a weighted average of navigation success rate and path length; Auxiliary loss for local alignment; Auxiliary loss for global alignment; , , , Preset hyperparameter weights; This is the L2 regularization term for all network parameters of the model.

[0054] Compared with the prior art, the beneficial effects of the present invention are:

[0055] 1. This invention proposes a dynamic graph memory mechanism for visual language navigation, which explicitly models visited viewpoints, candidate viewpoints and their connectivity during the navigation process, so that historical information participates in subsequent action decisions in a structured topological form, thereby alleviating the problem that implicit loop states are difficult to express long-term spatial dependencies.

[0056] 2. This invention designs a hierarchical cross-modal alignment mechanism, including two levels: global graph-language alignment and local vision-language alignment, which respectively constrain the semantic consistency between complete instructions and historical trajectories, and the fine-grained matching relationship between language lexical units and current candidate observations;

[0057] 3. This invention conducts comparative experiments, ablation experiments, and training process analysis on the R2R benchmark dataset. The results show that HGMA-VLN achieves simultaneous improvements in SR, SPL, and NE compared to PREVALENT in ValUnseen partitioning, verifying the effectiveness and complementarity of dynamic graph memory and hierarchical cross-modal alignment. Attached Figure Description

[0058] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a schematic diagram of the overall architecture of the HGMA-VLN of the present invention;

[0060] Figure 2 This is a schematic diagram of the dynamic topology memory module of the present invention;

[0061] Figure 3 This is a schematic diagram of the layered cross-modal alignment module of the present invention;

[0062] Figure 4 This is a schematic diagram comparing SR / SPL in the ablation experiment of the module of the present invention;

[0063] Figure 5 This is a schematic diagram of the SR / SPL training variation curve on ValUnseen in this invention;

[0064] Figure 6 This is a schematic diagram of the NE training variation curve on ValUnseen in this invention. Detailed Implementation

[0065] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0066] This invention proposes a visual language navigation method based on graph memory and hierarchical cross-modal alignment. It employs an enhanced visual language navigation framework, combining explicit graph structure memory with a hierarchical cross-modal alignment mechanism to improve the agent's global reasoning ability and fine-grained semantic implementation capability. Specifically, this invention constructs a dynamic topology graph online during navigation, explicitly modeling visited locations and their connectivity relationships, thereby providing the agent with a structured global environment representation.

[0067] Based on this, a two-level cross-modal alignment strategy is designed: on the one hand, through a global graph-language alignment mechanism, semantic matching of language instructions and graph structure representations is performed to enhance path-level semantic consistency; on the other hand, through a local vision-language alignment mechanism, fine-grained associations are established between language lexical units and current candidate visual observations, thereby improving the accuracy of action-level instruction delivery.

[0068] This invention implements the proposed method based on the idea of ​​cyclic cross-modal decision-making, and conducts a systematic evaluation on the standard R2R visual language navigation benchmark dataset.

[0069] This invention employs a cyclical cross-modal decision-making approach as its basic framework. Given a natural language instruction... and time step Panoramic visual observation The agent needs to be in the set of candidate reachable viewpoints. The next action is selected, and the target position is reached within a finite number of steps. Such frameworks typically model the interaction between instructions and the current observation through cross-modal Transformers and use cyclic states to store historical information. However, a single implicit state struggles to stably represent the topological relationships between visited viewpoints and lacks hierarchical constraints on the overall semantics of the instructions and local visual evidence.

[0070] In the standard VLN setting, navigation decision-making is not a single-step visual question-and-answer problem, but a sequential decision-making process that continuously accumulates historical information. The layout of corridors, hallways, or rooms seen by the agent in early time steps may continue to play a role in subsequent turning, backtracking, and stopping decisions. If this historical information is only compressed into an implicit vector, the model has difficulty distinguishing between "locations that have been visited but are not currently visible," "current candidate directions," and "historical nodes related to the target semantics." Therefore, this invention argues that it is necessary to introduce explicit spatial memory in addition to cyclic state modeling, so that historical information participates in decision-making in a structured form.

[0071] On the other hand, navigation instructions typically contain both global path descriptions and local action cues. For example, an instruction might first describe the overall target area, then provide several specific directions and object references. If the model only performs one attentional interaction between the current visual region and the complete sentence, it is easy to mix semantics at different levels, leading to attention dispersion. Based on this observation, this invention decomposes cross-modal alignment into graph-level global alignment and observation-level local alignment to constrain long-term path consistency and progressive action implementation, respectively.

[0072] This invention proposes HGMA-VLN (Hierarchical Graph Memory Alignment for Vision-and-Language Navigation). This method retains the advantages of cyclic cross-modal coding in sequence decision-making, while introducing a dynamic graph memory module and a hierarchical cross-modal alignment module, enabling the model to simultaneously obtain structured spatial memory, path-level semantic consistency, and word-level visual semantic implementation capabilities.

[0073] Specifically as follows:

[0074] A visual language navigation method based on graph memory and hierarchical cross-modal alignment includes the following steps:

[0075] S1: Multimodal feature extraction: Encode the input natural language navigation instructions to obtain language lexical features and sentence-level global language features; Encode the panoramic visual observations collected by the agent at the current time step to obtain the visual features of each candidate reachable viewpoint;

[0076] S2: Dynamic topology graph memory online update and representation learning, using visited viewpoints and candidate reachable viewpoints as graph nodes, and spatial reachability relationships between viewpoints and historical navigation trajectory relationships as graph edges, updates the dynamic topology graph online as the agent moves, and performs feature encoding and graph attention message passing on graph nodes to generate graph-level spatial memory context;

[0077] S3: Hierarchical cross-modal alignment and feature fusion. Global graph-language alignment is performed based on graph-level spatial memory context and sentence-level global language features. Local visual-language alignment is performed based on candidate viewpoint visual features and language lexical features. Global alignment information and local alignment information are fused through a gating mechanism to obtain hierarchical alignment fusion features.

[0078] S4: Navigation Action Prediction: Input the hierarchical alignment and fusion features into the cross-modal Transformer encoder to complete multimodal deep interaction, update the temporal navigation state in combination with the recurrent unit, match and score all candidate viewpoints based on the fusion features, and output the optimal navigation action.

[0079] S5: Multi-objective joint training: The model employs imitation learning loss, reinforcement learning loss, local alignment auxiliary loss, global alignment auxiliary loss, and parameter regularization loss to jointly optimize all network parameters.

[0080] See Figure 1 This invention, based on the HGMA-VLN (Hierarchical Graph Memory Alignment for Vision-and-Language Navigation) framework proposed by the aforementioned method, uses VLN-BERT as its foundation and introduces dynamic graph memory and hierarchical cross-modal alignment mechanisms to enhance the agent's spatial reasoning and semantic understanding capabilities during long-range navigation. The entire model consists of five parts: a language encoder, a visual encoder, an HGMA module, a cross-modal reasoning module, and an action prediction head.

[0081] First, for the input natural language navigation commands, the model uses the BERT encoder to extract contextualized language representations. Simultaneously, the current panoramic view is processed by the ResNet-152 visual encoder to extract visual features, thereby obtaining scene semantic information and candidate viewpoint representations within the environment.

[0082] Subsequently, linguistic and visual features are jointly input into the HGMA module proposed in this invention. This module consists of two parts: dynamic graph memory and hierarchical cross-modal alignment. The dynamic graph memory module continuously maintains the environmental topology during navigation. Node Vt represents visited viewpoints, candidate viewpoints, and key landmark information. It represents spatial connectivity and historical navigation trajectories within the environment. By continuously updating the graph structure, the model can explicitly preserve long-term spatial memory, thereby alleviating the problem that traditional cyclic hidden states struggle to model long-distance spatial dependencies.

[0083] Building upon dynamic graph memory, the hierarchical cross-modal alignment module further establishes semantic correspondences between language, visual observations, and graph memory. Specifically, this module comprises two levels: global graph-language alignment and local vision-language alignment. At the global level, it constrains the path semantic consistency between graph memory and complete instructions; at the local level, it models fine-grained correspondences between candidate visual observations and language lexical units. Through this hierarchical design, the model can simultaneously focus on global navigation goals and local action cues, thereby obtaining more robust cross-modal representations.

[0084] The HGMA-enhanced language-visual features are then fed into a cross-modal Transformer encoder for deep fusion. The cross-modal encoder models the interactions between language, visual, and graph memory information through multi-layer self-attention and cross-attention mechanisms, generating a unified multimodal representation. Subsequently, the fused representation is input into the VLN-BERT decision module, where GRU units maintain historical navigation states and update the current decision representation. .

[0085] Finally, the action prediction head scores and ranks candidate actions based on the fused navigation representation and outputs the next navigation action, including forward, left, right, and stop. Through the spatial structure information provided by the dynamic graph memory and the semantic constraints provided by the hierarchical alignment module, HGMA-VLN can achieve more accurate and stable visual-language navigation in complex indoor environments.

[0086] Dynamic topology graph memory

[0087] See Figure 2 The internal structure of the dynamic graph topology memory module is described. This module mainly consists of three parts: graph construction and online update, node feature representation learning, and graph message passing and aggregation. During navigation, the AI ​​continuously adds visited viewpoints and their candidate neighbors to the dynamic graph and maintains spatial reachability relationships through edge structure. Based on this, the model uses a graph message passing mechanism to aggregate neighborhood information, thereby forming a graph-level memory representation containing the global spatial topology. Compared to methods that only use cyclic hidden states, this design can more explicitly preserve the historical path structure and improve path stability and interpretability in long-range navigation.

[0088] Graph structure modeling

[0089] At time step This invention constructs a dynamic graph by showing the viewpoints that the agent has visited and their reachable neighbors. Each node For a given viewpoint, its node features consist of visual representation, orientation encoding, and navigation status:

[0090]

[0091] in, Representing the panoramic visual features of a viewpoint. Indicates relative position or orientation for embedding. It indicates status information such as whether a node has been visited and whether it is a candidate viewpoint. , and The viewpoint is a learnable projection matrix. and If it is directly reachable in the environment map, then in Add edge to middle This definition enables graph memory to simultaneously record historical trajectories and locally reachable spaces.

[0092] Node status The introduction of this concept allows the model to distinguish between different types of spatial nodes. Visited nodes typically contain more reliable historical visual information, while candidate nodes represent possible directions for further exploration; they play different roles in action decision-making. By modeling state encoding together with visual and positional features, the model can learn more nuanced spatial priors, such as avoiding meaningless return to already confirmed irrelevant nodes or reconsidering a historical branch based on target semantic cues.

[0093] The definition of edges not only represents physical reachability but also provides structural constraints for message passing. Compared to simply piecing together historical observation sequences, graph structures can retain the crucial information of "which viewpoints are adjacent to each other," making it easier for models to form a holistic understanding of the connections between rooms, corridors, and doorways in complex indoor scenes.

[0094] Online update mechanism

[0095] During navigation, the graph structure is incrementally updated as the agent moves. This updates occur when the agent reaches the current viewpoint. If the viewpoint does not already exist If a node is found to be reachable, it is added to the node set; simultaneously, based on environmental feedback, its reachable neighbors are added to the candidate nodes, and corresponding edges are established. The update process can be written as:

[0096]

[0097]

[0098] This online update method avoids prior reliance on the complete environment graph, which is more in line with the task setting of exploration and decision-making in VLN. Compared with implicit history vectors, dynamic graphs can explicitly preserve the connectivity between viewpoints, thereby reducing the risk of repeated exploration and semantic drift in long-path navigation.

[0099] Online updates can also naturally handle uncertainties in the navigation process. Candidate directions seen by an agent at a given location may not all be visited immediately, but these directions still provide local topological clues. Adding them as candidate nodes to the graph helps the model backtrack or replan in subsequent steps. Especially when instructions contain long-distance dependencies such as "go through a certain point and then turn," graph memory can provide the policy with more explicit path references than simple cyclic states.

[0100] To control the graph size, in practice, only visited nodes and their first-order candidate neighbors can be retained, or a decay weight can be set for low-confidence candidate nodes. This preserves the necessary spatial structure while avoiding excessive computational overhead as the trajectory grows.

[0101] Graph representation learning

[0102] To transform the topology into a memory representation usable for decision-making, this invention employs a graph message passing mechanism to update node features. For nodes... Its structured representation is obtained by aggregating neighborhood information:

[0103]

[0104] in, For nodes The neighborhood group, This represents the edge weights or attention weights. This is a non-linear activation function. Subsequently, attention pooling is used to obtain the graph-level context. :

[0105]

[0106]

[0107] Graph-level context The global spatial structure of the explored region is summarized and used as input for the subsequent global alignment module. This design allows the model to simultaneously reference current observations and historical topological memory at each decision step.

[0108] Attention weights during message passing It can learn from the spatial relationships between nodes, historical access frequency, and semantic relevance to the current instruction. Compared to fixed average aggregation, attention-based aggregation can highlight nodes that are more likely to be related to the target path and reduce the interference of irrelevant historical observations on decision-making.

[0109] Graph-level pooling uses learnable attention instead of simple averaging, which can further improve the expressive power of memory summaries. For scenarios with multiple rooms and long corridors, average pooling tends to dilute information from key nodes, while attention pooling allows the model to automatically focus on forks in the road, entrances to the target room, or locations of previously seen reference points that are more relevant to the current navigation stage.

[0110] Hierarchical cross-modal alignment

[0111] Figure 3The specific structure of the hierarchical cross-modal alignment module is presented. This module divides cross-modal alignment into two levels: local vision-language alignment and global graph-language alignment. Local alignment focuses on the fine-grained semantic landing relationship between the current candidate viewpoint and linguistic lexical units, enhancing action-level vision-semantic matching; global alignment utilizes the consistency between dynamic graph memory and sentence-level instruction representation to constrain the overall path direction. Subsequently, the model adaptively integrates local and global information through a gating fusion mechanism to generate the final fused representation used for action prediction. This hierarchical design can simultaneously address short-range action decision-making and long-range path planning.

[0112] Local visual-language alignment

[0113] Local alignment focuses on the fine-grained correspondences between visible objects, directional cues, and instruction terms in the current observation. Given the current visual representation... and language lexical representation This invention computes the most relevant linguistic context to the current observation through cross-modal attention:

[0114]

[0115]

[0116] Local alignment representation Strengthening the matching relationship between phrases such as “turn left,” “pass the sofa,” and “enter the kitchen” and visual evidence helps improve the accuracy of step-by-step action selection.

[0117] The key role of local alignment is to solve the problem of implementing action-level semantics. Many terms in VLN instructions are only valid at specific time steps. For example, direction terms usually constrain the current or next action, while object terms help the agent identify reference objects in the scene. Without explicitly modeling such fine-grained correspondences, the model may incorrectly follow irrelevant objects in visually similar regions or continue moving forward when it should stop.

[0118] To improve the stability of local alignment, this invention incorporates the features of the current candidate viewpoint into the attention calculation, enabling the model to not only focus on the current position image but also compare the matching degree between different candidate directions and language lexical units. In this way, the action prediction head can obtain more direct semantic evidence, rather than relying solely on the global hidden state to indirectly infer the next action.

[0119] Global Graph - Language Alignment

[0120] Relying solely on local alignment can easily cause the agent to overemphasize currently visible cues and ignore the path goal of the entire instruction. Therefore, this invention further incorporates graph-level context... Representation of global instructions Alignment is used to constrain consistency between the explored trajectory and the complete semantic target:

[0121]

[0122] Global alignment score It can be used to measure whether the current graph memory is consistent with the target path description. When the model faces multiple similar candidate directions, this module can make a more stable choice by utilizing historical topology and complete instruction semantics.

[0123] Global alignment primarily addresses trajectory-level consistency. For longer instructions, an agent might make seemingly reasonable actions locally, but the overall path gradually deviates from the goal. For example, a door or corridor seen at a certain step might match a local phrase, but only one direction conforms to the complete path description. By aligning the dynamic graph representation with [CLS] or sentence-level instruction representation, the model can determine at a macroscopic level whether the currently explored region is still on the correct semantic path.

[0124] Furthermore, global alignment can provide auxiliary criteria for stopping actions. When the key nodes in the graph memory and the current observation together satisfy the target description, the global similarity should be high; conversely, if the model sees locally similar objects but has not yet formed a complete path consistency, it should not stop prematurely.

[0125] Gating fusion

[0126] Local alignment and global alignment correspond to short-range perception landing and long-range path constraints, respectively. This invention employs a gating mechanism to adaptively fuse the two:

[0127]

[0128]

[0129] in, This is the hierarchical aligned representation ultimately used for action prediction. Gating coefficients. The relative weights of local cues and global memory are dynamically adjusted based on the current state, so that the model relies more on local evidence when precise turning is required, and more on graph memory when long-term planning or path recovery is required.

[0130] Gating fusion avoids manually fixing the ratio of local to global information. The need for these two types of information differs across navigation phases: the initial phase typically requires understanding the overall target and initial direction; the intermediate phase requires precise selection among multiple candidate directions; and approaching the target requires combining current visual evidence and complete instructions to determine whether to stop. Learnable gating can... , and The weights of the joint states are dynamically adjusted to make the fusion representation more suitable for the current decision-making needs.

[0131] Compared to simple splicing, gating fusion also offers better interpretability. Through analysis... By observing the changes at different time steps, we can observe when the model relies more on local visual evidence and when it relies more on global graphical memory. This facilitates subsequent qualitative analysis and error diagnosis.

[0132] Motion prediction and training goals:

[0133] For the candidate action set Each candidate viewpoint The model combines its visual-positional features with its fused representation. Perform matching and calculate action scores:

[0134]

[0135] The training phase employs a strategy combining imitation learning and reinforcement learning. Imitation learning uses expert shortest path actions as supervision signals to optimize cross-entropy loss; reinforcement learning uses a reward strategy related to navigation success rate and path efficiency for optimization. The overall objective function is defined as:

[0136]

[0137] in, To imitate learning loss, To reinforce learning loss, and These are the local and global alignment auxiliary losses, respectively. , , and where represents the weighting coefficients. Through joint optimization, the model not only learns executable navigation strategies, but also learns more stable cross-modal correspondences and spatial structure memory.

[0138] The imitation learning phase allows the model to quickly approximate expert trajectories, reducing training instability caused by early exploration; the reinforcement learning phase further optimizes long-term rewards related to the final navigation metrics, such as whether the target is successfully reached and whether the path is efficient. Combining the two can balance training stability and task objective consistency.

[0139] The auxiliary alignment loss does not directly replace action supervision, but rather serves as a constraint for representation learning. Local alignment loss encourages the correct visual region or candidate orientation to be close to the corresponding language segment, while global alignment loss encourages the graph-level trajectory representation to be close to the complete instruction semantics. Through this multi-objective training, the model can learn clearer spatial-linguistic correspondences in addition to predicting actions.

[0140] Based on the above embodiments, the following experimental setup is specifically proposed:

[0141] Create a dataset

[0142] This invention evaluates the proposed method on the Room-to-Room (R2R) dataset. R2R is based on Matterport3D indoor scenes, with each path accompanied by multiple manually annotated natural language navigation instructions. Following a standard partitioning, the training set is used for model learning, ValSeen is used to test the model's fitting ability in known environments, and ValUnseen is used to evaluate the model's generalization performance in unseen environments. This invention focuses on reporting results on ValUnseen, as this partitioning better reflects the model's spatial reasoning and language understanding capabilities in new environments.

[0143] The challenge of R2R lies in the significant uncertainty between linguistic description and three-dimensional spatial structure. The same path may correspond to multiple representations, and visually similar but semantically different areas may appear in different scenarios. For example, multiple rooms may contain doors, tables, or corridor entrances, and the agent must combine historical paths and complete instructions to determine which candidate direction best matches the target. The graph memory and hierarchical alignment proposed in this invention are designed to address this difficulty of balancing long-range spatial reasoning with fine-grained semantic matching.

[0144] Evaluation indicators

[0145] The experiment used common metrics in the VLN field for evaluation, including navigation error (NE), success rate (SR), path length weighted success rate (SPL), and trajectory length (TL). Among them, NE measures the shortest path distance between the endpoint and the target location; SR represents the proportion of trajectories where the endpoint is less than 3 meters from the target; SPL considers both mission success rate and path efficiency and is the core metric for measuring navigation quality; TL is used to analyze whether the model generates excessively long or repetitive trajectories.

[0146] Among these metrics, SR primarily reflects whether the model can reach the vicinity of the target, while SPL further penalizes detours and inefficient exploration, thus better reflecting the overall quality of the navigation strategy. A lower NE indicates more accurate destination localization, while TL helps determine whether the model trades excessively long paths for higher success rates. This invention focuses on the synchronous changes of SR and SPL when analyzing the results: if SR increases but SPL decreases, it indicates that the model may have improved success rates but insufficient path efficiency; if both increase simultaneously, it indicates that the method simultaneously improves target identification and path planning.

[0147] Implementation details

[0148] This invention implements HGMA-VLN based on a recurrent cross-modal navigation framework. Visual features are extracted using a pre-trained ResNet-152 visual encoder, and panoramic observations are organized into visual inputs according to candidate viewpoints. Language commands are represented at the word level by a pre-trained BERT language encoder. Training uses the AdamW optimizer with an initial learning rate of 1×10^-5, a batch size of 8, and a maximum of 300,000 training iterations. To reduce the risk of overfitting, dropout regularization is applied to both the multimodal encoding layer and the visual feature layer.

[0149] To ensure fair comparison, all methods were tested under the same data partitioning, the same visual features, and a unified evaluation protocol. For comparison results from existing work, this invention adopted the standard R2R validation set metric reported in their public publications; for the method of this invention, the ValSeen and ValUnseen partitions were evaluated according to the same evaluation script.

[0150] During training, the model employs an optimization strategy combining imitation learning and reinforcement learning. First, supervised training is performed using teacher actions to enable the model to acquire basic instruction following ability. Then, a sampling strategy and reward function are introduced to further optimize real-world navigation metrics. Hierarchical alignment loss is also optimized during the training phase to help the model form stable language-visual and language-graph correspondences.

[0151] To improve the reproducibility of the experiment, this invention fixes the random seed and unifies the visual feature type, maximum navigation steps, candidate action construction method, learning rate scheduling strategy, and model selection criteria.

[0152] Comparative experiment

[0153] To verify the overall effectiveness of HGMA-VLN, this invention compares it with representative methods such as Seq2Seq, Speaker-Follower, EnvDrop, PREVALENT, and cyclic cross-modal navigation models. These methods cover different technical approaches, from early sequence modeling and data augmentation to environmental perturbation training and pre-trained cross-modal Transformers, and can comprehensively reflect the improvement of the proposed method compared to existing VLN models.

[0154] Among them, Seq2Seq and Speaker-Follower represent early navigation methods based on encoder-decoder and data augmentation; EnvDrop emphasizes improving generalization ability through environmental perturbations; PREVALENT represents the application of pre-trained cross-modal representations in VLN; and recurrent cross-modal navigation models represent stronger sequence decision frameworks. Comparing HGMA-VLN with these methods can answer two questions: First, is the proposed method superior to traditional sequence decision models? Second, on top of a strong cross-modal navigation framework, can explicit graph memory and hierarchical alignment still bring additional benefits?

[0155] Table 1 Comparison of navigation performance on the R2R dataset.

[0156]

[0157] Table 1 presents an overall comparison of different methods on the R2R validation set. It can be seen that HGMA-VLN achieves good overall performance on both ValSeen and ValUnseen validation partitions, especially demonstrating stronger generalization ability in the unseen environment.

[0158] Specifically, in R2RValidationUnseen, HGMA-VLN achieved a SR of 61 and a SPL of 56, higher than PREVALENT's 58 and 53, respectively; meanwhile, NE decreased from 4.71 to 4.08, indicating that the proposed graph memory and hierarchical alignment mechanism not only improved navigation success rate but also improved destination positioning accuracy. Compared to EnvDrop, HGMA-VLN improved SR and SPL by 9 and 8 percentage points, respectively. Figure 4 As can be seen more intuitively, HGMA-VLN maintains its leading position in both success rate and path efficiency, indicating that the method has a more stable generalization ability in unseen environments.

[0159] ablation experiment

[0160] To analyze the contributions of each module, this invention designed three sets of ablation experiments: First, dynamic graph memory was removed, retaining only cyclic latent state modeling; second, global graph-language alignment was removed, using only local visual-language alignment; third, local visual-language alignment was removed, using only graph-level global semantic constraints. By comparing the differences in SR and SPL between the complete model and each ablation variant, it can be determined whether spatial structural memory and hierarchical alignment have complementary effects.

[0161] All ablation experiments maintained consistent training settings except for the target module, including the number of training epochs, optimizer, visual features, random seed, and action sampling strategy. This ensures that performance variations primarily stem from the removed module, rather than differences in training configuration. For each ablation version, this invention simultaneously observes SR and SPL, as different modules may affect success rate and path efficiency, respectively.

[0162] Table 2 Module Ablation Experiment Design

[0163]

[0164] Table 2 and Figure 4 The ablation results for each module are presented. The complete model outperforms all ablation versions in both SR and SPL, indicating that graph memory and hierarchical alignment are not simply superimposed, but rather complement each other in long-range spatial reasoning and fine-grained semantic landing. In particular, the most significant decrease in SR and SPL occurs after removing local alignment, indicating that lexical-level landing has a direct impact on action selection; removing graph memory or global alignment also causes performance degradation, suggesting that both structured spatial memory and path-level semantic constraints contribute to navigation decisions.

[0165] Furthermore, the difference between w / o global alignment and w / o local alignment further illustrates the necessity of hierarchical design. Global alignment focuses more on maintaining trajectory-level semantic consistency, while local alignment more directly affects candidate action selection. Together, they enable the model to simultaneously consider long-range path planning and short-range action execution.

[0166] Training process analysis

[0167] To further observe the convergence trend during model training, this invention plotted performance variation curves on the ValUnseen partition based on training logs. Compared to a single final result, training curves can more intuitively reflect the model's stability and generalization ability at different iteration stages. Figure 5 This shows how SR and SPL change with the number of training iterations. Figure 6 The trend of navigation error NE is shown, and both together illustrate that the model gradually acquires more stable command following and endpoint localization capabilities during the training process.

[0168] from Figure 5 It can be seen that SR and SPL both rise rapidly in the early stage of training, and then gradually stabilize in the middle and late stages, indicating that the model can learn effective instruction following strategies quickly and further optimize path efficiency in subsequent training. Figure 6 The overall NE (Neural Error) trend shows a downward trend, indicating that the endpoint localization error of the model continuously decreases as training progresses. Combined with the final results in Table 1, it can be seen that the proposed method not only achieves good performance in the final metrics, but also exhibits relatively stable convergence characteristics during its training process.

[0169] Compared to cross-modal navigation methods that rely solely on recurrent latent states, the core advantage of HGMA-VLN lies in extending historical states from a single latent vector to an updatable graph-structured memory. This memory preserves connectivity between viewpoints, making path recovery and global planning easier in unseen environments. Simultaneously, the hierarchical alignment mechanism alleviates the semantic vanishing problem that easily occurs with single-layer attention, enabling the model to focus on both the complete command objective and local cues relevant to the next action within the current field of view.

[0170] In terms of error types, models relying solely on implicit historical states are more prone to problems such as premature stopping, repeated exploration, or following incorrect objects in long paths and multi-turn instructions. The proposed method constrains trajectory-level semantic consistency through global graph-language alignment and enhances action-level evidence through local vision-language alignment, thus achieving more stable performance in complex room layouts and similar visual regions.

[0171] Combination Figure 2 and Figure 3 It can be seen that dynamic graph memory and hierarchical cross-modal alignment correspond to the two levels of spatial structure modeling and semantic implementation, respectively. The experimental results are consistent with the method design: graph memory helps reduce path deviation and repeated exploration, global alignment helps maintain trajectory-level semantic consistency, and local alignment improves the matching accuracy between the current action and key instruction fragments.

[0172] Of course, HGMA-VLN may still have limitations. For example, the local alignment module may still be misled when visual features cannot accurately identify key reference objects; and the increase in irrelevant nodes in the dynamic graph may also introduce noise when the exploration path is too long. Therefore, future work could consider introducing stronger target object recognition, node importance pruning, or instruction decomposition mechanisms based on large models to further improve robustness in complex scenes.

[0173] in conclusion:

[0174] This invention presents the experimental setup, comparative results, ablation analysis, and training process visualization of HGMA-VLN. The overall experiment revolves around two core questions: first, whether dynamic graph memory can improve long-range spatial reasoning ability in unseen environments; and second, whether hierarchical cross-modal alignment can simultaneously improve local action landing and global path consistency.

[0175] Experimental results show that HGMA-VLN achieves stable improvements in both ValSeen and ValUnseen partitions. Ablation experiments further demonstrate that dynamic graph memory, global graph-language alignment, and local vision-language alignment can complement each other in terms of spatial structure modeling, trajectory-level semantic constraints, and action-level semantic implementation.

[0176] In summary:

[0177] This invention proposes HGMA-VLN (Hierarchical Graph Memory Alignment for Vision-and-Language Navigation), which integrates dynamic graph memory and hierarchical cross-modal alignment mechanism into a cyclic cross-modal decision framework. This enables historical spatial structure, complete instruction semantics, and current visual evidence to collaboratively participate in action decision-making, thereby improving navigation stability and action selection accuracy in complex environments.

[0178] Specifically, the model constructs a topology graph online during navigation, explicitly stores visited viewpoints and their connectivity, and forms structured spatial memory through graph representation learning. Building upon this, two levels of global graph-language alignment and local vision-language alignment are further designed to constrain the semantic consistency between complete instructions and historical trajectories, and the fine-grained matching between linguistic terms and current candidate observations, respectively. Experiments on the R2R benchmark dataset show that HGMA-VLN achieves a 61% success rate (SR) and a 56% path length-weighted success rate (SPL) on the ValUnseen partition, representing improvements of 3 and 3 percentage points respectively compared to PREVALENT, and reducing the navigation error (NE) from 4.71 to 4.08. Ablation experiments further demonstrate that dynamic graph memory, global graph-language alignment, and local vision-language alignment all contribute to performance improvements, with local vision-language alignment being the most critical for action selection. These results indicate that explicit spatial memory and multi-level semantic constraints can effectively improve the generalization ability and path decision quality of the VLN model in unseen environments.

[0179] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A visual language navigation method based on graph memory and hierarchical cross-modal alignment, characterized in that: Includes the following steps: S1: Multimodal feature extraction, which encodes the input natural language navigation instructions to obtain language lexical features and sentence-level global language features; Encode the panoramic visual observations collected by the agent at the current time step to obtain the visual features of each candidate reachable viewpoint; S2: Dynamic topology graph memory online update and representation learning, using visited viewpoints and candidate reachable viewpoints as graph nodes, and spatial reachability relationships between viewpoints and historical navigation trajectory relationships as graph edges, updates the dynamic topology graph online as the agent moves, and performs feature encoding and graph attention message passing on graph nodes to generate graph-level spatial memory context; S3: Hierarchical cross-modal alignment and feature fusion. Global graph-language alignment is performed based on graph-level spatial memory context and sentence-level global language features. Local visual-language alignment is performed based on candidate viewpoint visual features and language lexical features. Global alignment information and local alignment information are fused through a gating mechanism to obtain hierarchical alignment fusion features. S4: Navigation Action Prediction: Input the hierarchical alignment and fusion features into the cross-modal Transformer encoder to complete multimodal deep interaction, update the temporal navigation state in combination with the recurrent unit, match and score all candidate viewpoints based on the fusion features, and output the optimal navigation action. S5: Multi-objective joint training: The model employs imitation learning loss, reinforcement learning loss, local alignment auxiliary loss, global alignment auxiliary loss, and parameter regularization loss to jointly optimize all network parameters.

2. The visual language navigation method based on graph memory and hierarchical cross-modal alignment according to claim 1, characterized in that: S2 includes the following steps: S21: The topology graph is incrementally constructed and updated in real time as the agent moves, with the visited viewpoints and candidate unvisited viewpoints as graph nodes and the spatially reachable connectivity between viewpoints as graph edges. S22: For each graph node, fuse viewpoint visual features, position orientation encoding, and node access status encoding to generate node features; S23: Aggregate node neighborhood information through graph attention message passing to update node representations, and then generate a graph-level global spatial memory context that summarizes the entire explored environment through attention pooling.

3. The visual language navigation method based on graph memory and hierarchical cross-modal alignment according to claim 2, characterized in that: The formula for calculating node features in S22 is as follows: in, For the first Features of each viewpoint node; Viewpoint visual features; Embedded relative position and orientation of viewpoint; Encode the node status to distinguish between visited nodes and candidate unvisited nodes; , , It is a learnable linear projection matrix.

4. The visual language navigation method based on graph memory and hierarchical cross-modal alignment according to claim 3, characterized in that: In S21, when the agent reaches the current viewpoint... At that time, the incremental update formulas for the node set and edge set of the topological graph are: in, For time steps The set of nodes; For time steps The set of edges; index Indicates the previous time step; From the current viewpoint The set of first-order reachable neighbor viewpoints; Forward viewpoint Any first-order reachable neighbor viewpoint node; Only the visited nodes and their first-order candidate neighbors are retained to control the graph computation scale, and feature decay weights are applied to low-confidence candidate nodes.

5. The visual language navigation method based on graph memory and hierarchical cross-modal alignment according to claim 4, characterized in that: S23 includes the following steps: S231: Graph attention message passing, updating node representations by aggregating neighborhood node features based on inter-node attention weights, using the following formula: in, For nodes The neighborhood group, For nodes , Attention weights between them It is a non-linear activation function. , The weight matrix is ​​a learnable weight matrix; S232: Attention pooling generates a graph memory context, and a weighted summation of all updated node representations is obtained to obtain the graph-level global spatial memory context. The weighting coefficients are calculated from the learnable vector softmax.

6. The visual language navigation method based on graph memory and hierarchical cross-modal alignment according to claim 2, characterized in that: S3 includes the following steps: S31: Perform local visual-language alignment, establish fine-grained semantic matching between visual features of the current candidate viewpoint and language lexical units based on cross-modal attention mechanism, and output local alignment context; S32: Perform global graph-language alignment, calculate the semantic similarity between the graph-level global spatial memory context and the sentence-level global language features, and obtain the trajectory global semantic consistency constraint; S33: Gated adaptive fusion, dynamically generates learnable gating weights based on temporal navigation state, local alignment context, and graph-level global spatial memory context, fuses local fine-grained visual semantic information and global trajectory semantic information, and outputs hierarchical alignment fusion features.

7. The visual language navigation method based on graph memory and hierarchical cross-modal alignment according to claim 6, characterized in that: Local alignment context in S31 The calculation method is as follows: in, This represents the current panoramic visual representation; This is the lexical feature matrix; , , For learnable projection matrix; For feature dimensions; Match attention weights to visual-language lexical units; The formula for calculating the global semantic consistency score in S32 is as follows: in, For graph-level global spatial memory context; Sentence-level global language features for navigation instructions; and For learnable projection matrix; This is the function for calculating cosine similarity. The gating weights and hierarchical alignment fusion feature calculations in S33 are as follows: in, For dynamic gating coefficients; The timing navigation state output by the loop unit; For local alignment context; For graph-level global spatial memory context; For the final layered alignment and fusion features; and It is a learnable weight matrix; the gating coefficients are dynamically adjusted according to the navigation phase, taking into account the weight ratio of local visual cues and global spatial memory.

8. The visual language navigation method based on graph memory and hierarchical cross-modal alignment according to claim 1, characterized in that: In S4, for each candidate viewpoint The action score is calculated as follows: in, For layered alignment and fusion features; Candidate viewpoints Visual features; This is an element-wise multiplication operation; For learnable linear layer weights; The probability distribution of each navigation action, Indicates the first The candidate actions are scored, and the action with the highest probability is selected as the output navigation action.

9. The visual language navigation method based on graph memory and hierarchical cross-modal alignment according to claim 8, characterized in that: The total loss function of the model in S5 is: in, To mimic the learning cross-entropy loss, the expert's shortest path action is used as the supervision label; To reinforce the learning loss, the reward value is calculated based on a weighted average of navigation success rate and path length; For local alignment auxiliary loss; Auxiliary loss for global alignment; , , , Preset hyperparameter weights; This is the L2 regularization term for all network parameters of the model.

Images