A visual language navigation method based on topological map pre-training
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING UNIV OF TECH
- Filing Date
- 2026-02-12
- Publication Date
- 2026-06-09
AI Technical Summary
Existing visual language navigation models lack effective pre-training on topological maps during navigation, making it difficult for the agent to backtrack and correct after deviating from the correct path, thus affecting navigation performance.
A topology map pre-training method is adopted, which enhances the model's ability to understand topology maps through tasks such as mask language modeling, mask region classification, single-step action prediction, topology map navigation sequence prediction, topology map node location prediction, and topology map-instruction matching prediction.
It improves the agent's global decision-making ability in visual language navigation tasks, thereby increasing navigation success rate and path efficiency.
Smart Images
Figure CN122174002A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision, and its goal is to enable intelligent agents to complete navigation tasks based on natural language instructions and visual observations. Background Technology
[0002] Vision-and-Language Navigation (VLN) aims to enable agents to navigate in unfamiliar environments based on observed panoramic views and natural language commands. Specifically, in a standard VLN setup for discrete environments, the environment is represented by a navigation connectivity graph. It means that, among them This represents K navigable nodes. Let {e} represent a reachable location. Reachability between nodes. The agent needs to understand natural language instructions. It explores the environment to reach the target location, where L represents the length of the natural language instruction, i.e., the number of words contained in the instruction.
[0003] At time step t, assume the agent's position in the environment is... It obtains a panoramic view of the observation environment. ,in This represents the i-th single view, and each single view... Relative position The heading angle ai and pitch angle bi are known. If ,So It is a candidate view at time step t, also known as a candidate location. Represents a single view The associated environmental location. At each time step t, the agent bases its actions on the panoramic view. and natural language commands Make navigation decisions, that is, from Select one of the candidate views, and the position associated with the selected candidate view is the position of the agent at time step t+1.
[0004] Early VLN models took instructions and visual information observed at each location as input, selecting only candidate locations directly connected to the current location at each navigation step. This approach relies on a local action space, making it difficult for the agent to backtrack and correct itself once it deviates from the correct path. More recent VLN models construct a topology map in real-time during navigation, adding each visited node and its candidate nodes to the topology map, and using this topology map as input to the model. The topology map constructed by the navigation model at time step t is represented as... =( ), Include There are navigable nodes, among which It is the visual feature of navigable nodes, edge set express The connectivity between nodes (i.e., reachability). This type of method can not only select candidate nodes directly connected to the current node during navigation, but also backtrack to candidate nodes connected to nodes that have already been visited in the topology map, thereby realizing global action decisions and improving the performance of the agent in visual language navigation tasks.
[0005] In recent years, pre-training methods have been widely applied to visual language understanding tasks, such as image text retrieval, visual question answering, image captioning, and video captioning. Recently, visual language navigation methods based on multimodal Transformer models using a pre-training-fine-tuning paradigm have achieved good performance. These methods pre-train the model using tasks such as masked language modeling and masked region classification, and then fine-tune the pre-trained model on navigation tasks. However, current pre-training tasks applied to visual language navigation models only focus on the model's understanding of instructions and visual observations, without addressing the pre-training task for the topological map during navigation. In topological map-based navigation methods, the model's understanding of the topological map is equally crucial. Summary of the Invention
[0006] This invention proposes a visual language navigation method based on topological map pre-training, which enhances the agent's understanding of topological maps in both time and space dimensions.
[0007] A visual-language navigation method based on topology map pre-training includes the following steps: In the pre-training phase, the baseline method pre-trains its model using masked language modeling, masked region classification, and single-step action prediction tasks. The three pre-training tasks of the baseline method are implemented using MLM, MRC, and SAP prediction heads, respectively. Three new pre-training tasks are employed to pre-train the baseline network during the pre-training phase to improve the model's understanding of the topology map: topology map navigation sequence prediction, topology map node orientation prediction, and topology map-instruction matching prediction, implemented by NOP, NSP, and MIP prediction heads, respectively. In the fine-tuning phase, similar to the baseline method, the pre-trained model is transferred to navigation tasks for training.
[0008] The process includes the following steps: Step 1, Baseline Network; Step 2, Topology Map Navigation Sequence Prediction Task; Step 3, Topology Map Node Location Prediction Task; Step 4, Topology Map-Command Matching Prediction Task; Step 5, Navigation Model Pre-training; Step 6, Navigation Model Fine-tuning.
[0009] The baseline network includes a BERT word segmenter, ViT, a text encoder, a visual encoder, a node embedding module, a coarse-scale encoder, a visual embedding module, a fine-scale encoder, a global action prediction module, a local action prediction module, and a dynamic fusion module.
[0010] First, the embeddings of instruction words extracted using the frozen BERT segmenter are processed using a text encoder and a visual encoder, respectively. and panoramic view features extracted using frozen ViT Encode to obtain context word representation and contextual panoramic view features .
[0011] By using a node embedding module, stop nodes are added to the topology map, and enhanced node features are obtained by embedding the node features into their positions and navigation steps. The location embedding includes orientation and distance information, with the orientation being a four-dimensional vector. =(sinak,cosak,sinbk,cosbk), where ak and bk are the single views corresponding to the nodes. The heading and pitch angles relative to the agent's position at time step t; the distance is the distance between the node's associated position and the agent's position at time step t. This will enhance node characteristics. With context words The inputs are fed into a coarse-scale encoder for encoding. The coarse-scale encoder is a multi-layer graph-aware cross-modal Transformer structure, comprising a cross-modal attention layer, a graph-aware self-attention layer, and a feedforward layer. The cross-modal attention layer receives enhanced node features output from the node embedding module. With context words Cross-modal attention computation is performed, and cross-modal node features are obtained by passing residual connections and then performing layer normalization. The graph-aware self-attention layer receives cross-modal nodes. The graph-aware self-attention function is calculated by fusing graph structure information and then performing layer normalization after residual connection to obtain the graph-aware node features. Feedforward layer Encoding is performed using a feedforward network, followed by residual connections and layer normalization to output the final node feature representation. ,Will The data is fed into a global action prediction head consisting of two two-layer feedforward networks for action prediction.
[0012] Using the visual embedding module to analyze contextual panoramic view features By performing relative and absolute position embedding and adding a stop token to the features to represent the stop action, position-aware panoramic view features are obtained. Here, absolute position embedding refers to the orientation of the agent's position at time step t relative to its starting position. and distance The relative position embedding is the orientation of the view's associated position relative to the agent's position at time step t. and distance If the view is not a candidate view, relative position embedding is not performed. With context words The inputs are fed into a fine-scale encoder for encoding. The fine-scale encoder is a multi-layer cross-modal Transformer structure, containing a cross-modal attention layer, a self-attention layer, and a feedforward layer. The cross-modal attention layer receives position-aware panoramic view features output from the visual embedding module. With context words Cross-modal attention computation is performed, and the cross-modal panoramic view features are obtained by passing the residual connections and then performing layer normalization. Self-attention layer Self-attention calculation is performed, and then residual connections are used to obtain the global perception panoramic view features through layer normalization. Feedforward layer The final fine-scale panoramic view feature representation is obtained by encoding using a feedforward network, passing residual connections, and then performing layer normalization. ,Will The data is fed into a local action prediction head consisting of two two-layer feedforward networks for action prediction.
[0013] The dynamic fusion module is used to fuse the prediction results of two actions and output the final action predicted by the model.
[0014] Topology map at time step t The construction process is as follows: 1) In Add the current node and its edges with the nodes visited by the agent at time step (t-1). The visual features of the current node are based on the panoramic view features at time step t, and the average pooling of the context panoramic view features output by the visual encoder is used; 2) In The process adds candidate nodes reachable from the current node, as well as edges between the new candidate nodes and the current node. The visual features of the new candidate nodes are based on the panoramic view features at time step t, and the single-view features corresponding to the candidate nodes are encoded in the context panoramic view features output by the visual encoder.
[0015] Baseline pre-training task;
[0016] Masked language modeling task: The goal is to predict word embeddings. The masked First, using a training sample from the dataset, the frozen BERT tokenizer is used to process natural language instructions. Encoding yields word embeddings With a certain probability In Perform masking to obtain the masked word embeddings. Then, the masked words are embedded. Expert demonstration path based on a total time step of T Constructed topology map as well as Panoramic view features at time step T Both are input into the model. The masking language modeling task requires the model to adjust the masked representation based on the input. The model performs predictions. It averages the output features of the fine-scale encoder and the coarse-scale encoder, and then inputs the averaged features into an MLM prediction head consisting of two fully connected layers to predict the masked image. Optimization is achieved by minimizing the negative log-likelihood loss.
[0017] 2) Masked Region Classification Task: The task objective is to predict the semantic label of a masked single view. For a training sample in the dataset, an expert demonstration path with a total time step count of T is used. Panoramic view features at time step T For each single-view feature Randomly select whether to set the value to zero, i.e., whether to mask the corresponding view, to obtain the masked panoramic view features. Then, combine the masked panoramic view features with word embeddings. The common input is the model shown in the dashed box below. The masked region classification task requires the model to be based on... The task involves predicting the semantic labels of a masked single view based on the features of the masked panoramic view. Two fully connected layers are used as the MRC prediction head.
[0018] 3) Single-step action prediction task: The task requires the model to predict a given expert demonstration path. In the case of embedding according to instruction words Demonstration path by some experts Infer the next action. Embed words. ,based on Constructed topology map And panoramic view features at time step t The input model, representing an auction house, uses two SAP prediction heads, each consisting of two fully connected layers, to generate action prediction scores based on the output features of the coarse-scale encoder and the fine-scale encoder, respectively. A dynamic fusion module, also consisting of two fully connected layers, then fuses these scores to obtain the final predicted action.
[0019] Topology map navigation sequence prediction task;
[0020] The goal of this task is to enable the model to recover the expert demonstration path based on a total of T time steps. Constructed topology map The order in which nodes on a shuffled navigation path are visited. (This refers to the topology map.) Nodes in Divided into two subsets, Includes nodes that have been visited up to time step T. Includes nodes that have not been visited up to time step T, i.e. and First, The visited nodes are arranged in the order they were visited to obtain sequence S. Then, the nodes in the sequence are reordered with a certain probability to obtain a new sequence. Visited nodes in The order in the map is defined as the order in which the elements are visited, and a topology map is built based on the new visiting order. The navigation path in the text. The nodes in the data are arranged according to their relationship with... In the middle node The connection relationship in it is added to When embedding nodes, The navigation steps for each visited node are embedded as zero. The instructions... Word embedding The input is fed into a text encoder to obtain the context word representation. Then, the node embedding module will be used based on The resulting enhanced node features and context word representations The input is fed into a coarse-scale encoder, and the encoded features are then fed into a NOP prediction head composed of multilayer perceptron (MLP) layers to predict the visit order of each visited node in the topology graph. This pre-training task is optimized by minimizing the negative log-likelihood.
[0021] Topology map node location prediction task;
[0022] The goal of this task is to enable the model to predict node locations, thereby improving the model's spatial awareness. This is based on an expert demonstration path with a total time step count of T. Build a topology map Utilizing the node embedding module for computation Augmented node features of the middle node. When calculating augmented node features, randomly select... In a node, set its direction Set as zero vector, distance Set to zero for masking. Combine the enhanced node features output by the node embedding module with the context word representation. The features output from the coarse-scale encoder are then fed into the NSP prediction head, which is composed of multiple perceptron (MLP) layers, to predict the orientation of node k. and distance Regarding direction The prediction is optimized using cosine similarity loss. and The weighted sum is used as the total loss for the topology map node orientation prediction task NSP. .
[0023] Topology map - instruction matching and prediction task;
[0024] The goal of this task is to enable the model to predict whether the topology map matches the instructions, thereby improving the model's ability to perceive the temporal and spatial information in the topology map. This is achieved by using a frozen BERT tokenizer to process natural language instructions. Encoded word embeddings The input is fed into a text encoder to obtain the context word representation. Simultaneously, the node embedding module is used based on the topology map. Calculate the enhanced node features and combine them with the context word representations The features are fed into a coarse-scale encoder, and the encoded features are then fed into a multi-layer perceptron (MLP) prediction head to predict the topology map. and instructions Whether it matches. This includes the topology map input to the coarse-scale encoder. There is a certain probability and instruction Mismatched topology map There is also a certain probability that it is related to the instruction. Matching topology map . It is an instruction that is randomly selected from the training dataset. The mismatched topology map is optimized using binary cross-entropy loss in this pre-training task.
[0025] Navigation model pre-training;
[0026] During the pre-training phase, the model is pre-trained using the aforementioned six pre-training tasks based on the ground truth paths labeled in the dataset. Among these, the masked language modeling task, masked region classification task, and single-step action prediction task are pre-training tasks of the baseline method; the topology map navigation sequence prediction task, topology map node orientation prediction task, and topology map-instruction matching prediction task are proposed new pre-training tasks. To balance the learning of different tasks, the six pre-training tasks participate in pre-training randomly and alternately with equal probability. The pre-training data consists of two parts: 1) the training set from the R2R dataset; 2) augmented data obtained by generating navigation instructions for the shortest path trajectory using the Speaker model in the training environment. Both parts contain a total of 6582k image-instruction-action triples, where the images represent expert demonstration paths. The panoramic view above, the command is a complete navigation command, and the action is to follow the expert demonstration path. Expert decision-making is implemented step by step in the middle.
[0027] Navigation model fine-tuning;
[0028] During the fine-tuning phase, the model is trained using the training set from the R2R dataset. In this phase, the model is based on the expert demonstration path. Perform single-step action prediction while simultaneously predicting the path within the complete navigation episode: at each time step t, the agent predicts the path based on the input instructions. Panoramic view of time step t and dynamically updated topology maps The system decides and outputs the action to be performed at time step t until the navigation task is completed or the maximum number of navigation steps is reached. The fine-tuning process optimizes the model based on path prediction and single-step action prediction tasks. Supervision for the path prediction task comes from a pseudo-interactive demonstrator. At each time step t, based on the environment's navigation connectivity graph... , calculate The distance between the positions associated with all candidate nodes and the target position is used as the pseudo-supervisory action. . Attached Figure Description
[0029] Figure 1This is a framework diagram of the method. MLM, MRC, and SAP are prediction heads used for existing masked language modeling tasks, masked region classification tasks, and single-step action prediction tasks, respectively. NOP, NSP, and MIP are prediction heads used for the topology map navigation sequence prediction task, topology map node location prediction task, and topology map-instruction matching prediction task proposed in this invention, respectively. In the topology map, black nodes are visited nodes, white nodes are the current node, and gray nodes are unvisited navigable nodes.
[0030] Figure 2 It is a coarse-scale encoder.
[0031] Figure 3 It is a fine-scale encoder.
[0032] Figure 4 It is a topology map navigation sequence prediction task ( The nodes A, B, C, and D are visited in the order of 1, 2, 3, and 4, respectively. The order in which nodes A, B, C, and D are visited is 1, 3, 2, and 4, respectively.
[0033] Figure 5 Topology map - instruction matching prediction task. Detailed Implementation
[0034] This invention proposes a visual language navigation method based on topological map pre-training, which enhances the agent's understanding of topological maps in both time and space dimensions.
[0035] This invention uses DUET, which follows a pre-training-fine-tuning paradigm, as the baseline method and proposes a visual language navigation method based on topological map pre-training.
[0036] During the pre-training phase, the baseline method pre-trains its network model using masked language modeling, masked region classification, and single-step action prediction tasks. The three pre-training tasks of the baseline method utilize... Figure 1 (a) shows the implementation of the MLM prediction head, MRC prediction head, and SAP prediction head. Based on this, this invention proposes three new pre-training tasks to pre-train the baseline network during the pre-training phase, thereby improving the model's understanding of the topology map: the topology map navigation order prediction task, the topology map node location prediction task, and the topology map-instruction matching prediction task, which are respectively implemented by… Figure 1 The NOP, NSP, and MIP prediction heads are implemented in (a). During the fine-tuning phase, the method of this invention is the same as the baseline method, transferring the pre-trained model to a navigation task for training.
[0037] This invention first introduces the baseline network, then describes the three pre-training tasks proposed in this invention, and finally explains how to use the pre-training tasks to pre-train the navigation model of the baseline network and fine-tune the navigation model after pre-training. This invention includes the following: 1. Baseline network; 2. Topology map navigation order prediction task; 3. Topology map node location prediction task; 4. Topology map-instruction matching prediction task; 5. Navigation model pre-training; 6. Navigation model fine-tuning.
[0038] The specific descriptions of each item are as follows:
[0039] Baseline network
[0040] Figure 1 (b) shows the fine-tuning stage of the network, which is the baseline network structure, including BERT word segmenter, ViT, text encoder, visual encoder, node embedding module, coarse-scale encoder, visual embedding module, fine-scale encoder, global action prediction module, local action prediction module and dynamic fusion module.
[0041] This invention first introduces the baseline method flow, and then introduces the pre-training task used by the baseline network.
[0042] Baseline Methodology Flow
[0043] First, the text encoder and visual encoder are used to embed the instruction words extracted using the frozen BERT segmenter. and panoramic view features extracted using frozen ViT Encode to obtain context word representation and contextual panoramic view features .
[0044] By using a node embedding module, stopping nodes (all-zero vectors) are added to the topology map, and enhanced node features are obtained by embedding the node features into their positions and navigation steps. The location embedding includes orientation and distance information, with the orientation being a four-dimensional vector. =(sinak,cosak,sinbk,cosbk), where ak and bk are the single views corresponding to the nodes. The heading and pitch angles relative to the agent's position at time step t; the distance is the distance between the node's associated position and the agent's position at time step t. The unit is meters. For visited nodes, the navigation step embedding is the time step of its most recent visit; for unvisited nodes, the navigation step embedding is zero. This will enhance node features. With context words The common inputs are fed into a coarse-scale encoder for encoding. The coarse-scale encoder is a multi-layer graph-aware cross-modal Transformer structure. For example... Figure 2 As shown, the coarse-scale encoder comprises a cross-modal attention layer, a graph-aware self-attention layer, and a feedforward layer. The cross-modal attention layer receives enhanced node features output by the node embedding module. With context words Cross-modal attention computation is performed, and cross-modal node features are obtained by passing residual connections and then performing layer normalization. The graph-aware self-attention layer receives cross-modal nodes. The graph-aware self-attention function is calculated by fusing graph structure information and then performing layer normalization after residual connection to obtain the graph-aware node features. Feedforward layer Encoding is performed using a feedforward network, followed by residual connections and layer normalization to output the final node feature representation. Then The data is fed into a global action prediction head consisting of two two-layer feedforward networks for action prediction.
[0045] Simultaneously, the visual embedding module is used to analyze the features of the contextual panoramic view. Perform relative and absolute position embeddings and add stop tokens (all-zero vectors) to the features to represent the stop action to obtain position-aware panoramic view features. Here, absolute position embedding refers to the orientation of the agent's position at time step t relative to its starting position. and distance The relative position embedding is the orientation of the view's associated position relative to the agent's position at time step t. and distance If the view is not a candidate view, relative position embedding is not performed. With context words The inputs are fed into the fine-scale encoder for encoding. The fine-scale encoder is a multi-layer, cross-modal Transformer structure, such as... Figure 3 As shown, it includes a cross-modal attention layer, a self-attention layer, and a feedforward layer. The cross-modal attention layer receives position-aware panoramic view features output from the visual embedding module. With context words Cross-modal attention computation is performed, and the cross-modal panoramic view features are obtained by passing the residual connections and then performing layer normalization. Self-attention layer Self-attention calculation is performed, and then residual connections are used to obtain the global perception panoramic view features through layer normalization. Feedforward layer The final fine-scale panoramic view feature representation is obtained by encoding using a feedforward network, passing residual connections, and then performing layer normalization. Then The data is fed into a local action prediction head consisting of two two-layer feedforward networks for action prediction.
[0046] Then, the dynamic fusion module is used to fuse the prediction results of the two actions and output the final action predicted by the model.
[0047] Topology map at time step t The construction process is as follows: 1) In Add the current node and its edges with the nodes visited by the agent at time step (t-1). The visual features of the current node are based on the panoramic view features at time step t, and the average pooling of the context panoramic view features output by the visual encoder is used; 2) In The process adds candidate nodes reachable from the current node, as well as edges between the new candidate nodes and the current node. The visual features of the new candidate nodes are based on the panoramic view features at time step t, and the single-view features corresponding to the candidate nodes are encoded in the context panoramic view features output by the visual encoder.
[0048] Baseline pre-training task
[0049] Masked language modeling task: The goal is to predict word embeddings. The masked First, using a training sample from the dataset, the frozen BERT tokenizer is used to process natural language instructions. Encoding yields word embeddings With a certain probability In Perform masking to obtain the masked word embeddings. Then, the masked words are embedded. Expert demonstration path based on a total time step of T Constructed topology map as well as Panoramic view features at time step T Common input Figure 1 (a) The model shown in the dashed box at the bottom. The masking language modeling task requires the model to adapt the masked data based on the input. The model performs predictions. It averages the output features of the fine-scale encoder and the coarse-scale encoder, and then inputs the averaged features into an MLM prediction head consisting of two fully connected layers to predict the masked image. This task is optimized by minimizing the negative log-likelihood loss, with the objective function as follows:
[0050] (1)
[0051] Among them, the expert demonstration path It is the truth path labeled in the dataset, which contains a panoramic view of each node on the truth path.
[0052] 2) Masked Region Classification Task: The task objective is to predict the semantic label of a masked single view. For a training sample in the dataset, an expert demonstration path with a total time step count of T is used. Panoramic view features at time step T For each single-view feature Randomly select whether to set the value to zero (i.e., whether to mask the corresponding view) to obtain the masked panoramic view features. Then, combine the masked panoramic view features with word embeddings. Common input Figure 1 (a) The model shown in the dashed box at the bottom. The masked region classification task requires the model to be based on... The task involves predicting the semantic labels of a masked single view based on the features of the masked panoramic view. Two fully connected layers are used as the MRC prediction head, with the loss function being:
[0053] (2)
[0054] in, The semantic probability distribution of the masked single view predicted by the pre-trained ViT-B / 16. The semantic probability distribution of the masked single view predicted by the model. This represents the number of semantic categories.
[0055] 3) Single-step action prediction task: The task requires the model to predict a given expert demonstration path. In the case of embedding according to instruction words Demonstration path by some experts Infer the next action. Embed words. ,based on Constructed topology map And panoramic view features at time step t enter Figure 1 (a) The model shown in the lower dashed box uses two SAP prediction heads, each consisting of two fully connected layers, to generate action prediction scores based on the output features of the coarse-scale encoder and the fine-scale encoder, respectively. A dynamic fusion module, also consisting of two fully connected layers, then fuses the scores to obtain the final predicted action. The loss function for this task is:
[0056] (3)
[0057] in, The truth action for time step t.
[0058] Topology map navigation sequence prediction task;
[0059] The goal of this task is to enable the model to recover the expert demonstration path based on a total of T time steps. Constructed topology map The order in which nodes on a shuffled navigation path are visited is determined to improve the model's ability to perceive temporal information. This is achieved by using a topology map. Nodes in Divided into two subsets, Includes nodes that have been visited up to time step T. Includes nodes that have not been visited up to time step T, i.e. and First, The visited nodes are arranged in the order they were visited to obtain sequence S. Then, the nodes in the sequence are reordered with a certain probability to obtain a new sequence. Visited nodes in The order in the map is defined as the order in which the elements are visited, and a topology map is built based on the new visiting order. The navigation path in the text. The nodes in the data are arranged according to their relationship with... In the middle node The connection relationship in it is added to When embedding nodes, The navigation steps for each visited node are embedded as zero. For example... Figure 4 As shown, the present invention will instruct Word embedding The input is fed into a text encoder to obtain the context word representation. Then, the node embedding module will be used based on The resulting enhanced node features and context word representations The input is fed into a coarse-scale encoder, and the encoded features are then fed into a NOP prediction head composed of multilayer perceptron (MLP) layers to predict the visit order of each visited node in the topology graph. This pre-training task is optimized by minimizing the negative log-likelihood.
[0060] (4)
[0061] Wherein, P( ) indicates the model prediction Visited nodes The probability of the actual order in which they are accessed. Represents a node The actual order in which they were accessed, | | indicates the cardinality of the set.
[0062] Topology map node location prediction task;
[0063] The goal of this task is to enable the model to predict node locations, thereby improving the model's spatial awareness. This is based on an expert demonstration path with a total time step count of T. Build a topology map Utilizing the node embedding module for computation Augmented node features of the middle node. When calculating augmented node features, randomly select... Given a node (e.g., node k), set its direction. Set as zero vector, distance Set to zero for masking. Combine the enhanced node features output by the node embedding module with the context word representation. The features output from the coarse-scale encoder are then fed into the NSP prediction head, which consists of multilayer perceptron (MLP) layers, to predict the orientation of node k. and distance Regarding direction The prediction in this invention uses cosine similarity loss for optimization:
[0064] (5)
[0065] in, Let L2 norm be represented. For distance... The prediction in this invention is optimized using L1 loss:
[0066] (6)
[0067] in, This represents the L1 norm. Then, the present invention uses... and The weighted sum is used as the total loss for the topology map node orientation prediction task (NSP). :
[0068] (7)
[0069] in, and This is a hyperparameter.
[0070] Topology map - instruction matching and prediction task;
[0071] The goal of this task is to enable the model to predict whether the topology map matches the instructions, thereby further improving the model's ability to perceive the temporal and spatial information in the topology map. For example... Figure 5 As shown, this invention will utilize a frozen BERT word segmenter for natural language instructions. Encoded word embeddings The input is fed into a text encoder to obtain the context word representation. Simultaneously, the node embedding module is used based on the topology map. Calculate the enhanced node features and combine them with the context word representations The features are fed into a coarse-scale encoder, and the encoded features are then fed into a MIP prediction head composed of multilayer perceptron (MLP) layers to predict the topology map. and instructions Whether it matches. This includes the topology map input to the coarse-scale encoder. There is a certain probability that it is related to the instruction. Mismatched topology map There is also a certain probability that it is related to the instruction. Matching topology map . It can be any instruction chosen from the training dataset. Mismatched topology maps. This pre-training task is optimized using binary cross-entropy loss:
[0072] (8)
[0073] Where y represents the truth value, that is, if Is with instructions Matching topology map If y=1, then y=1; otherwise, y=0. P( ) indicates model prediction With instructions The probability of a match.
[0074] Navigation model pre-training;
[0075] In the pre-training phase, this invention pre-trains the model using the aforementioned six pre-training tasks based on ground truth paths annotated in the dataset. Among these, the masked language modeling task, masked region classification task, and single-step action prediction task are pre-training tasks of the baseline method; the topology map navigation sequence prediction task, topology map node location prediction task, and topology map-instruction matching prediction task are new pre-training tasks proposed in this invention. To balance the learning of different tasks, the six pre-training tasks participate in pre-training randomly and alternately with equal probability. The pre-training data consists of two parts: 1) the training set from the R2R dataset; 2) augmented data obtained by generating navigation instructions for the shortest path trajectory using the Speaker model in the training environment. The two parts together contain 6582k image-instruction-action triples, where the images are expert demonstration paths. The panoramic view above, the command is a complete navigation command, and the action is to follow the expert demonstration path. Expert decision-making is implemented step by step in the middle.
[0076] Navigation model fine-tuning;
[0077] During the fine-tuning phase, this invention trains the model using the training set from the R2R dataset. In this phase, the model is based on the expert demonstration path. Perform single-step action prediction while simultaneously predicting the path within the complete navigation episode: at each time step t, the agent predicts the path based on the input instructions. Panoramic view of time step t and dynamically updated topology maps The system decides and outputs the action to be performed at time step t until the navigation task is completed or the maximum number of navigation steps is reached. The fine-tuning process optimizes the model based on path prediction and single-step action prediction tasks. Supervision for the path prediction task comes from a pseudo-interactive demonstrator. At each time step t, based on the environment's navigation connectivity graph... , calculate The distance between the positions associated with all candidate nodes and the target position is used as the pseudo-supervisory action. The loss function is:
[0078] (9);
[0079] in, This represents the path the model has traversed up to time step t-1. Single-step action prediction loss. The calculation remains consistent with the pre-training phase, fine-tuning the total loss. for:
[0080] (10);
[0081] in, This is a hyperparameter.
[0082] This invention uses the R2R dataset for training and evaluating the model. The R2R dataset is a dataset with fine-grained instructions, containing 7189 shortest path trajectories, each trajectory associated with 3 instructions of average length 32 words. The average length of the expert demonstration path is 6 steps. The dataset is divided into a training set, a visible validation set, an invisible validation set, and an invisible test set, containing 61, 56, 11, and 18 indoor scenes, respectively. The evaluation metrics for the experiment include: trajectory length (TL), the average path length of the agent navigation in meters; navigation error (NE), the average distance between the agent's stopping position and the target position in meters; success rate (SR), the proportion of the agent's stopping position within 3 meters of the target position; and path length-weighted success rate (SPL).
[0083] Experimental setup: For pre-training, this invention uses one NVIDIA RTX A6000 GPU, sets the batch size to 64, and iterates 200,000 times; for fine-tuning, this invention uses one NVIDIA RTX A6000 GPU to train the model, sets the batch size to 8, and iterates 200,000 times.
[0084] This invention achieves significant improvements on multiple evaluation metrics of the R2R dataset, fully demonstrating the effectiveness of the proposed model.
[0085] As shown in Table 1, all the methods presented in the table utilize a multimodal Transformer based on a pre-training-fine-tuning paradigm. Among them, methods such as "GELA," "CASP," "LSAL," "VLN-SIG," and "HOP" focus on designing pre-training tasks. The "GELA" method proposes three pre-training tasks: entity phrase prediction, landmark bounding box prediction, and entity-landmark semantic alignment, explicitly supervising the learning of fine-grained cross-modal alignment between entity phrases and environmental landmarks. The "CASP" method proposes two tasks: mask fragment modeling under trajectory conditions and contrastive semantic alignment modeling, encouraging agents to learn cross-modal dependencies and cross-modal semantically consistent representations. The "LSAL" method improves upon the "CASP" method by introducing action prediction based on mask observations and hierarchical triple contrastive learning. "VLN-SIG" proposes mask panorama modeling, mask trajectory modeling, and image-based action prediction, enabling agents to generate semantics for future navigation views. The "HOP" method proposes history-based action prediction, trajectory sequence modeling, and group sequence modeling tasks. The trajectory sequence modeling task differs from the topology map navigation sequence prediction task proposed in this invention. This task is based on visual observation rather than a topology map, i.e., on an expert-demonstrated path. The candidate views selected in each step are shuffled, and then the model is allowed to predict the correct order.
[0086] As shown in Table 1, the method of this invention exhibits the best performance in terms of success rate. Furthermore, DUET* represents the experimental results of the replicated baseline method. Compared to the baseline method, the success rate and path length-weighted success rate of the proposed method are improved by 3 percentage points and 2 percentage points, respectively, on the invisible validation set; on the invisible test set, both metrics are improved by 3 percentage points and 1 percentage point, respectively. This demonstrates that the pre-training task proposed in this invention can further improve the navigation performance of the visual-language navigation model.
[0087] Table 1. Performance comparison of the proposed method with other methods on the R2R dataset.
[0088]
Claims
1. A visual-language navigation method based on topological map pre-training, characterized in that: The process includes the following steps: In the pre-training phase, the baseline method pre-trains the baseline network model using masked language modeling, masked region classification, and single-step action prediction tasks. These three pre-training tasks utilize MLM, MRC, and SAP prediction heads, respectively. Three new pre-training tasks are then used to pre-train the baseline network during the pre-training phase to improve the model's understanding of the topology map: topology map navigation order prediction, topology map node orientation prediction, and topology map-instruction matching prediction, implemented by NOP, NSP, and MIP prediction heads, respectively. In the fine-tuning phase, similar to the baseline method, the pre-trained model is transferred to navigation tasks for training. This includes the baseline network, topology map navigation order prediction, topology map node orientation prediction, topology map-instruction matching prediction, navigation model pre-training, and navigation model fine-tuning.
2. The visual language navigation method based on topological map pre-training according to claim 1, characterized in that: The baseline network includes a BERT word segmenter, ViT, a text encoder, a visual encoder, a node embedding module, a coarse-scale encoder, a visual embedding module, a fine-scale encoder, a global action prediction module, a local action prediction module, and a dynamic fusion module. First, the embeddings of instruction words extracted using the frozen BERT segmenter are processed using a text encoder and a visual encoder, respectively. and panoramic view features extracted using frozen ViT Encode to obtain context word representation and contextual panoramic view features ; By using a node embedding module, stop nodes are added to the topology map, and enhanced node features are obtained by embedding the node features into their positions and navigation steps. The location embedding includes direction and distance information, with the direction being a four-dimensional vector. =(sinak,cosak,sinbk,cosbk), where ak and bk are the single views corresponding to the nodes. The heading and pitch angles relative to the agent's position at time step t; the distance is the distance between the node's associated position and the agent's position at time step t. This will enhance node features. With context words The common inputs are fed into the coarse-scale encoder for encoding; the coarse-scale encoder is a multi-layer graph-aware cross-modal Transformer structure, which includes a cross-modal attention layer, a graph-aware self-attention layer, and a feedforward layer; the cross-modal attention layer receives the enhanced node features output by the node embedding module. With context words Cross-modal attention computation is performed, and cross-modal node features are obtained by passing residual connections and then performing layer normalization. The graph-aware self-attention layer receives cross-modal nodes. The graph-aware self-attention function is calculated by fusing graph structure information and then performing layer normalization after residual connection to obtain the graph-aware node features. ; Feedforward layer Encoding is performed using a feedforward network, followed by residual connections and layer normalization to output the final node feature representation. ,Will The data is fed into a global action prediction head consisting of two two-layer feedforward networks for action prediction. Using the visual embedding module to analyze contextual panoramic view features By performing relative and absolute position embedding and adding a stop token to the features to represent the stop action, position-aware panoramic view features are obtained. Wherein, absolute position embedding refers to the direction of the agent's position at time step t relative to the starting position. and distance The relative position embedding is the orientation of the position associated with the view relative to the agent's position at time step t. and distance If the view is not a candidate view, then relative position embedding is not performed; With context words The inputs are fed into a fine-scale encoder for encoding; the fine-scale encoder is a multi-layer cross-modal Transformer structure, which includes a cross-modal attention layer, a self-attention layer, and a feedforward layer; the cross-modal attention layer receives the position-aware panoramic view features output from the visual embedding module. With context words Cross-modal attention computation is performed, and the cross-modal panoramic view features are obtained by passing the residual connections and then performing layer normalization. Self-attention layer Self-attention calculation is performed, and then residual connections are established before layer normalization to obtain the global perception panoramic view features. ; Feedforward layer The final fine-scale panoramic view feature representation is obtained by encoding using a feedforward network, passing residual connections, and then performing layer normalization. ,Will The local action prediction head, consisting of two two-layer feedforward networks, is fed into the system for action prediction. The dynamic fusion module is used to fuse the prediction results of two actions and output the final action predicted by the model. Topology map at time step t The construction process is as follows: 1) In Add the current node and its edges with the nodes visited by the agent at time step (t-1). The visual features of the current node are based on the panoramic view features at time step t, and the average pooling of the context panoramic view features output by the visual encoder is used; 2) In The process adds candidate nodes reachable from the current node, as well as edges between the new candidate nodes and the current node. The visual features of the new candidate nodes are based on the panoramic view features at time step t. The single-view features corresponding to the candidate nodes are encoded in the context panoramic view features output by the visual encoder.
3. The visual language navigation method based on topological map pre-training according to claim 1, characterized in that: Baseline pre-training task; Masked language modeling task: The goal is to predict word embeddings. The masked First, using a training sample in the dataset, the frozen BERT tokenizer is used to process natural language instructions. Encoding yields word embeddings With a certain probability In Perform masking to obtain the masked word embeddings. ; Then, the masked words are embedded. Expert demonstration path based on a total time step of T Constructed topology map as well as Panoramic view features at time step T The common inputs are fed into the model; the masking language modeling task requires the model to perform masking based on the inputs. The model performs predictions; it averages the output features of the fine-scale encoder and the coarse-scale encoder, and then inputs the averaged features into an MLM prediction head consisting of two fully connected layers to predict the masked image. Optimization is achieved by minimizing the negative log-likelihood loss. 2) Masked Region Classification Task: The task objective is to predict the semantic label of a masked single view; for a training sample in the dataset, an expert demonstration path with a total time step of T is classified. Panoramic view features at time step T For each single-view feature Randomly select whether to set to zero, i.e. whether to mask the corresponding view, to obtain the panoramic view features after masking; Subsequently, the masked panoramic view features and word embeddings are... The common input is the model shown in the dashed box below; the masked region classification task requires the model to be based on... The task uses two fully connected layers as the MRC prediction head to predict the semantic labels of the masked single view based on the panoramic view features after masking. 3) Single-step action prediction task: The task requires the model to predict a given expert demonstration path. In the case of embedding according to instruction words Demonstration path by some experts Inferring the next action; embedding words ,based on Constructed topology map and panoramic view features at time step t The input model is an auction house. It uses two SAP prediction heads, each consisting of two fully connected layers, to generate action prediction scores based on the output features of the coarse-scale encoder and the fine-scale encoder, respectively. The scores are then fused using a dynamic fusion module consisting of two fully connected layers to obtain the final predicted action.
4. The visual language navigation method based on topological map pre-training according to claim 1, characterized in that: Topology map navigation sequence prediction task; The goal of this task is to enable the model to recover the expert demonstration path based on a total of T time steps. Constructed topology map The order in which nodes on a navigation path are visited in a scrambled manner; the topology map. Nodes in Divided into two subsets, Includes nodes that have been visited up to time step T. Includes nodes that have not been visited up to time step T, i.e. and First, The visited nodes are arranged in the order they were visited to obtain sequence S. Then, the nodes in the sequence are reordered with a certain probability to obtain a new sequence. ; Visited nodes in The order in the map is defined as the order in which the elements are visited, and a topology map is built based on the new visiting order. Navigation path in; for The nodes in the data are arranged according to their relationship with... In the middle node The connection relationship in it is added to When embedding nodes, The navigation steps for each visited node are embedded as zero; the instructions are... Word embedding The input is fed into a text encoder to obtain the context word representation. Then, the node embedding module will be used based on The resulting enhanced node features and context word representations The input is fed into a coarse-scale encoder, and the encoded features are fed into a NOP prediction head composed of multilayer perceptron (MLP) layers to predict the visiting order of each visited node in the topology graph; this pre-training task is optimized by minimizing the negative log-likelihood.
5. The visual language navigation method based on topological map pre-training according to claim 1, characterized in that: Topology map node location prediction task; The goal of this task is to enable the model to predict node locations to improve its spatial awareness; based on an expert demonstration path with a total time step count of T. Build a topology map Utilizing the node embedding module for computation Enhanced node features in the middle node; When calculating the enhanced node features, randomly select In a node, set its direction Set as zero vector, distance Zeroing is used for masking; the enhanced node features output by the node embedding module are combined with the context word representation. The features are fed into a coarse-scale encoder for encoding; the features output by the coarse-scale encoder are then fed into an NSP prediction head composed of a multilayer perceptron (MLP) layer to predict the orientation of node k. and distance Regarding direction The prediction is optimized using cosine similarity loss; and The weighted sum is used as the total loss for the topology map node orientation prediction task NSP. ; Topology map - instruction matching and prediction task; The goal of this task is to enable the model to predict whether the topology map matches the instructions, thereby improving the model's ability to perceive the temporal and spatial information in the topology map; and to utilize a frozen BERT word segmenter to process natural language instructions. Encoded word embeddings The input is fed into a text encoder to obtain the context word representation. Simultaneously, the node embedding module is used based on the topology map. Calculate the enhanced node features and combine them with the context word representations The features are fed into a coarse-scale encoder, and the encoded features are then fed into a multi-layer perceptron (MLP) prediction head to predict the topology map. and instructions Does it match? The topology map input to the coarse-scale encoder is among them. There is a certain probability and instruction Mismatched topology map There is also a certain probability that it is related to the instruction. Matching topology map ; It is an instruction that is randomly selected from the training dataset. For mismatched topology maps, this pre-training task uses binary cross-entropy loss for optimization.
6. The visual language navigation method based on topological map pre-training according to claim 1, characterized in that: Navigation model pre-training; During the pre-training phase, the model is pre-trained using the aforementioned six pre-training tasks based on the ground truth paths labeled in the dataset. Among these, the masked language modeling task, masked region classification task, and single-step action prediction task are pre-training tasks of the baseline method; the topology map navigation sequence prediction task, topology map node orientation prediction task, and topology map-instruction matching prediction task are proposed new pre-training tasks. To balance the learning of different tasks, the six pre-training tasks participate in pre-training randomly and alternately with equal probability. The pre-training data consists of two parts: 1) the training set from the R2R dataset; 2) augmented data obtained by generating navigation instructions for the shortest path trajectory using the Speaker model in the training environment. These two parts together contain 6582k image-instruction-action triples, where the images represent expert demonstration paths. The panoramic view above, the command is a complete navigation command, and the action is to follow the expert demonstration path. Expert decision-making is implemented step by step in the middle.
7. The visual language navigation method based on topological map pre-training according to claim 1, characterized in that: Navigation model fine-tuning; During the fine-tuning phase, the model is trained using the training set from the R2R dataset; During the fine-tuning phase, the model is based on the expert demonstration path. Perform single-step action prediction while simultaneously predicting the path within the complete navigation episode: at each time step t, the agent predicts the path based on the input instructions. Panoramic view of time step t and dynamically updated topology maps The system decides and outputs the action to be performed at time step t until the navigation task is completed or the maximum number of navigation steps is reached. The fine-tuning process optimizes the model based on the path prediction task and the single-step action prediction task. The supervision for the path prediction task comes from the PseudoInteractiveDemonstrator. At each time step t, based on the navigation connectivity graph of the environment... , calculate The distance between the positions associated with all candidate nodes and the target position is used as the pseudo-supervisory action. .