Waypoint prediction method fusing instruction landmark features in visual language navigation

Abstract

The invention discloses a waypoint prediction method fusing instruction landmark features in visual language navigation, and belongs to the crossing field of artificial intelligence and computer vision. The method is realized through the following steps: firstly, extracting landmarks and co-occurrence landmarks thereof from a natural language instruction by utilizing a large language model, and generating a candidate landmark sequence; then, the occurrence probability of the candidate landmarks in actual observation is calculated based on a visual large model CLIP, and noise interference is dynamically corrected and suppressed through a learnable co-occurrence scoring module; secondly, fusing the corrected landmark features with the depth visual features, inputting a double-layer Transform model to model a spatial relationship, generating a waypoint probability heat map, and outputting adjacent waypoints through non-maximum suppression (NMS); and finally, topological mapping is carried out on the predicted waypoints, an optimal path is obtained through cross-modal path planning, and an intelligent agent is guided to execute low-level actions. The method does not need to depend on a predefined environment map or manually annotate data, navigation errors in a continuous environment are remarkably reduced, the generalization ability of unseen scenes is improved, and the method is suitable for open world navigation scenes such as robot navigation and automatic driving.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of embodied artificial intelligence and computer vision, and relates to waypoint prediction technology for performing visual language navigation tasks in continuous environments. In particular, it relates to a method for embedding landmark semantic features in natural language instructions into waypoint probability distribution to reduce navigation errors and improve adaptability to unseen scenes. Background Technology

[0002] As a key task of embodied intelligence, Virtual Language Navigation (VLN) has been a research hotspot for researchers both domestically and internationally since its inception. It requires agents to navigate through complex and unknown visual environments based on natural language instructions, ultimately reaching a target location. The core challenge lies in achieving accurate cross-modal alignment, meaning the agent needs to dynamically correlate the spatial semantics implicit in textual instructions with real-time visual observations and make sequential action decisions based on this. In the early stages of research, to simplify the task, researchers reduced the VLN task to traversing a predefined graph of the environment, which significantly narrowed down the possible locations of the agent and the target. However, discrete environments rely on predefined connected graphs, and agents execute efficient high-level actions through view selection and waypoint transfer. This dependence limits its generalization ability in real continuous spaces. To make the VLN task more realistic, researchers proposed a VLN task in continuous environments, which no longer relies on predefined connected graphs, allowing the agent to move freely with low-level actions.

[0003] While continuous environments are closer to reality, they require agents to perform low-level actions, leading to long decision sequences, training difficulties, and limitations in directly utilizing model architectures that have achieved significant success in discrete environments. Agents with the same architecture exhibit a significant success rate gap between discrete and continuous environments. To bridge this gap, researchers proposed using a waypoint predictor to dynamically generate virtual waypoints in continuous environments, constructing a local navigation map. This predictor is trained using a reconstructed Habitat-MP3D connected graph, outputting a set of reachable locations. This enables agents originally designed for discrete environments and reliant on high-level actions to operate efficiently in continuous environments, effectively reducing the discrete-continuous gap.

[0004] Traditional waypoint prediction methods often rely solely on intuitive visual features, typically using depth reachability as the primary basis for prediction. Since the predicted waypoints are not directly related to mission instructions, the prediction results may deviate from the mission objectives. Furthermore, traditional VLN models are mainly trained on limited manually labeled navigation data, lacking robustness to unseen scenes and objects. Landmarks mentioned in the instructions may be invisible, occluded, or have altered shapes in the current view, requiring the agent to reason using open-world common sense. Therefore, there is an urgent need for a waypoint prediction method that integrates instruction landmark features and is adaptable to unseen scenes. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing waypoint prediction techniques and provide a waypoint prediction method that can integrate co-occurring landmark features from instructions and adapt to unseen scenes in continuous visual-language navigation tasks. This method considers both visual depth features and landmark features from task instructions. It extracts landmarks and co-occurring landmarks from instructions based on a large language model. Then, it calculates the highest-scoring landmark based on a comparison between the large visual model and actual observations, using it as input features. The depth features and landmark features are fused and input into a two-layer Transformer to model spatial relationships, generating a waypoint heatmap and predicting waypoint positions within the agent's neighborhood.

[0006] To achieve the above objectives, the present invention provides a waypoint prediction method in visual language navigation, comprising the following steps:

[0007] Step 1: Receive natural language instructions, extract landmark words and their corresponding K co-occurring landmarks from the instructions using a large language model, and generate a candidate landmark sequence. ,in As a command landmark, To create a shared landmark;

[0008] Step 2: Acquire the currently observed RGB and depth panoramic images, and extract RGB features using two ResNet50 networks. and depth features ;

[0009] Step 3: Calculate the probability of candidate landmarks appearing in the currently observed image using a large visual model, and dynamically correct the probability using a learnable co-occurrence scoring module to obtain the corrected landmark features. The co-occurrence scoring module includes a linear layer, a ReLU activation function, layer normalization, and a Dropout operation.

[0010] Step 4: With corrected landmark features Fusion characteristics are obtained after fusion. The system inputs a Transformer network containing a multi-layer self-attention mechanism, models spatial relationships and generates a waypoint probability heatmap, performs non-maximum suppression (NMS) on the waypoint probability heatmap, and selects the k nearest waypoints with the highest probability as navigation targets.

[0011] Step 5: Perform topological mapping on the predicted waypoints and encode the nodes to obtain node features. Input the node features and command features into the cross-modal planning module to obtain the node navigation probability.

[0012] Step 6: Through The path planning algorithm obtains the optimal topological path and guides the agent to perform low-level actions such as moving forward, rotating, and stopping.

[0013] Step 7: Repeat the above steps until the agent enters the target threshold range or reaches the maximum number of steps.

[0014] Furthermore, in step 1, the large language model is Deepseek or a pre-trained model with similar parameter scale, and its landmark extraction is based on the grammatical dependency relationship between noun phrases and co-occurring words in the instruction.

[0015] Furthermore, in step 3, the large visual model is a CLIP model, and CLIP is used to obtain the observation features at each time step. Obtained through CLIP text encoder Adjacent landmark text features The formula for calculating the probability of a landmark appearing is:

[0016]

[0017] Where sim() is the cosine similarity. For temperature parameters and .

[0018] Furthermore, in step 3, the co-occurrence scoring module corrects the probability through the following steps:

[0019] (1) The command text features and the observed image features are concatenated and then input into a linear layer to output state features. ;

[0020] (2) Based on Calculate co-occurring landmark scores ,in The function is a sigmoid function, where W and b are learnable parameters;

[0021] (3) The corrected landmark prediction probability is obtained by weighted summation with the initial probability.

[0022] Furthermore, in step 4, the Transformer network contains 3 encoding layers, each containing 12 self-attention heads, and the input dimension of the fused feature FM is 1024.

[0023] Furthermore, in step 4, the NMS suppression threshold is 0.7, and the output k waypoints satisfy:

[0024] (1) The Euclidean distance between waypoints is greater than the minimum spacing. rice;

[0025] (2) The waypoint is located within the currently observed passable area, and its depth does not exceed the preset maximum depth. rice.

[0026] Furthermore, the above method runs on edge computing devices with a single inference latency of less than 50 milliseconds, and does not rely on predefined environment maps or additional manually labeled data.

[0027] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0028] A visual language navigation system includes a processor, a memory, and a computer-readable storage medium as described in claim 8, wherein the processor is configured to:

[0029] (1) Acquire real-time observation images using an RGBD camera;

[0030] (2) When executing the program on the storage medium, implement the above method to generate waypoints;

[0031] (3) Perform topology mapping and cross-modal path planning, and control the mobile chassis to navigate to the target location according to the waypoint sequence.

[0032] The effects and benefits of this invention are as follows: When an agent performs VLN tasks in a continuous environment without prior knowledge, the method utilizes a large model to provide rich open-world landmark co-occurrence knowledge, obtaining landmark features and their co-occurrence features with navigation guidance significance contained in the task instructions. Furthermore, the constraints of actual observation reduce the noise impact of prior knowledge from the large model. The introduction of the large model eliminates the dependence on labeled data, improving the generalization ability of the method. This invention does not rely on predefined environment maps or manually labeled data, significantly reducing navigation errors in continuous environments and improving generalization ability for unseen scenes. It is applicable to open-world navigation scenarios such as robotics and autonomous driving. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of a waypoint prediction process that integrates command landmark features according to an embodiment of the present invention.

[0034] Figure 2 This is a system architecture diagram according to an embodiment of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings.

[0036] A waypoint prediction method that integrates instruction landmark features in a continuous environment task of visual language navigation includes the following steps:

[0037] When an agent receives a task instruction in a continuous environment without any prior knowledge and needs to reach a destination according to the instruction, the agent first analyzes the instruction using a large language model and extracts the landmark information contained in the instruction, and obtains K co-occurring landmarks U for each landmark. Since not all of the instruction landmarks and co-occurring landmarks obtained by the agent can actually appear in observations, they cannot be directly used for feature extraction. Therefore, a learnable co-occurrence scoring module is introduced to perform correctable landmark discovery based on a large visual model, resulting in landmark features with real guiding significance. ;

[0038] Deep features were processed using two ResNet50 networks. and landmark features Encode and fuse the enhanced features of the fusion. The input is fed into a two-layer Transformer network to model spatial relationships between views. The classifier is a multilayer perceptron that projects the Transformer output onto the probabilities of neighboring waypoints in space. Non-maximum suppression (NMS) is applied to the generated heatmap to obtain the k nearest waypoints. See also Figure 1 The following are the specific operational steps of a waypoint prediction method that integrates command landmark features in a continuous visual language navigation task according to the present invention:

[0039] Step 1: For a given instruction I, we first use the Big Oracle model to map I. Extraction and corresponding K co-occurring landmarks Generate, obtain landmark sequence First, the CLIP image encoder is used to obtain the observed features at each time step. Obtained through CLIP text encoder Adjacent landmark text features Calculate the current landmark as The probability of:

[0040]

[0041] Where exp() is the exponential function and sim() is the similarity function. To adjust the parameters. Similarly, we can obtain... .like Then it is effectively marked as Otherwise it is .

[0042] Step 2: Based on the valid targets determined in Step 1 and their co-occurrence CLIP is used to calculate each single-view observation Appeared in The probability of:

[0043]

[0044] Similarly, calculate the probability of each co-occurring landmark appearing in actual observations. A learnable co-occurrence scoring module is introduced to dynamically adjust co-occurrence importance based on actual observations, suppressing noise introduced by landmark shifting and discovery. Specifically, this is achieved by introducing an El() function containing a linear layer, ReLU activation function, layer normalization function, and Dropout operation to process command features and observation features to obtain state features. The score for co-occurring landmarks is:

[0045]

[0046] Similarly, we can obtain Finally, the corrected landmark prediction probability was obtained through the above two scores:

[0047]

[0048] Through the above-mentioned landmark movement, landmark discovery, and co-occurrence scoring modules, correctable landmark discovery is performed to obtain corrected landmark features. .

[0049] Step 3: For any point in the open space, acquire its depth panoramic image, and obtain the depth feature after encoding by a visual encoder. Landmark features and depth features are passed through a two-layer nonlinear Transformer containing 12 self-attention heads per layer to obtain fused features:

[0050]

[0051] Step 4: The Multilayer Perceptron (MLP) projects the Transformer output as the probability distribution of neighboring waypoints and selects the top k neighboring waypoints through Non-Maximum Suppression (NMS).

[0052] The NMS suppression threshold is 0.7, and the output k waypoints satisfy:

[0053] (1) The Euclidean distance between waypoints is greater than the minimum spacing. rice;

[0054] (2) The waypoint is located within the currently observed passable area, and its depth does not exceed the preset maximum depth. rice.

[0055] Step 5: Perform topological mapping on the predicted waypoints and encode the nodes to obtain node features. Input the node features and command features into the cross-modal planning module to obtain the node navigation probability.

[0056] Step 6: Through The path planning algorithm obtains the optimal topological path and guides the agent to perform low-level actions such as moving forward, rotating, and stopping.

[0057] Step 7: Repeat the above steps until the agent enters the target threshold range or reaches the maximum number of steps.

[0058] The above method runs on edge computing devices, with a single inference latency of less than 50 milliseconds, and does not rely on predefined environment maps or additional manually labeled data.

[0059] In some specific embodiments, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above method.

[0060] In some embodiments, a visual language navigation system is provided, the system including a processor, a memory, and the aforementioned computer-readable storage medium, wherein the processor is configured to:

[0061] (1) Acquire real-time observation images through an RGBD camera.

[0062] (2) Generate waypoints by performing the above method.

[0063] (3) Topology mapping and cross-modal path planning are performed to control the mobile chassis to navigate to the target location according to the waypoint sequence.

[0064] This embodiment constructs a dataset containing command primary landmarks and co-occurring landmarks based on R2R-CE and RxR-CE. As shown in Table 1, navigation performance is analyzed using five evaluation metrics: TL (trajectory length), NE (error distance), SR (navigation success rate), OSR (ideal success rate), and SPL (path-weighted success rate). The results show that this method improves SPL by 3%, SR by 1%, and TL and NE by a significant decrease. This indicates that by introducing command landmark enhancement features, this method optimizes the waypoint predictor network structure, improves waypoint prediction quality, and thus improves navigation performance.

[0065] Table 1. Comparison experiments with existing methods on the R2R-CE dataset.

[0066]

[0067] Table 2 Ablation Experiment of Command Landmark Feature Module

[0068]

[0069] Table 2 further describes the contribution of command landmark features to waypoint prediction quality and navigation performance, and analyzes waypoint prediction quality using four evaluation indicators: (Difference between the number of target waypoints and predicted waypoints), %Open (Ratio of waypoints on non-obstacle surfaces in open space), (The average error between the target waypoint set and the predicted waypoint set) (Maximum error between the target waypoint set and the predicted waypoint set). Using two classic navigation methods, CWP-CMA and ETPNav, as benchmarks, the impact of commanded landmark features on their performance indicators is compared. The results show that adding commanded landmark features to the benchmark methods improves... The value was effectively reduced by 0.18, which indicates that the predicted waypoints are closer to the mission objective after adding the command landmark features, effectively improving the correlation with the target waypoints and thus improving the quality of waypoint prediction.

[0070] The above embodiments are only used to illustrate the present invention. Any equivalent transformations and improvements made on the basis of the technical solutions of the present invention should not be excluded from the protection scope of the present invention.

Claims

1. A waypoint prediction method in visual language navigation, characterized in that, Includes the following steps: Step 1: Receive natural language instruction I, extract landmark words and their corresponding K co-occurring landmarks from the instruction using a large language model, and generate a candidate landmark sequence. ,in As a command landmark, To create a shared landmark; Step 2: Acquire the currently observed RGB and depth panoramic images, and extract RGB features using two ResNet50 networks. and depth features ; Step 3: Calculate the probability of candidate landmarks appearing in the currently observed image using a large visual model, and dynamically correct the probability using a learnable co-occurrence scoring module to obtain the corrected landmark features. The co-occurrence scoring module includes a linear layer, a ReLU activation function, layer normalization, and a Dropout operation. Step 4: With corrected landmark features Fusion characteristics are obtained after fusion. The system inputs a Transformer network containing a multi-layer self-attention mechanism, models spatial relationships and generates a waypoint probability heatmap, performs non-maximum suppression (NMS) on the waypoint probability heatmap, and selects the k nearest waypoints with the highest probability as navigation targets. Step 5: Perform topological mapping on the predicted waypoints and encode the nodes to obtain node features. Input the node features and command features into the cross-modal planning module to obtain the node navigation probability. Step 6: Through The path planning algorithm obtains the optimal topological path and guides the agent to perform low-level actions such as moving forward, rotating, and stopping. Step 7: Repeat the above steps until the agent enters the target threshold range or reaches the maximum number of steps.

2. The method according to claim 1, characterized in that, In step 1, the large language model is Deepseek or a pre-trained model with similar parameter size, and its landmark extraction is based on the grammatical dependency relationship between noun phrases and co-occurring words in the instructions.

3. The method according to claim 1, characterized in that, In step 3, the large visual model is a CLIP model, and CLIP is used to obtain the observation features at each time step. Obtained through CLIP text encoder Adjacent landmark text features The formula for calculating the probability of a landmark appearing is: ； in sim() is the cosine similarity. For temperature parameters and .

4. The method according to claim 1, characterized in that, In step 3, the co-occurrence scoring module corrects the probability through the following steps: (1) The command text features and the observed image features are concatenated and then input into a linear layer to output state features. ; (2) Based on Calculate co-occurring landmark scores ,in The function is a sigmoid function, where W and b are learnable parameters; (3) The corrected landmark prediction probability is obtained by weighted summation with the initial probability.

5. The method according to claim 1, characterized in that, In step 4, the Transformer network contains two coding layers, each with 12 self-attention heads, and fuses features. The input dimension is 1024.

6. The method according to claim 1, characterized in that, In step 4, the NMS suppression threshold is 0.7, and the output k waypoints satisfy: (1) The Euclidean distance between waypoints is greater than the minimum spacing. rice; (2) The waypoint is located within the currently observed passable area, and its depth does not exceed the preset maximum depth. rice.

7. The method according to claim 1, characterized in that, The method operates on edge computing devices, with a single inference latency of less than 50 milliseconds, and does not rely on predefined environment maps or additional manually labeled data.

8. A computer-readable storage medium, characterized in that: The storage medium shown stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-7.

9. A visual language navigation system, characterized in that: The system includes a processor, a memory, and a computer-readable storage medium as described in claim 8, wherein the processor is configured to: (1) Acquire real-time observation images using an RGBD camera; (2) When executing a program on a storage medium, the method described in any one of claims 1-7 is implemented to generate waypoints; (3) Control the mobile chassis to navigate to the target location according to the waypoint sequence.

Images