A visual language navigation method and system based on dynamic grid map

Abstract

This paper proposes a visual language navigation method and system based on a dynamic grid map, belonging to the field of navigation technology. The method comprises: extracting visual language features from RGB panoramic images using the CLIP model; combining the visual language features and depth features of the depth image with the absolute coordinates of the grid cells to construct a grid map; updating the grid map based on the semantic relevance between the grid cells and navigation instructions; obtaining cross-modal interaction features corresponding to the grid map based on a cascaded attention mechanism, and analyzing them using the Mamba module to output predicted waypoints; using DD-PPO as a local strategy, performing action analysis with the predicted waypoints as input to generate a probability distribution; and selecting and executing navigation actions based on the probability distribution. The method can quickly respond to dynamic and continuous environmental changes while avoiding long-range dependencies, thereby achieving precise visual language navigation for intelligent agents.

Description

Technical Field

[0001] The present invention belongs to the field of navigation technology, and in particular relates to a visual language navigation method and system based on a dynamic grid map. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] With the rapid development of artificial intelligence and robotics, enabling intelligent agents to understand natural language commands and autonomously navigate complex 3D environments—a key research topic and challenge in the field of embodied AI—has become a hot topic and a core challenge. VLN requires not only powerful visual perception and natural language understanding capabilities, but also effective cross-modal information fusion, environmental memory, path planning, and decision-making.

[0004] However, despite significant progress in existing visual-language navigation methods, there are still some technical challenges in existing visual-language navigation methods for intelligent agents in continuous environments, such as:

[0005] (1) Existing environmental maps mainly include topological maps and fixed semantic maps. Among them, although topological maps can provide global path guidance, they lack environmental details (such as object geometry and semantic categories), making them unable to support fine-grained navigation decisions; while fixed semantic maps are based on fixed semantic labels and are difficult to generalize to complex scenes containing unseen objects. Therefore, existing methods cannot construct a map representation that can accurately represent environmental geometry and semantic information and dynamically adapt to environmental changes according to navigation instructions.

[0006] (2) Existing methods have difficulty effectively handling temporal dependencies and dynamic environmental changes in long-distance navigation tasks. For example, Transformer-based models (such as BEBVert) rely on the self-attention mechanism, and the computational complexity increases quadratically with the sequence length, resulting in reduced real-time performance in long instruction scenarios, making it difficult to adapt to continuous dynamic environmental changes.

[0007] (3) Some existing methods also rely on large language models (such as MC-GPT and NavGPT) for planning, but lack deep integration with visual features, resulting in a disconnect between semantic understanding and environmental perception. Summary of the Invention

[0008] In order to overcome the shortcomings of the above-mentioned existing technologies, the present invention provides a visual language navigation method and system based on dynamic grid maps, which can quickly respond to dynamic and continuous environmental changes while avoiding long-distance dependence, so as to achieve accurate visual language navigation for intelligent agents.

[0009] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0010] A first aspect of the present invention provides a visual language navigation method based on a dynamic grid map.

[0011] A visual language navigation method based on a dynamic grid map, comprising:

[0012] Get the current RGB panoramic image and depth image;

[0013] The CLIP model is used to extract visual language features from the RGB panoramic image, and the obtained visual language features and depth features of the depth image are combined with the absolute coordinates of the grid cells to construct a grid map. The grid map is updated based on the semantic relevance between the grid cells and the current navigation instructions.

[0014] The updated grid map features and the text features of the current navigation instructions are processed based on the cascaded attention mechanism to obtain cross-modal interaction features. The obtained cross-modal interaction features are analyzed using the Mamba module to output predicted waypoints.

[0015] DD-PPO is adopted as the local strategy, and the predicted waypoints are used as input for action analysis to generate a probability distribution; navigation actions are selected and executed according to the obtained probability distribution.

[0016] Furthermore, the visual language features and depth features of the depth image are combined with the absolute coordinates of the grid cells to construct a grid map, including: using the visual language features extracted from the RGB panoramic image by the CLIP model as grid features, adjusting the size of the corresponding depth features to match the grid features; and calculating the absolute coordinates based on the grid features and depth features.

[0017] Furthermore, the visual language features and the depth features of the depth image are combined with the absolute coordinates of the grid cells to construct a grid map, which also includes: storing the grid features and the absolute coordinates in the grid cells to achieve the fusion of semantic information and geometric information to construct the grid map.

[0018] Furthermore, the grid map is updated based on the semantic relevance between the grid unit and the current navigation instruction, including: performing cosine similarity matching between the current navigation instruction and the feature vector under the grid unit to obtain an attention weight; and performing weighted fusion on the map features of the grid map based on the obtained attention weight to update the grid map.

[0019] Furthermore, the feature vector under the grid unit is: the feature vector observed at the same position and different viewing angles under a set time step.

[0020] Furthermore, the Mamba module is used to analyze the cross-modal interaction features to output predicted waypoints, including: inputting the obtained cross-modal interaction features into the Mamba module, processing long sequence dependencies through a selective state space mechanism to output temporal enhancement features; and generating predicted waypoints based on the obtained temporal enhancement features through decoding at the fully connected layer.

[0021] Furthermore, DD-PPO is adopted as a local strategy, and action analysis is performed with predicted waypoints as input to generate a probability distribution, including: inputting the coordinates of the obtained predicted waypoints as sub-goals into a distributed proximal policy optimization module, and generating a probability distribution of discrete navigation actions based on a policy generation network within the distributed proximal policy optimization module.

[0022] A second aspect of the present invention provides a visual language navigation system based on a dynamic grid map.

[0023] A visual language navigation system based on a dynamic grid map, comprising:

[0024] The image acquisition module is configured to: obtain the RGB panoramic image and depth image at the current moment;

[0025] a map construction module configured to: extract visual language features from the RGB panoramic image using a CLIP model, combine the obtained visual language features with depth features from the depth image and the absolute coordinates of the grid cells to construct a grid map; and update the grid map based on the semantic relevance between the grid cells and the current navigation instructions;

[0026] The waypoint prediction module is configured to: process the map features of the updated grid map and the text features of the current navigation instructions based on the cascaded attention mechanism to obtain cross-modal interaction features; use the Mamba module to analyze the obtained cross-modal interaction features to output predicted waypoints;

[0027] The action decision module is configured to: adopt DD-PPO as a local strategy, perform action analysis with the obtained predicted waypoints as input to generate a probability distribution; and select and execute navigation actions according to the obtained probability distribution.

[0028] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of a visual language navigation method based on a dynamic grid map as described in the first aspect of the present invention.

[0029] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and runnable on the processor. When the processor executes the program, it implements the steps in a visual language navigation method based on a dynamic grid map as described in the first aspect of the present invention.

[0030] One or more of the above technical solutions have the following beneficial effects:

[0031] (1) The present invention uses the CLIP model to extract the visual language features of the RGB panoramic image, and combines the obtained visual language features and the depth features of the depth image with the absolute coordinates of the grid unit to construct a grid map. Through this dynamic visual language grid map (DGVL-Map) construction method, the CLIP model is used to extract the visual language features of the open vocabulary and the spatial geometric information of the depth image is integrated, so that each grid unit can store semantic features and three-dimensional coordinates at the same time, so as to solve the problem of lack of details in the topological map and insufficient generalization of the fixed semantic map. In addition, the present invention realizes real-time dynamic update of the map by calculating the semantic relevance (attention weight) between the grid unit and the navigation instruction, which can give priority to strengthening the regional features related to the task, so that the map representation can be dynamically adjusted according to the instruction requirements, thereby significantly improving the precision of the environment representation and the task adaptability. As a result, the present invention can construct a map representation that can not only accurately represent the environment geometry and semantic information, but also dynamically adapt to environmental changes according to the navigation instruction.

[0032] (2) The present invention processes grid map features and text features of navigation instructions based on a cascaded attention mechanism to obtain cross-modal interaction features; the Mamba module is then used to analyze the cross-modal interaction features to derive predicted waypoints. By introducing the instruction-based cascaded attention mechanism and the Mamba model, and utilizing Mamba's selective state space module to achieve long sequence modeling with linear time complexity, the problem of computational complexity increasing with the square of sequence length due to the self-attention mechanism in the existing method architecture can be avoided. This mechanism dynamically aggregates historical observations and instruction features through incremental state updates, which can capture long-distance dependencies while ensuring real-time navigation efficiency. It can effectively solve the problem of reduced real-time performance in long instruction scenarios, so as to better adapt to continuous dynamic environmental changes.

[0033] (3) The cascaded attention mechanism designed in this paper uses the instruction features extracted by BERT as key-value pairs and the visual language features of the grid map as queries, and achieves cross-modal deep fusion with the Mamba model through multi-head attention. This mechanism quantifies the semantic relevance of instructions and visual features in a unified embedding space, guiding the model to focus on task-related environmental areas. This can solve the problem of disconnection between large language model planning and visual perception, and achieve precise alignment of semantic understanding and environmental features, thereby improving the accuracy of waypoint prediction and the robustness of navigation decisions.

[0034] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0036] Figure 1 This is a flowchart of a visual language navigation method based on a dynamic grid map in Example 1 of the present invention.

[0037] Figure 2 This is an architectural diagram of a visual language navigation system based on a dynamic grid map in Example 2 of the present invention.

[0038] Figure 3 This is a structural diagram of the cross-attention Mamba network in Example 2 of the present invention. DETAILED DESCRIPTION

[0039] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0040] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0041] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0042] The overall concept proposed by this invention is as follows: This invention provides a visual language navigation method based on a dynamic grid map. First, a pre-trained CLIP model is used to extract visual language features from RGB panoramic images and fuse the corresponding depth information to construct a grid map containing rich semantic and geometric information. Subsequently, a real-time dynamic update method for the grid map is designed. By calculating the semantic relevance (attention weight) between grid cells and navigation instruction features, areas relevant to the navigation task are updated preferentially, achieving effective weighted fusion of map features. Regarding waypoint prediction, to efficiently extract key features from a grid map containing a large amount of potentially irrelevant information, an innovative instruction-based cascaded attention mechanism is introduced. Combined with the powerful long-sequence processing capabilities of the Mamba model, map features highly relevant to instructions are aggregated to output accurate waypoint predictions. Finally, the DD-PPO strategy is used for action decision-making to generate navigation actions based on predicted waypoints.

[0043] Example 1

[0044] This embodiment discloses a visual language navigation method based on a dynamic grid map.

[0045] like Figure 1 As shown, a visual language navigation method based on a dynamic grid map includes:

[0046] Step S1: Obtain the RGB panoramic image and depth image at the current moment;

[0047] Step S2: extracting visual language features from the RGB panoramic image using the CLIP model, and combining the obtained visual language features with the depth features of the depth image and the absolute coordinates of the grid cells to construct a grid map; updating the grid map based on the semantic relevance between the grid cells and the current navigation instruction;

[0048] Step S3: processing the map features of the updated grid map and the text features of the current navigation instructions based on the cascaded attention mechanism to obtain cross-modal interaction features; analyzing the obtained cross-modal interaction features using the Mamba module to output predicted waypoints;

[0049] Step S4: Using DD-PPO as the local strategy, performing action analysis with the predicted waypoints as input to generate a probability distribution; selecting and executing a navigation action based on the obtained probability distribution.

[0050] Based on the above method, the present invention can quickly respond to dynamic and continuous environmental changes while avoiding long-range dependencies, thereby achieving precise visual language navigation for intelligent agents. To facilitate understanding of the technical solution of the present invention, the specific implementation method of the technical solution of the present invention is further explained and illustrated below.

[0051] In step S1, the RGB panoramic image and depth image at the current moment are obtained.

[0052] The agent is equipped with an RGB camera in a navigation environment (such as a Habitat simulator) to capture panoramic visual information of the current node, namely the RGB panoramic image. Simultaneously, it is also equipped with a depth camera to synchronously capture depth information of the current node. Furthermore, each single-view depth image captured by the depth camera corresponds one-to-one with the perspective of the RGB image.

[0053] While existing approaches have made significant progress in VLNs for continuous environments, several key challenges remain, particularly in areas such as the agent's understanding of the environment and the improvement of multimodal fusion techniques. This paper addresses the design of navigation environment representation and navigation planning. By dynamically constructing a grid map that fuses visual features and command information, combined with a Mamba model-based waypoint prediction mechanism, this paper aims to improve the agent's understanding of complex scenes, long-term memory, and the efficiency and robustness of task execution in continuous environments. Specifically:

[0054] In step S2, the CLIP model is used to extract visual language features from the RGB panoramic image. The obtained visual language features and the depth features of the depth image are combined with the absolute coordinates of the grid cells to construct a grid map. The grid map is updated based on the semantic relevance between the grid cells and the current navigation instructions. This can be achieved by the following methods:

[0055] Step S2-1: Grid map construction.

[0056] The visual language features extracted from the RGB panoramic image by the CLIP model are used as grid features, and the corresponding depth features are resized to match the grid features; then, the absolute coordinates are calculated based on the grid features and the depth features. Specifically, the RGB panoramic image is recorded as:

[0057] ;

[0058] in, represents an RGB panoramic image, Represents the agent at time step Observed panoramic images; Indicates the component number of the RGB panoramic image.

[0059] For the obtained RGB panoramic image, the pre-trained CLIP-ViT-B / 32 model is used to extract the visual language features of the RGB panoramic image, namely the grid features:

[0060] ;

[0061] in, Represents the grid features of the RGB panoramic image, Represents the time step Next grid features of each viewpoint or region, represents the set of real numbers, represents the height dimension, represents the width dimension, Represents the channel dimension. The grid feature of the hth row and wth column is recorded as .

[0062] The corresponding depth image is downsampled to the same scale as the RGB panoramic image and expressed as , to ensure that the depth value is aligned with the RGB extracted grid features in the spatial dimension; where, Represents the depth value corresponding to the resized depth image. Correspondingly, the depth value (depth feature) of the hth row and wth column after resizing is recorded as For convenience, all indexes Recorded as , The value range of is 1 to 1, and Therefore, the grid features Change to , the depth feature after resizing Change to On this basis, the grid characteristics can be calculated The absolute coordinates of ,Right now:

[0063] ;

[0064] in, Represents the current position of the agent and the grid features The top angle between Representing Agent and Grid Features The Euclidean distance between and are the horizontal and vertical coordinates of the agent in the two-dimensional space, respectively, and are used to represent its position in the global coordinate system (i.e., the current coordinates of the agent).

[0065] After that, all grid features and their absolute coordinates are stored in the grid cells to achieve the fusion of semantic and geometric features to construct the grid map, namely:

[0066] ;

[0067] in, Represents the time step The grid memory stores the visual features extracted by the agent during navigation and their corresponding absolute coordinates.

[0068] Based on the above steps, this paper designs a dynamic grid visual language map. Using the CLIP model, this approach achieves semantic understanding of an open vocabulary, rather than being limited to predefined labels. Furthermore, the visual language semantics extracted by the CLIP model and the spatial geometry extracted from depth images provide a more fine-grained representation of the environment for the agent. This eliminates the existing reliance on predefined labels and enhances generalization across location environments.

[0069] Step S2-2: Update the grid map according to the semantic relevance between the grid cells and the current navigation instruction.

[0070] The grid map is updated based on a real-time dynamic update algorithm, namely:

[0071] ;

[0072] in, Indicates the number of features accumulated in the current grid cell, the variable Represents the global feature at the hth row and wth column of the grid map; Indicates the time step The feature vectors observed at the same position but different viewing angles are: Represents the attention weight. In order to quantify the semantic relevance between the grid unit and the navigation instruction, the feature vector and navigation command features Match by cosine similarity in the unified embedding space to obtain attention weights , used to guide the weighted fusion and update of map features, namely:

[0073] ;

[0074] This update mechanism prioritizes regions relevant to the navigation task. Visual instructions are combined with observations from new viewpoints to enhance task alignment. By averaging features within each grid cell, it effectively fuses multi-view observations of the same object while maintaining focus on task-relevant regions.

[0075] Based on the above steps, this invention integrates the visual language map with the grid map. This allows for dynamic and focused updates of map content based on instruction information during navigation, effectively avoiding the limitations of existing methods that rely on static map construction. Furthermore, by dynamically updating and integrating map grid features based on the current instruction's attention weight, task relevance can be more directly incorporated into the map representation.

[0076] In step S3, the map features of the updated grid map and the text features of the current navigation instructions are processed based on the cascaded attention mechanism to obtain cross-modal interaction features; and the obtained cross-modal interaction features are analyzed using the Mamba module to output predicted waypoints.

[0077] The constructed grid map will be gradually updated as the agent gradually visits all environments. In order to understand the navigation progress, the present invention designs an instruction positioning module to predict waypoints in combination with the visual information of the current time step. Due to the complexity of the navigation environment, each grid cell will store a large number of grid features, but most of these features may be irrelevant to the navigation task. Therefore, the agent needs key grid information that is highly relevant to the navigation instructions to understand the environment. In addition, the agent needs to complete planning predictions quickly. Therefore, the present invention proposes an instruction-based cascade attention mechanism, which uses the Mamba model with lower time complexity to aggregate key features from the grid to output predicted waypoints. Specifically, the text information of the navigation instructions and the encoded information of the image are processed respectively using a text encoder and a visual encoder, that is:

[0078] A. Text Encoder: Using BERT model for navigation instructions Extract text features and use Indicates the The encoding features of the words.

[0079] B. Visual Encoder: Use a pre-trained convolutional neural network (ConvNet) to encode the observed RGB-D information to obtain RGB features and deep features 。

[0080] Use the text features T from the text encoder as the key matrix The sum value matrix V, and the grid map features The Mamba attention mechanism then computes the most relevant map features relative to the current instruction. Map Features It is still represented as a matrix of shape H × W × D, capturing the spatial and target context related to the task. The detailed process is as follows:

[0081] First, the grid features and text features are processed through the multi-headed attention mechanism MHA (Multi-Headed Attention). MHA uses multiple "attention heads" to parallelly calculate the feature associations of different subspaces, allowing the model to simultaneously focus on the semantic information of different positions in the input sequence, thereby improving the interaction efficiency of cross-modal features. For the input query, key, and value, the attention weight is calculated by dot product. Specifically, the grid map feature is used as the query. , instruction features as keys Sum value V, thus obtaining the intermediate representation ,Right now:

[0082] ;

[0083] in, Represents the combined operation of residual connection (Add) and layer normalization (Norm), that is, the residual connection inputs the query It is added to the output of MHA to solve the gradient degradation problem in deep network training; layer normalization standardizes the feature dimension to stabilize the training process. Represents a multi-head attention mechanism, by computing the query and key The semantic similarity of the map is used to generate the map feature weight that is most relevant to the instruction, and the weighted aggregation value is calculated based on the weight. information.

[0084] Then, through FFN For processing, FFN (Feed Forward Network) is a multi-layer neural network whose neurons only transmit signals in one direction between adjacent layers without feedback connection. In the Transformer architecture, FFN is usually connected after the multi-head attention mechanism (MHA) to further transform the features of the attention output; its standard form contains two layers of linear mapping (fully connected layers) with a nonlinear activation function inserted in the middle. Its formula can actually be expressed as: FFN(Z)=ReLU(xW1+b1)W2+b2; where W1 and W2 are weight matrices, b1 and b2 are bias terms, and the ReLU activation function is used to introduce nonlinear capabilities. Specifically, through FFN Processing, namely:

[0085] ;

[0086] in, Express The processed feature sequence is the cross-modal interaction feature. Represents a feedforward operation, that is: first, the input Projecting the image into a high-dimensional space, the weight matrix W1 enhances the expressive power of the features. After activation via the ReLU activation function, the image is projected back to the weight matrix W2 of the original dimension, achieving nonlinear transformation and compression of the features. Through high-dimensional projection and dimensionality reduction, FFN can enhance key features (such as the visual language characteristics of task-related objects) and suppress noise, thereby improving feature discriminability.

[0087] The Mamba module is used to analyze cross-modal interaction features to output predicted waypoints. Specifically, the obtained cross-modal interaction features are input into the standard Mamba (SSM) module, which processes long sequence dependencies through the selective state space mechanism to output temporal enhancement features. Based on the obtained temporal enhancement features, the fully connected layer decodes and generates predicted waypoints. Specifically:

[0088] The Mamba module uses its selective state space mechanism to scan the sequence (i.e., cross-modal interaction features) to capture long-range dependencies and contextual information. At this time, the output of the Mamba (SSM) module is recorded as The output feature sequence The dimension of the input cross-modal interaction features Similarly, this process is expressed as:

[0089] ;

[0090] In order to encode the characteristic sequence of the Mamba module To connect with the subsequent cross attention mechanism, it is necessary to start from the feature sequence The query required to generate the crisscross attention layer in ),key( ) and value( ). Inspired by the parameter generation of the self-attention mechanism in the Transformer architecture, the present invention uses independent linear projection to achieve this transformation. Specifically, the present invention defines three learnable linear transformation moments to generate queries, keys, and values, respectively, namely:

[0091] ;

[0092] ;

[0093] ;

[0094] in, 、 and denote the learnable linear transformation matrices used to generate queries, keys, and values, respectively, 、 and Represents the target dimensions for query, key, and value respectively.

[0095] Furthermore, the generation process of attention parameters is as follows:

[0096] ;

[0097] ;

[0098] ;

[0099] in, 、 and is an optional bias term. After these transformations, the query sequence can be obtained , key sequence , and the sequence of values .

[0100] Generating a query sequence , key sequence Sum value sequence After that, it is passed as input to the subsequent cross-attention module to achieve interaction and information aggregation between different modalities or different levels of features, namely:

[0101] ;

[0102] in, Represents the features after cross attention fusion, Represents the cross-attention mechanism fusion operation.

[0103] The combination of Mamba module and cross attention represents a cascaded process of cross-modal deep feature fusion, namely:

[0104] ;

[0105] in, Indicates the The Mamba module of the layer is used to capture long sequence dependencies; the final output is the map feature matrix . After obtaining the map feature matrix most relevant to the current instruction After that, a region-based pooling mechanism is applied to reduce the dimensionality of the map feature matrix by focusing on the regions that are most similar to the current task. from Matrix conversion to pooling feature vector ,Right now:

[0106] ;

[0107] in, Represents the region pooling operation, which is used to compress the spatial dimension and retain key features. Finally, the pooled feature vector is extracted with the visual encoder to RGB features and deep features Concatenate to form the input feature vector of the current time step ,Right now:

[0108] ;

[0109] This paper uses the SSM in the Mamba model to model historical trajectory context. Unlike traditional RNN architectures, Mamba uses a selective state space dynamic update mechanism with linear time complexity to more efficiently capture long-range temporal dependencies. While retaining temporal modeling capabilities, the state-space-based Mamba module significantly improves computational efficiency and feature representation breadth, making it particularly suitable for continuous sub-goal generation tasks in large-scale semantic scenarios.

[0110] Specifically, the time step is input into the sequence Enter the Mamba module for timing modeling:

[0111] ;

[0112] in, and The hidden states of the current and previous time steps are respectively. The Mamba module models temporal dependencies through incremental state space updates. At each time step t , the agent only takes the current input features and the hidden state of the previous step Enter Mamba and dynamically update the current step status to This mechanism avoids recalculating the historical sequence, and the time complexity is O (1), significantly improving the efficiency of real-time navigation. Subsequently, a fully connected layer is used to map the hidden state output by Mamba to the two-dimensional coordinate prediction of the sub-target at the current time step, namely:

[0113] ;

[0114] in, Represents the predicted two-dimensional coordinates of the sub-target at the current time step, that is, the predicted waypoint; represents the weight matrix, represents the bias term of the fully connected layer.

[0115] To obtain the ground truth waypoint coordinates, do the following:

[0116] First, we apply the language-aligned waypoint supervision method, LAW, to vision-language navigation tasks in continuous environments. This addresses the issue of inconsistent instructions in existing goal-oriented supervision methods when handling off-path scenarios. Specifically, LAW selects the nearest waypoint, meaning that the language-aligned path differs from the shortest path to the goal. Especially when the agent deviates from the reference path, LAW supervision encourages the agent to move towards the nearest waypoint on the language-aligned path with each step, rather than directly towards the goal.

[0117] Subsequently, the CMA (Cross-Modal Attention) algorithm is used to plan the shortest path from the agent's current position to the nearest waypoint on the command-aligned path. The CMA algorithm model consists of two recurrent networks: one for encoding the agent's state history and the other for predicting actions based on the visual and command features it attends to. The CMA algorithm model takes natural language commands and RGB-D images as input, providing the perception and decision-making capabilities for shortest path planning. Based on the trained CMA algorithm model, when the agent receives a natural language command, the model calculates the shortest path to the nearest waypoint on the language-aligned path based on the agent's current position (which is perceived by obtaining environmental information from RGB-D images).

[0118] Finally, a circle with a radius of 3 meters is drawn with the agent as the center, and the intersection of the path and the circle is considered as the real waypoint.

[0119] Furthermore, the predicted sub-target position is evaluated using the mean square error loss (MSE). Optimize, that is:

[0120] ;

[0121] in, Represents the waypoint prediction loss, which is used to measure the error between the predicted waypoint and the actual waypoint; Represents the time step The actual waypoint coordinates.

[0122] In step S4, DD-PPO is used as a local strategy, and the obtained predicted waypoints are used as input to perform action analysis to generate a probability distribution; and a navigation action is selected and executed according to the obtained probability distribution.

[0123] DD-PPO is used as the local strategy, with waypoints As input, it generates probability distributions for four discrete actions. DD-PPO is a distributed algorithm developed based on proximal policy optimization (PPO). As a policy gradient method, PPO aims to achieve data efficiency and reliability similar to trust region policy optimization (TRPO) using first-order optimization. DD-PPO builds on this foundation by implementing distributed training, allowing multiple machines to participate in the training process simultaneously to improve training efficiency. Action selection is implemented through a policy network and optimized through reinforcement learning. Specifically, the action probability is calculated as follows:

[0124] ;

[0125] in, represents the action probability distribution; Represents the time step The hidden state of , generated by mamba, contains the context information of historical observations and instructions; Represents the Softmax function that normalizes the Mamba output to produce a valid probability distribution.

[0126] From the action probability distribution The sampling or selection actions are as follows:

[0127] ;

[0128] in, Represents the agent at time high-level discrete actions performed (e.g., go forward, turn left, turn right, stop), Represents high-level discrete actions; Representation based on hidden state , the agent selects high-level discrete actions The conditional probability distribution of .

[0129] In order to optimize the local strategy, the present invention also designs two loss functions, namely a multi-class cross entropy loss function for measuring the difference between the actual navigable action and the predicted action probability, and a binary cross entropy loss function for evaluating the difference between the actual stop action and the predicted stop probability. The weighted sum of these two loss functions is used as the total loss function. Specifically:

[0130] A. Multi-class cross entropy loss function During the training process, the model will adjust the parameters according to this loss value, so that the predicted action probability is closer to the probability distribution of the actual action, guiding the intelligent agent to make correct high-level navigation decisions, such as choosing the correct forward and turning direction. The multi-class cross entropy loss function is expressed as:

[0131] ;

[0132] in, represents the multi-class cross entropy loss function, Represents a real navigable action.

[0133] B. The binary cross entropy loss function is used to help the model better judge when to stop moving to avoid missing the target or over-moving. The binary cross entropy loss function is expressed as:

[0134] ;

[0135] in, represents the binary cross entropy loss function, Indicates a real stop action. represents the predicted stopping probability.

[0136] After obtaining the accurate probability distribution, the navigation action can be selected and executed according to the obtained probability distribution, that is, the navigation action with the highest probability is directly selected for execution.

[0137] Based on the dynamic grid map-based visual language navigation method provided by this invention, a dynamic visual language grid map (DGVL-Map) is first constructed. This method can integrate visual language features and depth information extracted by CLIP and dynamically update them in real time based on navigation instructions, enhancing the task relevance and environmental adaptability of the map representation. Secondly, a command-based cascaded attention mechanism is designed to aggregate key information within the grid and filter out map features related to the instructions, thereby improving navigation efficiency. Finally, a waypoint prediction method combined with the Mamba model is proposed. This method can predict sub-goals based on the input, effectively improving the ability to extract key information from complex map features and model long-range dependencies.

[0138] Example 2

[0139] This embodiment discloses a visual language navigation system based on a dynamic grid map.

[0140] A visual language navigation system based on a dynamic grid map, comprising:

[0141] The image acquisition module is configured to: obtain the RGB panoramic image and depth image at the current moment;

[0142] a map construction module configured to: extract visual language features from the RGB panoramic image using a CLIP model, combine the obtained visual language features with depth features from the depth image and the absolute coordinates of the grid cells to construct a grid map; and update the grid map based on the semantic relevance between the grid cells and the current navigation instructions;

[0143] The waypoint prediction module is configured to: process the map features of the updated grid map and the text features of the current navigation instructions based on the cascaded attention mechanism to obtain cross-modal interaction features; use the Mamba module to analyze the obtained cross-modal interaction features to output predicted waypoints;

[0144] The action decision module is configured to: adopt DD-PPO as a local strategy, perform action analysis with the obtained predicted waypoints as input to generate a probability distribution; and select and execute navigation actions according to the obtained probability distribution.

[0145] like Figure 2 Figure 1 shows a visual-linguistic navigation system based on a dynamic grid map. The overall architecture design is as follows: First, the CLIP model processes the RGB panoramic image observed by the agent and integrates depth-based absolute coordinates to construct a dynamic visual-linguistic grid map. This dynamic visual-linguistic grid map, along with text-encoded instructions, is then input into the CAM module for processing to identify map features most relevant to the instructions. Finally, these features are combined with the agent's visual information and input into the Mamba model for waypoint prediction, which then leads to action decisions.

[0146] like Figure 3 As shown in Figure 2, the Cross-Attention Mamba Network (CAM) consists of two parallel branches: one for instruction features extracted by the text encoder, and the other for map features. CAM uses text features as keys (K) and values ​​(V), and map features as queries (Q) to compute the map features most relevant to the current instruction.

[0147] Based on the above systematic architectural design, the present invention can quickly respond to dynamic and continuous environmental changes while avoiding long-distance dependence, so as to achieve precise visual language navigation for intelligent agents.

[0148] Example 3

[0149] The purpose of this embodiment is to provide a computer-readable storage medium.

[0150] A computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements the steps of a visual language navigation method based on a dynamic grid map as described in the first embodiment of the present disclosure.

[0151] Example 4

[0152] The purpose of this embodiment is to provide an electronic device.

[0153] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps of a visual language navigation method based on a dynamic grid map as described in the first embodiment of the present disclosure are implemented.

[0154] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0155] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0156] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. A visual language navigation method based on dynamic grid map, characterized in that: include: Get the current RGB panoramic image and depth image; The CLIP model is used to extract visual language features from RGB panoramic images, and the obtained visual language features and depth features of the depth image are combined with the absolute coordinates of the grid cells to construct a grid map. updating the grid map according to the semantic relevance between the grid cells and the current navigation instruction; The updated grid map features and the text features of the current navigation instructions are processed based on the cascaded attention mechanism to obtain cross-modal interaction features. The obtained cross-modal interaction features are analyzed using the Mamba module to output predicted waypoints. DD-PPO is used as a local strategy, and the predicted waypoints are used as input for action analysis to generate a probability distribution. Navigation actions are selected and executed based on the obtained probability distribution. The updating of the grid map according to the semantic relevance between the grid unit and the current navigation instruction includes: performing cosine similarity matching between the current navigation instruction and the feature vector under the grid unit to obtain an attention weight; and performing weighted fusion on the map features of the grid map according to the obtained attention weight to update the grid map; The method of using the Mamba module to analyze cross-modal interaction features to output predicted waypoints includes: inputting the obtained cross-modal interaction features into the Mamba module, processing long sequence dependencies through a selective state space mechanism to output temporal enhancement features; and generating predicted waypoints based on the obtained temporal enhancement features through decoding through a fully connected layer.

2. A visual language navigation method based on a dynamic grid map as claimed in claim 1, characterized in that: The visual language features and depth features of the depth image are combined with the absolute coordinates of the grid cells to construct a grid map, including: using the visual language features extracted from the RGB panoramic image by the CLIP model as grid features, adjusting the size of the corresponding depth features to match the grid features; and calculating the absolute coordinates based on the grid features and depth features.

3. A visual language navigation method based on a dynamic grid map according to any one of claims 1 to 2, characterized in that: Combining visual language features and depth features of the depth image with the absolute coordinates of the grid cells to construct a grid map also includes: storing the grid features and the absolute coordinates in the grid cells to achieve the fusion of semantic information and geometric information to construct the grid map.

4. The visual language navigation method based on dynamic grid map according to claim 1, characterized in that: The eigenvector under the grid unit is: the eigenvector observed at the same position and different viewing angles under a set time step.

5. The visual language navigation method based on dynamic grid map according to claim 1, characterized in that: The DD-PPO is adopted as a local strategy, and the predicted waypoints are used as input to perform action analysis to generate a probability distribution, including: inputting the coordinates of the obtained predicted waypoints as sub-goals into a distributed proximal policy optimization module, and generating a probability distribution of discrete navigation actions based on a policy generation network within the distributed proximal policy optimization module.

6. A visual language navigation system based on dynamic grid map, characterized in that: include: The image acquisition module is configured to: obtain the RGB panoramic image and depth image at the current moment; A map construction module is configured to: extract visual language features from the RGB panoramic image using a CLIP model, and combine the obtained visual language features with the depth features of the depth image and the absolute coordinates of the grid cells to construct a grid map; updating the grid map according to the semantic relevance between the grid cells and the current navigation instruction; The waypoint prediction module is configured to: process the map features of the updated grid map and the text features of the current navigation instructions based on the cascaded attention mechanism to obtain cross-modal interaction features; use the Mamba module to analyze the obtained cross-modal interaction features to output predicted waypoints; The action decision module is configured to: adopt DD-PPO as a local strategy, perform action analysis with the obtained predicted waypoints as input to generate a probability distribution; select and execute navigation actions based on the obtained probability distribution; The updating of the grid map according to the semantic relevance between the grid unit and the current navigation instruction includes: performing cosine similarity matching between the current navigation instruction and the feature vector under the grid unit to obtain an attention weight; and performing weighted fusion on the map features of the grid map according to the obtained attention weight to update the grid map; The method of using the Mamba module to analyze cross-modal interaction features to output predicted waypoints includes: inputting the obtained cross-modal interaction features into the Mamba module, processing long sequence dependencies through a selective state space mechanism to output temporal enhancement features; and generating predicted waypoints based on the obtained temporal enhancement features through decoding through a fully connected layer.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps of the visual language navigation method based on a dynamic grid map as described in any one of claims 1 to 5 are implemented.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the visual language navigation method based on a dynamic grid map as described in any one of claims 1 to 5 are implemented.

Images