A visual language navigation method integrating grid map and topological map

Abstract

The present invention discloses a visual language navigation method integrating a grid map and a topological map, belonging to the field of navigation technology. The method receives RGB and depth information through a waypoint predictor and linearly fuses them into RGBD information, thereby generating useful reachable nodes; in short-term reasoning, a grid map is used to model local historical observations during navigation, and time and space information is used to describe the environment of local visits; in long-term reasoning, a topological map is used to spontaneously update the topological map through the waypoints generated by the waypoint predictor, and the waypoints are effectively added to the dynamic topological map; then, a path is planned across modalities through visual and semantic information, and long-term and short-term navigation scores are dynamically integrated to make navigation actions. The present invention improves the accuracy of navigation.

Description

Technical Field

[0001] The invention belongs to the field of navigation technology, and in particular relates to a visual language navigation method integrating a grid map and a topological map. Background Art

[0002] Vision language navigation (VLN) is a very challenging cross-domain research problem. It aims to enable intelligent agents to interpret human instructions and perform a series of actions to navigate in unknown environments. Most of the current work in this field is unrelated to robots and focuses on highly abstract simulators. After the continuous environment is introduced into the VLN task, it abandons the strong graph assumption of discrete environments and requires the agent to freely perform low-level operations in the 3D grid, further increasing the difficulty of vision language navigation.

[0003] In the prior art, some methods construct access environments and perform global planning by constructing topological maps. However, topological maps are a highly abstract form of representation, which makes it difficult for these methods to represent the spatial relationship between scenes in historical observations, thereby losing a lot of detailed information. Therefore, the work uses top-down semantic maps to model the navigation environment, which can more accurately express the spatial relationship between objects. However, since the pre-defined semantic labels are very limited, the map cannot represent this information when undefined objects or scenes appear.

[0004] Therefore, it is necessary to provide a visual language navigation method that integrates grid maps and topological maps to solve the problems raised in the above background technology. Summary of the invention

[0005] The present invention discloses a visual language navigation method integrating grid maps and topological maps. In short-term reasoning, the method uses grid maps to model local historical observations during navigation and uses time and space information to describe the local visited environment. In long-term reasoning, the method uses topological maps to spontaneously update the topological maps through path points generated by a waypoint predictor, thereby improving the accuracy of navigation.

[0006] In order to solve the above technical problems, the present invention provides a visual language navigation method integrating a grid map and a topological map, comprising the following steps:

[0007] Step S1, collecting panoramic images of the current environment observed by the agent at different viewing angles, inputting the panoramic images into a waypoint predictor, and predicting the path points of the agent;

[0008] Step S2, based on the path points predicted by the waypoint predictor, a grid map is constructed through short-term planning, and a topology map is constructed through long-term planning;

[0009] Step S3, collect the instructions obtained by the intelligent agent, use the grid map and the topological map to perform the VLN navigation task respectively, obtain the navigation scores of different nodes under the grid map navigation and the topological map navigation, dynamically merge the two navigation scores to obtain the final navigation score, and guide the intelligent agent to reach the target node with the highest navigation score.

[0010] As a preferred improvement, in step S1, the viewing angle for panoramic image acquisition is the viewing angle corresponding to the intelligent agent at horizontal heading angles of 0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300° and 330°, and an RGB image and a depth image are collected at each viewing angle to form the panoramic image.

[0011] As a preferred improvement, the waypoint predictor includes a visual encoder, a Transformer model and a nonlinear classifier, wherein the visual encoder is used to encode the collected panoramic image into visual features; the Transformer model adopts a self-attention mechanism to model the spatial relationship in the image, infer adjacent waypoints, and obtain contextual deep embedding; the nonlinear classifier is used to project the embedded information output by the Transformer model to the probability of adjacent waypoints in space, obtain a heat map representing the probability of adjacent waypoints in space, and use non-maximum suppression to process the heat map to obtain multiple adjacent waypoints.

[0012] As a preferred improvement, the visual encoder includes an RGB feature encoder and a depth feature encoder, the RGB feature encoder is used to extract features from the RGB image to obtain an RGB feature sequence; the depth feature encoder is used to extract features from the depth image to obtain a depth feature sequence; the nonlinear classifier is a multilayer perceptron.

[0013] As a preferred improvement, the construction of the topological map specifically includes the following steps:

[0014] The path point where the agent is currently located is marked as the current node, and the agent can access the panoramic image observed at the current node; the path points near the current node are marked as observation nodes, indicating the reachable locations near the agent; the path points that the agent has visited are marked as visited nodes; any two adjacent nodes are connected by an edge, and all nodes and edges together form a topological map of the current environment. The topological map is updated in real time according to the position of the agent at different time steps; each node stores visual features and location information; each edge stores the Euclidean distance between the two nodes connected to it.

[0015] As a preferred improvement, the updating process of the topology map is expressed as follows:

[0016] For any new path point, calculate its Euclidean distance to all nodes in the topological map, using a pre-set distance threshold To determine the type of the new waypoint:

[0017] If the distance between the new path point and all nodes is greater than the distance threshold , then the path point is classified as a new observation node;

[0018] If the distance between the new path point and a node is less than the distance threshold , and the node is a visited node, delete the new path point and add an edge between the current node and the visited node;

[0019] If the distance between the new path point and a node is less than the distance threshold , and the node is an observation node, the observation node is deleted, and the new path point is classified as a new observation node to update the observation node.

[0020] As a preferred improvement, the construction of the grid map specifically includes the following steps:

[0021] Step S21, based on the panoramic image and the current posture of the intelligent agent, a grid map generation model is used to generate an original grid map of the current environment, and the RGB features in each grid are used as visual features of the current grid;

[0022] Step S22: Reduce the depth image collected by the agent to the same size as the visual feature, and then calculate the absolute coordinates of the current grid using the depth value of the depth image. , the calculation process is expressed as:

[0023] ;

[0024] In the formula, Respectively represent k The horizontal and vertical coordinates of the grid; Represents the time step t When , the horizontal and vertical coordinates of the agent; Represents the agent and k The Euclidean distance of the grid; Indicates the current direction of the agent and k The heading angle between grids;

[0025] Step S23, storing the visual features and absolute coordinates of all grids in a grid memory as a grid memory , expressed as:

[0026] ;

[0027] In the formula, Represents the time step t Grid memory at -1;

[0028] Step S24, using the current position of the agent as the coordinate origin and the current direction of the agent as y The positive direction of the axis is perpendicular to the horizontal plane. y The direction of the axis is x Axis direction, define the dynamic grid map coordinate system, project all visual features and position information in the grid memory into the dynamic grid coordinate system to form a dynamic grid map.

[0029] As a preferred improvement, in the VLN navigation process, text embedding, grid map embedding and topology map embedding are also required; wherein:

[0030] Text embedding specifically includes: language instructions Each word embedding in adds a position embedding and a tag type embedding, and then inputs the multi-layer Transformer network to obtain the word representation and obtain the instruction encoding ;

[0031] Grid map embedding specifically involves aggregating multiple grid features in each cell into an embedding vector, and the encoded cell is represented as ;

[0032] The topological map embedding specifically includes: the encoded nodes and word embeddings are sent to a multi-layer transformer for node-level cross-modal fusion. The multi-layer transformer contains a bidirectional cross-attention sublayer, two self-attention sublayers, and two feedforward sublayers. After that, the visual self-attention sublayer is replaced by the graphic self-attention, and finally the encoded nodes are obtained. The topological map and instruction encoding are used for node-level cross-modal fusion. The encoded nodes are represented as .

[0033] As a preferred improvement, during the grid map embedding process, an instruction association method is introduced to aggregate the features in each cell. The instruction association method specifically includes the following process:

[0034] Step S31, setting the evaluation matrix A To evaluate the grid features in each cell With instruction encoding The correlation, evaluation matrix A It is expressed as:

[0035] ;

[0036] In the formula, All represent learnable parameters; T Represents matrix transpose;

[0037] Step S32, evaluating the matrix A Perform row-wise max pooling to evaluate the grid features in each cell With navigation instruction token Relevance , expressed as:

[0038] ;

[0039] In the formula, Indicates the number of features in the cell; L Indicates the length of the language instruction;

[0040] Step S33, aggregate the grid features in each cell into an embedding vector , expressed as:

[0041] ;

[0042] In the formula, represents the intermediate parameter, ; represents a learnable parameter; Represents the positional features in the cell;

[0043] In order to express spatial relationships, location information is introduced into the grid map, and the map features are expressed as:

[0044] ;

[0045] In the formula, represents learnable parameters; represents the Euclidean distance between the cell center and the agent; Represents the relative orientation angle between the agent and the cell.

[0046] As a preferred improvement, the navigation score of each node in the topological map and the grid map is expressed as:

[0047] ;

[0048] ;

[0049] ;

[0050] In the formula, Indicates i The navigation score of each node in the topological map; No. i The navigation score of a node in a grid map; Indicates that a two-layer feedforward network is used for processing; represents the weight coefficient, ,in, represents the feature representation of the stop node, Indicates the feature representation of the center cell.

[0051] The beneficial effects of the present invention are:

[0052] (1) The present invention adds a topological map to visual language navigation, so that the generated waypoints are used to construct a topological map, which enhances the agent's use of environmental information, strengthens the backtracking ability, and enhances the navigation performance and long-term planning ability of visual language navigation, solving the problems of the common limitations of waypoint prediction and insufficient capture of the global environment layout;

[0053] (2) The grid map provided by the present invention contains rich semantic and environmental information, which is helpful for building spatial relationships. By integrating the grid map and the topological map and combining the advantages of the two maps, the navigation efficiency is further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0055] Figure 1 A flowchart of a visual language navigation method integrating a grid map and a topological map provided by the present invention;

[0056] Figure 2 This is the architecture diagram of the waypoint predictor;

[0057] Figure 3 Architectural diagram generated for grid map;

[0058] Figure 4 Architectural diagrams generated for topology diagrams;

[0059] Figure 5 This is a schematic diagram of short-distance navigation results;

[0060] Figure 6 This is a schematic diagram of long-distance navigation results. DETAILED DESCRIPTION

[0061] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0062] Please refer to Figure 1-Figure 6 The present invention provides a visual language navigation method integrating a grid map and a topological map, comprising the following steps:

[0063] Step S1, collecting panoramic images of the current environment observed by the agent at different viewing angles, inputting the panoramic images into a waypoint predictor, and predicting the path points of the agent.

[0064] Specifically, the viewing angle for panoramic image acquisition is the viewing angle of the agent at 12 equidistant horizontal heading angles, namely, the viewing angles corresponding to the horizontal heading angles of 0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300° and 330°. An RGB image and a depth image are collected at each viewing angle to form the panoramic image, which is expressed as: , where Represents the time step The panoramic image collected at the time; Represents the time step RGB image collected at the time; Represents the time step The depth image collected at that time.

[0065] The waypoint predictor includes a visual encoder, a Transformer model and a nonlinear classifier.

[0066] The visual encoder is used to encode the collected panoramic image into visual features, which includes an RGB feature encoder and a depth feature encoder. The RGB feature encoder is used to encode the RGB image into RGB features, which are expressed as: , where represents the viewing angle index, Indicates The RGB features of each perspective; the depth feature encoder is used to encode the depth image into depth features, expressed as: , where Indicates Because there are 12 images from different perspectives in the panoramic view, in order to distinguish the features from different perspectives, directional features are added to each perspective, expressed as: , express The directional characteristics of the viewing angle, , where Indicates The RGB features, depth features, and direction features of all 12 perspectives at each time step are combined and sent to the Transformer model in parallel.

[0067] In this embodiment, the ViT-B / 32 module in the CLIP model is used as the RGB feature encoder; the ResNet50 model pre-trained on point target navigation is used as the deep feature encoder.

[0068] The Transformer model uses a self-attention mechanism to model the spatial relationship in the image, and infers adjacent path points based on the spatial relationship in the image to obtain contextual deep embedding. The Transformer model includes two layers, each with 12 self-attention heads.

[0069] The nonlinear classifier is a multilayer perceptron, which is used to project the embedded information output by the Transformer model to the probability of adjacent path points in space, obtain a heat map representing the probabilities of adjacent path points in space, and use non-maximum suppression (NMS) to process the heat map to obtain multiple adjacent path points.

[0070] Step S2, based on the path points predicted by the waypoint predictor, a grid map is constructed through short-term planning, and a topology map is constructed through long-term planning.

[0071] The construction of the topological map includes the following steps:

[0072] The path point where the agent is currently located is marked as the current node, and the agent can access the panoramic image observed at the current node; the path points near the current node are marked as observation nodes, indicating the reachable locations near the agent; the path points that the agent has visited are marked as visited nodes; any two adjacent nodes are connected by an edge, and all nodes and edges together constitute the topological map of the current environment.

[0073] Time step t The constructed topological map is represented as ,in Represents the nodes in the topological map. There are three types of nodes: current node, observed node, and visited node. Each node stores visual features and location information. Represents the edges in the topological map, and each edge stores the Euclidean distance between two adjacent nodes.

[0074] For a new path point, its type is determined by calculating the Euclidean distance between the path point and all nodes in the topological map, and determining the type of the new path point by a pre-set distance threshold:

[0075] If the distance between the new path point and all nodes is greater than the distance threshold , then the path point is classified as a new observation node;

[0076] If the distance between the new path point and a node is less than the distance threshold , and the node is a visited node, delete the new path point and add an edge between the current node and the visited node;

[0077] If the distance between the new path point and a node is less than the distance threshold , and the node is an observation node, the observation node is deleted, and the new path point is classified as a new observation node to update the observation node.

[0078] At each time step t The topological map is constructed by self-organizing the waypoints predicted by the waypoint predictor, without assuming prior knowledge of the environment structure. t As the process progresses, the topology map of the current environment is gradually updated.

[0079] The construction of the grid map includes the following steps:

[0080] Step S21, based on the panoramic image and the current posture of the intelligent body, a grid map generation model is used to generate an original grid map of the current environment, and the RGB features in each grid are used as the visual features of the current grid.

[0081] The process of generating a grid map using a grid map generation model belongs to conventional techniques in the art, and this embodiment will not be described in detail. The RGB feature is extracted by the RGB feature encoder in the waypoint predictor. k The visual features of a grid are represented as .

[0082] Step S22: Reduce the depth image collected by the agent to the same size as the visual feature, and then calculate the absolute coordinates of the current grid using the depth value of the depth image. , the calculation process is expressed as:

[0083] ;

[0084] In the formula, Respectively represent k The horizontal and vertical coordinates of the grid; Represents the time step tWhen , the horizontal and vertical coordinates of the agent; Represents the agent and k The Euclidean distance of the grid; Indicates the current direction of the agent and k The heading angle between grids.

[0085] Step S23, storing the visual features and absolute coordinates of all grids in a grid memory as a grid memory , expressed as:

[0086] ;

[0087] In the formula, Represents the time step t Grid memory at -1.

[0088] In a continuous environment, each time the agent reaches a path point, it memorizes the grid Update, grid memory It contains fine-grained visual features and position information of each grid, allowing the agent to make full use of semantic features and thus enhance short-term planning capabilities.

[0089] Step S24, using the current position of the agent as the coordinate origin and the current direction of the agent as y The positive direction of the axis is perpendicular to the horizontal plane. y The direction of the axis is x Axis direction, define the dynamic grid map coordinate system, project all visual features and position information in the grid memory into the dynamic grid coordinate system to form a dynamic grid map.

[0090] The size of the dynamic grid map will expand as the agent visits the larger environment. The agent is always located at the center of the dynamic grid map. After all visual features and location information are projected, the dynamic grid map is divided into N × N cells, aggregate the visual features in all cells into an embedding vector to form a dynamic grid map feature . Then for the dynamic grid map feature A local action space is equipped for short-term reasoning. The local action space consists of the current node and the nodes adjacent to the current node. Then the transformation operation from node to cell is performed, the postures of all nodes in the local action space are projected onto the dynamic grid map, the coordinates of all nodes on the dynamic grid map are calculated, and the nodes are assigned to different cells.

[0091] Step S3, collect the language instructions obtained by the intelligent agent, use the grid map and the topological map to perform VLN navigation tasks respectively, obtain the navigation scores of different nodes under the grid map and the topological map navigation respectively, dynamically merge the two navigation scores to obtain the final navigation score, and guide the intelligent agent to reach the target node with the highest navigation score.

[0092] In the VLN navigation process, text embedding, grid map embedding and topology map embedding are also required.

[0093] Text Embedding: A Language Instruction Each word embedding in adds a position embedding and a tag type embedding, and then inputs the multi-layer Transformer network to obtain the word representation and obtain the instruction encoding .

[0094] Grid map embedding: Aggregate multiple grid features in each cell into one embedding vector. Due to the complexity of the navigation environment, the grid features in each cell area are not all required for the agent to navigate, which is inconsistent with the language instructions. Highly relevant key information is what the intelligent agent needs most. Therefore, this paper adopts an instruction association method to aggregate the features in each cell to better understand the environment. The encoded cell is represented as .

[0095] The instruction association method specifically includes the following process:

[0096] Step S31, setting the evaluation matrix A To evaluate the grid features in each cell With instruction encoding The correlation, evaluation matrix A It is expressed as:

[0097] ;

[0098] In the formula, All represent learnable parameters; T Represents matrix transpose;

[0099] Step S32, evaluating the matrix A Perform row-wise max pooling to evaluate the grid features in each cell With navigation instruction token Relevance :

[0100] ;

[0101] In the formula, Indicates the number of features in the cell; LIndicates the length of the language instruction;

[0102] Step S33, aggregate the grid features in each cell into an embedding vector , expressed as:

[0103] ;

[0104] In the formula, represents the intermediate parameter, ; represents a learnable parameter; Represents the positional features in the cell;

[0105] In order to express spatial relationships, location information is introduced into the grid map, and the map features are expressed as:

[0106] ;

[0107] In the formula, represents learnable parameters; represents the Euclidean distance between the cell center and the agent; Represents the relative orientation angle between the agent and the cell.

[0108] Topological map embedding: The encoded node and word embeddings are fed into a multi-layer transformer for node-level cross-modal fusion. It contains a bidirectional cross-attention sublayer, two self-attention sublayers, and two feed-forward sublayers. After that, the visual self-attention sublayer is replaced by the graphical self-attention (GASA) to finally obtain the encoded nodes. The topological map and instruction encoding are used for node-level cross-modal fusion, and the encoded nodes are represented as The specific steps include:

[0109] Add position coding and navigation step coding to each node in the topological map, where position coding is to embed the node position into the self-centered view; navigation step coding is to give each node a time step for its visit; at the same time, add a stop node to the topological map to represent the stop action, and connect the stop node to other nodes;

[0110] The nodes and instruction codes with added position codes and navigation step codes are fed into a multi-layer cross-modal Transformer model for interaction to predict the navigation target score of each node.

[0111] The navigation scores of the current node and the visited nodes are shielded. The agent selects the node with the highest navigation score based on the navigation scores of the observation node and the stop node. If an observation node is selected, the shortest path to the target location is calculated using the Dikjstra algorithm. If a stop node is selected, the current navigation is terminated.

[0112] Specifically, position encoding is to add the direction and Euclidean distance of any node relative to the current node to the visual features; in navigation step encoding, the navigation step code of the observed node is assigned to 0, which can encode the visited nodes into different navigation histories and promote alignment with the original instructions.

[0113] The standard self-attention layer only considers the visual similarity between nodes, and often ignores the more relevant neighboring nodes of distant nodes. To improve this situation, the present invention uses a graph-aware self-attention layer (GASA), which further considers the graph topology when calculating attention. The execution process of the graph-aware self-attention layer is It is expressed as:

[0114] ;

[0115] In the formula, X Represents a node; E represents the spatial matrix of all-pairs shortest distances obtained from the edges of the topological map, All are learnable matrices; d Indicates the number of nodes; b e Indicates the offset; Indicates the use of softmax Function activation.

[0116] The purpose of fusion is to integrate the long-term action space and the short-term action space, and integrate the short-term reasoning results of the grid map into the topological map. Specifically, first, the local action space Using the cell-to-node operation (the inverse of the node-to-cell operation, i.e. mapping the features on the cell to the node), the converted cell is represented as: , where represents the global action space; represents a subset of the global action space.

[0117] Then, the navigation score of each node in the topological map and grid map is expressed as:

[0118] ;

[0119] ;

[0120] The final navigation score is expressed as:

[0121] ;

[0122] In the formula, Indicates i The navigation score of each node in the topological map; No. i The navigation score of a node in a grid map; Indicates that a two-layer feedforward network is used for processing; represents the weight coefficient, ,in, represents the feature representation of the stop node, Indicates the feature representation of the center cell.

[0123] In the VLN task, there is no need to visit the same node repeatedly, so the navigation scores of the current node and the visited nodes are masked.

[0124] Example 1

[0125] In order to verify the feasibility of a visual language navigation method for integrating grid maps and topological maps provided by the present invention, a comparative test is set up to compare the method of the present invention with several visual language navigation methods in the prior art. The test results are summarized in Tables 1 and 2, wherein Table 1 shows a performance comparison table of the present application and several methods in the prior art on the Val-Seen segmentation set of the R2R-CE dataset; Table 2 shows a performance comparison table of the present application and several methods in the prior art on the Val-UnSeen segmentation set of the R2R-CE dataset. The performance indicators involved in Tables 1 and 2 are defined as follows:

[0126] Success rate SR It is used to evaluate the probability of successful arrival of navigation, which is defined as follows:

[0127] ;

[0128] In the formula, N Indicates the number of test data; k Represents any test data; Represents the docking coordinates of the agent.

[0129] Track length TL It is used to measure the step length required for the agent to reach the destination, and is defined as follows:

[0130] ;

[0131] In the formula, m represents the target point coordinates, d th Represents the agent navigation distance.

[0132] Navigation Error NE It is used to measure the shortest distance between the agent and the target point, which is defined as follows:

[0133] ;

[0134] In the formula, d m Indicates the target node location d k Indicates the actual node position.

[0135] Oracle Success Rate OSR The success rate used for the Horizontal Oracle Stop strategy.

[0136] Path length weighted success rate SPL , which is used to measure the acceptance of navigation success rate, is defined as follows:

[0137] ;

[0138] In the formula, d s Indicates the distance between the starting point and the target point of the current navigation.

[0139] Table 1 Test results of various methods on the Val-Seen segmentation set of the R2R-CE dataset

[0140]

[0141] Table 2 Test results of various methods on the Val-UnSeen segmentation set of the R2R-CE dataset

[0142]

[0143] As can be seen from Tables 1 and 2, the navigation success rate and path length weighted success rate of navigation on the Val-Seen segmentation set are higher than those of previous studies. The same Val-Unseen segmentation set also maintains excellent performance, proving the effectiveness of adding a grid map. Among them, compared with the GridMM method, this method also uses a grid map, but uses direct image selection in long-term planning. The important indicator SPL in the Val-Unseen segmentation set is five points better than GridMM. This is due to the long-term planning using topological maps, which has stronger generalization ability in the face of unknown environments. Overall, adding a navigation map to visual language navigation can greatly improve the performance of navigation. The orientation information and stored visual semantic information provided by the map can enable the intelligent agent to have a strong adaptability to the navigation environment, while strengthening the navigation backtracking ability and weakening the navigation error capital. The improvement analysis of specific modules is analyzed one by one by the following ablation experiments.

[0144] As shown in Table 3 below, the ablation experiment of map selection includes four experiments, namely, not adding any map, adding topological map, adding grid map and multi-map fusion of the present application. It can be seen from Table 3 that the effect of fusion map is much better than that of single map or no map.

[0145] Table 3 Ablation experiment of map selection

[0146]

[0147] As shown in Table 4 below, this is an ablation experiment on map size. The grid map is divided into three different sizes: 7×7, 14×14, and 21×21, and the navigation effects brought about by selecting different sizes are compared.

[0148] It can be seen from Table 4 that the navigation performance of the map with a scale of 14×14 is the best. Normally, as the map size increases, a larger map can accommodate more environmental details and more accurately represent spatial relationships. However, as the map size reaches a certain level, the amount of calculation will increase, which in turn leads to a decrease in performance.

[0149] Table 4 Ablation experiment of map size

[0150]

[0151] As shown in Table 5 below, this is the ablation experiment of the pre-training task.

[0152] Table 5 Ablation experiments on pre-training tasks

[0153]

[0154] As can be seen from the first row, without pre-training, the navigation performance is the worst, because without pre-training of speech, the learned mapping lacks a universal multimodal representation. As can be seen from the second row, the method shows good performance after adding MLM. As can be seen from the third row, the agent also has the ability to imagine the fuzzy environment and enhances the ability to adapt to the location environment. As can be seen from the fourth row, the SAP single-step action prediction added in the fourth row also improves the performance to a certain extent, and it makes the path length shorter, allowing the agent to find the target node more effectively.

[0155] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A visual language navigation method integrating grid map and topology map, characterized in that: The steps include: Step S1, collecting panoramic images of the current environment observed by the agent at different viewing angles, inputting the panoramic images into a waypoint predictor, and predicting the path points of the agent; Step S2, based on the waypoints predicted by the waypoint predictor, a grid map is constructed through short-term planning, and a topological map is constructed through long-term planning; the waypoint predictor includes a visual encoder, a Transformer model and a nonlinear classifier, the visual encoder is used to encode the collected panoramic image into visual features; the Transformer model adopts a self-attention mechanism to model the spatial relationship in the image, infer adjacent waypoints, and obtain contextual deep embedding; the nonlinear classifier is used to project the embedding information output by the Transformer model to the probability of adjacent waypoints in space, obtain a heat map representing the probability of adjacent waypoints in space, and use non-maximum suppression to process the heat map to obtain multiple adjacent waypoints; Step S3, collect the instructions obtained by the intelligent agent, use the grid map and the topological map to perform the VLN navigation task respectively, obtain the navigation scores of different nodes under the grid map navigation and the topological map navigation, dynamically merge the two navigation scores to obtain the final navigation score, and guide the intelligent agent to reach the target node with the highest navigation score.

2. The visual language navigation method integrating grid map and topology map according to claim 1, characterized in that: In step S1, the viewing angles for panoramic image acquisition are the viewing angles corresponding to the intelligent agent at horizontal heading angles of 0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300° and 330°. An RGB image and a depth image are collected at each viewing angle to form the panoramic image.

3. The visual language navigation method integrating grid map and topology map according to claim 1, characterized in that: The visual encoder includes an RGB feature encoder and a depth feature encoder, wherein the RGB feature encoder is used to extract features from the RGB image to obtain an RGB feature sequence; the depth feature encoder is used to extract features from the depth image to obtain a depth feature sequence; The nonlinear classifier is a multi-layer perceptron.

4. The visual language navigation method integrating grid map and topology map according to claim 1, characterized in that: The construction of the topological map includes the following steps: The path point where the agent is currently located is marked as the current node, and the agent can access the panoramic image observed at the current node; the path points near the current node are marked as observation nodes, indicating the locations that the agent can reach nearby; The path points that the agent has visited are marked as visited nodes; any two adjacent nodes are connected by an edge, and all nodes and edges together constitute the topological map of the current environment. The topological map is updated in real time according to the position of the agent at different time steps; each node stores visual features and location information; each edge stores the Euclidean distance between the two nodes connected to it.

5. The visual language navigation method integrating grid map and topology map according to claim 4, characterized in that: The updating process of the topological map is expressed as: For any new path point, calculate its Euclidean distance to all nodes in the topological map, using a pre-set distance threshold To determine the type of the new waypoint: If the distance between the new path point and all nodes is greater than the distance threshold , then the path point is classified as a new observation node; If the distance between the new path point and a node is less than the distance threshold , and the node is a visited node, delete the new path point and add an edge between the current node and the visited node; If the distance between the new path point and a node is less than the distance threshold , and the node is an observation node, the observation node is deleted, and the new path point is classified as a new observation node to update the observation node.

6. The visual language navigation method integrating grid map and topology map according to claim 3, characterized in that: The construction of the grid map includes the following steps: Step S21, based on the panoramic image and the current posture of the intelligent agent, a grid map generation model is used to generate an original grid map of the current environment, and the RGB features in each grid are used as visual features of the current grid; Step S22: Reduce the depth image collected by the agent to the same size as the visual feature, and then calculate the absolute coordinates of the current grid using the depth value of the depth image. , the calculation process is expressed as: ； In the formula, Respectively represent k The horizontal and vertical coordinates of the grid; Represents the time step t When , the horizontal and vertical coordinates of the agent; Represents the agent and k The Euclidean distance of the grid; Indicates the current direction of the agent and k The heading angle between grids; Step S23, storing the visual features and absolute coordinates of all grids in a grid memory as a grid memory , expressed as: ； In the formula, Represents the time step t Grid memory at -1; Step S24, using the current position of the agent as the coordinate origin and the current direction of the agent as y The positive direction of the axis is perpendicular to the horizontal plane. y The direction of the axis is x Axis direction, define the dynamic grid map coordinate system, project all visual features and position information in the grid memory into the dynamic grid coordinate system to form a dynamic grid map.

7. The visual language navigation method integrating grid map and topology map according to claim 1, characterized in that: In the VLN navigation process, text embedding, grid map embedding and topology map embedding are also required; among them: Text embedding specifically includes: language instructions Each word embedding in adds a position embedding and a tag type embedding, and then inputs the multi-layer Transformer network to obtain the word representation and obtain the instruction encoding ; Grid map embedding specifically involves aggregating multiple grid features in each cell into an embedding vector, and the encoded cell is represented as ; The topological map embedding specifically includes: the encoded nodes and word embeddings are sent to a multi-layer transformer for node-level cross-modal fusion. The multi-layer transformer contains a bidirectional cross-attention sublayer, two self-attention sublayers, and two feedforward sublayers. After that, the visual self-attention sublayer is replaced by the graphic self-attention. Finally, the encoded nodes are obtained, and the topological map and instruction encoding are used for node-level cross-modal fusion. The encoded nodes are represented as .

8. The visual language navigation method integrating grid map and topology map according to claim 7, characterized in that: During the grid map embedding process, the instruction association method is introduced to aggregate the features in each cell. The instruction association method specifically includes the following process: Step S31, setting the evaluation matrix A To evaluate the grid features in each cell With instruction encoding The correlation, evaluation matrix A It is expressed as: ； In the formula, All represent learnable parameters; T Represents matrix transpose; Step S32, evaluating the matrix A Perform row-wise max pooling to evaluate the grid features in each cell With navigation instruction token Relevance , expressed as: ； In the formula, Indicates the number of features in the cell; L Indicates the length of the language instruction; Step S33, aggregate the grid features in each cell into an embedding vector , expressed as: ； In the formula, represents the intermediate parameter, ; represents a learnable parameter; Represents the positional features in the cell; In order to express spatial relationships, location information is introduced into the grid map, and the map features are expressed as: ； In the formula, represents learnable parameters; represents the Euclidean distance between the cell center and the agent; Represents the relative orientation angle between the agent and the cell.

9. The visual language navigation method integrating grid map and topology map according to claim 8, characterized in that: The navigation score of each node in the topological map and grid map is expressed as: ； ； ； In the formula, Indicates i The navigation score of each node in the topological map; No. i The navigation score of a node in a grid map; Indicates that a two-layer feedforward network is used for processing; represents the weight coefficient, ,in, represents the feature representation of the stop node, Feature representation of the center cell.

Images