Visual language navigation method based on topological graph

Abstract

The present invention provides a visual language navigation method based on a topological map, belonging to the field of navigation technology. The present invention is based on a waypoint predictor, and uses the waypoint predictor to construct a topological map of the current environment. The intelligent agent plans the path based on the nodes in the topological map, thereby increasing the ability to capture the global environment layout, and can well trace back to the previous position, thereby improving the long-term planning ability of the intelligent agent.

Description

Technical Field

[0001] The invention belongs to the field of navigation technology, and in particular relates to a visual language navigation method based on a topological graph. Background Art

[0002] VLN-CE (Vision-and-Language Navigation in Continuous Environments) is a new type of navigation technology. Its principle is to train an intelligent agent to perform positioning and navigation tasks in a 3D continuous environment by fusing language instructions with visual perception.

[0003] There are mainly two types of visual language navigation tasks: one is a visual language navigation method based on discrete environments, which uses a pre-trained language Transformer model or a scene memory method to simplify navigation into a predefined graph of a convenient environment, greatly simplifying the possible locations of the agent. However, this navigation method cannot reflect the state of the agent in the real environment. The other is a visual language navigation method based on a continuous environment, which forms a connected graph by constructing a set of accessible nodes in the entire environment. The agent moves between nodes with high-level actions, but this navigation method requires building prior knowledge of the environment, which restricts the usage scenario.

[0004] In order to solve the above problems, the related technology provides a visual language navigation method based on waypoint prediction. The intelligent agent collects images of the surrounding environment, and then predicts the path points of the intelligent agent through a waypoint predictor to guide the intelligent agent to the target location required in the instructions. However, the waypoint prediction method also has limitations and fails to capture the global environment layout. This makes it very difficult for the intelligent agent to trace back to its previous position, hindering the intelligent agent's long-term planning ability. Summary of the invention

[0005] The present invention provides a visual language navigation method based on a topological map, which is based on a waypoint predictor and uses the waypoint predictor to construct a topological map of the current environment. The intelligent agent plans the path using the nodes in the topological map, thereby increasing the ability to capture the global environment layout and being able to trace back past positions well, thereby improving the long-term planning ability of the intelligent agent.

[0006] In order to solve the above-mentioned technical problems, the present invention is achieved as follows:

[0007] The present invention provides a visual language navigation method based on a topological map, comprising the following steps:

[0008] Step S1, collecting panoramic images of the current environment observed by the agent at different viewing angles, inputting the panoramic images into a waypoint predictor, and predicting the path points of the agent;

[0009] Step S2, based on the path points predicted by the waypoint predictor, mark the path point where the agent is currently located as the current node, and the agent can access the panoramic image observed at the current node; mark the path points near the current node as observation nodes, indicating the reachable locations near the agent; mark the path points that the agent has visited as visited nodes; any two adjacent nodes are connected by an edge, and all nodes and edges together form a topological map of the current environment, and the topological map is updated in real time according to the position of the agent at different time steps;

[0010] Step S3, introducing a sub-instruction generation algorithm, converting the original instruction consisting of a long sentence obtained by the agent into a sub-instruction consisting of a short sentence, extracting features of the original instruction and the sub-instruction respectively, and using the extracted two features together as the instruction code;

[0011] In step S4, the topological map and instruction encoding are used to perform cross-modal fusion at the node level, predict the position of the agent under the topological map, and guide the agent to move to the target location along the specific path required in the original instruction.

[0012] As a preferred improvement, in step S1, the viewing angle for panoramic image acquisition is the viewing angle corresponding to the intelligent agent at horizontal heading angles of 0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300° and 330°, and an RGB image and a depth image are collected at each viewing angle to form a panoramic image.

[0013] As a preferred improvement, the waypoint predictor includes a visual encoder, a Transformer model and a nonlinear classifier, wherein the visual encoder is used to encode the collected panoramic image into visual features; the Transformer model adopts a self-attention mechanism to model the spatial relationship in the image, infer adjacent waypoints, and obtain contextual deep embedding; the nonlinear classifier is used to project the embedded information output by the Transformer model to the probability of adjacent waypoints in space, obtain a heat map representing the probability of adjacent waypoints in space, and use non-maximum suppression to process the heat map to obtain multiple adjacent waypoints.

[0014] As a preferred improvement, the visual encoder includes an RGB feature encoder and a depth feature encoder, the RGB feature encoder is used to extract features from the RGB image to obtain an RGB feature sequence; the depth feature encoder is used to extract features from the depth image to obtain a depth feature sequence; the nonlinear classifier is a multilayer perceptron.

[0015] As a preferred improvement, each node in the topological map stores visual feature information and location information; each edge stores the Euclidean distance between two nodes connected to it.

[0016] As a preferred improvement, the updating process of the topology map is expressed as follows:

[0017] For any new path point, calculate its Euclidean distance to all nodes in the topological map, using a pre-set distance threshold To determine the type of the new waypoint:

[0018] If the distance between the new path point and all nodes is greater than the distance threshold , then the path point is classified as a new observation node;

[0019] If the distance between the new path point and a node is less than the distance threshold , and the node is a visited node, delete the new path point and add an edge between the current node and the visited node;

[0020] If the distance between the new path point and a node is less than the distance threshold , and the node is an observation node, the observation node is deleted, and the new path point is classified as a new observation node to update the observation node.

[0021] As a preferred improvement, in step S3, the specific execution process of the sub-instruction generation algorithm includes the following steps: cleaning through predefined rules, deleting escape characters in the original instruction, and then dividing the cleaned original instruction into multiple sub-instructions through a sentence marker, and finally using a rule set containing heuristic rules to refine the sub-instructions according to the part-of-speech tagging results.

[0022] As a preferred improvement, in step S3, "extracting features from the original instruction and the sub-instruction respectively, and using the extracted two features together as instruction encoding" specifically includes the following process:

[0023] For the feature extraction of the original instruction, the position embedding of each word in the original instruction sentence and the type embedding of the text are added, and then the embedding information is input into the multi-layer Transformer model to extract the original instruction features. For the feature extraction of sub-instructions, the text Transformer module in the CLIP model is used as the instruction encoder to obtain the sub-instruction features, and the original instruction features and the sub-instruction features are used together as the instruction encoding.

[0024] As a preferred improvement, step S4 specifically includes the following steps:

[0025] Step S41, adding a position code and a navigation step code to each node in the topological map, wherein the position code is to embed the node position into the egocentric view; the navigation step code is to assign a time step for each node to visit; at the same time, a stop node for indicating a stop action is added to the topological map, and the stop node is connected to other nodes;

[0026] Step S42, sending the node and instruction encoding with added position encoding and navigation step encoding into a multi-layer cross-modal Transformer model for interaction, and predicting the navigation target score of each node;

[0027] Step S43, shielding the navigation scores of the current node and the visited nodes. The intelligent agent selects the node with the highest navigation score according to the navigation scores of the observation node and the stop node. If an observation node is selected, the shortest path to the target location is calculated using the Dikjstra algorithm. If a stop node is selected, the current navigation is terminated.

[0028] As a preferred improvement, each layer of the multi-layer cross-modal Transformer model consists of a bidirectional cross-attention layer and a self-attention layer for graphics perception, which respectively model the relationship between nodes and instructions and encode the environment layout. The execution process of the self-attention layer for graphics perception is expressed as:

[0029] ;

[0030] In the formula, represents the operation process of the self-attention layer for graph perception; X represents nodes; E represents the spatial matrix of all-pair shortest distances obtained from the edges of the topological map, All are learnable matrices; d Indicates the number of points; b e Indicates the offset; T Represents matrix transpose; Indicates the use of softmax Function activation;

[0031] The navigation score of each node in the topological map is expressed as:

[0032] ;

[0033] In the formula, Indicates j The navigation score of the node; Indicates that a two-layer feedforward network is used for processing; Represents the output of the multi-layer cross-modal Transformer model j Multimodal feature representation of nodes.

[0034] The beneficial effects of the present invention are:

[0035] (1) Based on the waypoint predictor, a topological map of the current environment is constructed through the waypoint predictor. The intelligent agent plans the path based on the nodes in the topological map, which increases the ability to capture the global environment layout. At the same time, it can well trace back to the past position, improving the long-term planning ability of the intelligent agent.

[0036] (2) The spontaneous topological mapping method based on waypoint prediction abstracts the continuous environment that is difficult to navigate and meets the navigation needs in real scenarios without any environmental experience;

[0037] (3) A sub-command generation algorithm was introduced to encode the sub-commands and original commands together. The multi-command input method improved the navigation performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 Represents the topological map of the previous time step;

[0039] Figure 2 A diagram showing the process of determining the type of the new path point D1;

[0040] Figure 3 A diagram showing the process of determining the type of the new path point D2;

[0041] Figure 4 A diagram showing the process of determining the type of a new path point D3;

[0042] Figure 5 Represents the topological map constructed at the current time step;

[0043] Figure 6 Schematic diagram representing the navigation scores of different nodes in the topological map;

[0044] Figure 7 A schematic diagram showing a training environment;

[0045] Figure 8 A graph representing the agent's path through the simulator. DETAILED DESCRIPTION

[0046] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Please refer to Figure 1-Figure 8 The present invention provides a visual language navigation method based on a topological map, comprising the following steps:

[0048] Step S1, collect panoramic images of the current environment observed by the agent at different viewing angles, input the panoramic images into a waypoint predictor, and predict the path points of the agent.

[0049] Specifically, the perspective for panoramic image acquisition is the perspective of the agent at 12 equidistant horizontal heading angles, namely, the perspectives corresponding to the horizontal heading angles of 0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300°, and 330°. At each perspective, an RGB image and a depth image are collected to form a panoramic image, which is expressed as: , where Represents the time step Panoramic images collected at the time; Represents the time step RGB image collected at the time; Represents the time step The depth image collected at that time.

[0050] The waypoint predictor includes a visual encoder, a Transformer model and a nonlinear classifier.

[0051] The visual encoder is used to encode the collected panoramic image into visual features, which includes an RGB feature encoder and a depth feature encoder. The RGB feature encoder is used to encode the RGB image into RGB features, which are expressed as: , where represents the viewing angle index, Indicates The RGB features of each perspective; the depth feature encoder is used to encode the depth image into depth features, expressed as: , where Indicates Because there are 12 images from different perspectives in the panoramic view, in order to distinguish the features from different perspectives, directional features are added to each perspective, expressed as: , express The directional characteristics of the viewing angle, , where Indicates The RGB features, depth features, and direction features of all 12 perspectives at each time step are combined and sent to the Transformer model in parallel.

[0052] In this embodiment, the ViT-B / 32 module in the CLIP model is used as the RGB feature encoder; the ResNet50 model pre-trained on point target navigation is used as the deep feature encoder.

[0053] The Transformer model uses a self-attention mechanism to model the spatial relationship in the image, and infers adjacent path points based on the spatial relationship in the image to obtain contextual deep embedding. The Transformer model has a two-layer structure, with 12 self-attention heads in each layer.

[0054] The nonlinear classifier is a multilayer perceptron, which is used to project the embedded information output by the Transformer model to the probability of adjacent path points in space, obtain a heat map representing the probabilities of adjacent path points in space, and use non-maximum suppression (NMS) to process the heat map to obtain multiple adjacent path points.

[0055] Step S2, based on the path points predicted by the waypoint predictor, mark the path point where the agent is currently located as the current node, and the agent can access the panoramic image observed at the current node; mark the path points near the current node as observation nodes, indicating the reachable locations near the agent; mark the path points that the agent has visited as visited nodes; any two adjacent nodes are connected by an edge, and all nodes and edges together constitute a topological map of the current environment.

[0056] Time step t The constructed topological map is represented as ,in Represents the nodes in the topology map. There are three types of nodes: current node, observation node, and visited node. Represents the edges in the topological map, and each edge stores the Euclidean distance between two adjacent nodes.

[0057] Specifically, the way to determine the type of a path point is: for any new path point, the Euclidean distance between it and all nodes in the topological map is calculated, and the type of the new path point is determined by a pre-set distance threshold, which includes:

[0058] If the distance between the new path point and all nodes is greater than the distance threshold , then the path point is classified as a new observation node;

[0059] If the distance between the new path point and a node is less than the distance threshold , and the node is a visited node, delete the new path point and add an edge between the current node and the visited node;

[0060] If the distance between the new path point and a node is less than the distance threshold , and the node is an observation node, the observation node is deleted, and the new path point is classified as a new observation node to update the observation node.

[0061] Figure 1-Figure 5 The process of determining the types of new waypoints D1, D2, and D3 is shown, where Figure 1 Represents the topological map of the previous time step; Figure 2 A diagram showing the process of determining the type of the new path point D1; Figure 3 A diagram showing the process of determining the type of the new path point D2; Figure 4 It represents the process of determining the D3 type of the new path point. Figure 5 Indicates the topological map constructed at the current time step. In the figure, A represents the current node, B represents the observed node, and C represents the visited node.

[0062] At each time step t The topological map is constructed by self-organizing the waypoints predicted by the waypoint predictor, without assuming prior knowledge of the environment structure. t As the process progresses, the topology map of the current environment is gradually updated.

[0063] Step S3, introduce the sub-instruction generation algorithm, convert the original instructions consisting of long sentences obtained by the intelligent agent into sub-instructions consisting of short sentences, extract features from the original instructions and sub-instructions respectively, and use the two extracted features together as instruction encoding.

[0064] The original instructions are provided by a third party that interacts with the agent. When the agent is trained, the original instructions are provided by an instruction database; when the agent is running in an actual environment, the original instructions are provided by the user who controls the agent.

[0065] Original instructions usually consist of long sentences of a certain length. When extracting features from original instructions, the problem of distraction is likely to occur due to the excessive length of the sentences. Therefore, the present invention introduces a sub-instruction generation algorithm, cleans up through predefined rules, deletes escape characters in the original instructions, and then divides the cleaned original instructions into multiple sub-instructions through a sentence marker. Finally, a rule set containing heuristic rules is used to refine the sub-instructions according to the part-of-speech tagging results, so as to more accurately determine where to start a new sub-instruction, and solve the defect that the sentence marker is prone to sentence segmentation failure when facing ambiguous expressions. The introduction of the sub-instruction algorithm can eliminate some unnecessary words, reduce the influence of noise words, and obtain a more accurate context representation in the feature extraction process.

[0066] For the feature extraction of the original instruction, the position embedding of each word in the original instruction sentence and the type embedding of the text are added, and then these embedding information are input into the multi-layer Transformer model to extract the original instruction features; for the feature extraction of sub-instructions, the text Transformer module in the CLIP model is used as the instruction encoder to obtain the sub-instruction features, and the original instruction features and the sub-instruction features are used together as the instruction encoding.

[0067] In step S4, the topological map and instruction encoding are used to perform cross-modal fusion at the node level, predict the position of the agent under the topological map, and guide the agent to move to the target location along the specific path required in the original instruction.

[0068] Step S4 specifically includes the following steps:

[0069] Step S41, adding a position code and a navigation step code to each node in the topological map, wherein the position code is to embed the node position into the egocentric view; the navigation step code is to assign a time step for each node to visit; at the same time, a stop node for indicating a stop action is added to the topological map, and the stop node is connected to other nodes;

[0070] Step S42, sending the node and instruction encoding with added position encoding and navigation step encoding into a multi-layer cross-modal Transformer model for interaction, and predicting the navigation target score of each node;

[0071] Step S43, shielding the navigation scores of the current node and the visited nodes. The intelligent agent selects the node with the highest navigation score according to the navigation scores of the observation node and the stop node. If an observation node is selected, the shortest path to the target location is calculated using the Dikjstra algorithm. If a stop node is selected, the current navigation is terminated.

[0072] Specifically, position encoding is to add the direction and Euclidean distance of any node relative to the current node to the visual features; in navigation step encoding, the navigation step code of the observed node is assigned to 0, which can encode the visited nodes into different navigation histories and promote alignment with the original instructions.

[0073] The standard self-attention layer only considers the visual similarity between nodes, and often ignores the more relevant neighboring nodes of distant nodes. To improve this situation, the present invention uses a graph-aware self-attention layer (GASA), which further considers the graph topology when calculating attention. The execution process of the graph-aware self-attention layer is It is expressed as:

[0074] ;

[0075] In the formula, represents the operation process of the self-attention layer for graph perception; X represents nodes; E represents the spatial matrix of all-pair shortest distances obtained from the edges of the topological map, All are learnable matrices; d Indicates the number of points; b e Indicates the offset; T Represents matrix transpose; Indicates the use of softmax Function activation;

[0076] The navigation score of each node in the topological map is expressed as:

[0077] ;

[0078] In the formula, Indicates j The navigation score of the node; Indicates that a two-layer feedforward network is used for processing; Represents the output of the multi-layer cross-modal Transformer model j Multimodal feature representation of nodes.

[0079] In the VLN task, there is no need to visit the same node repeatedly, so the navigation scores of the current node and the visited nodes are masked.

[0080] After the model is built, it needs to be trained. The training of the model includes three stages: the first stage is the pre-training of the topological map; the second stage is to estimate the strategy through imitation learning; and the third stage is to fine-tune the overall strategy through long-term target sampling strategy.

[0081] In the first stage, the input topology map is constructed offline and derived from a predefined graph used by the Matterport 3D simulator. Given a discrete expert trajectory, a topology map is generated by marking the current node, visited nodes, and observed nodes along the trajectory, while inheriting node positions and edges from the predefined graph. In addition, the rendered RGBD images in the Habitat simulator are aligned to the predefined graph for feature mapping during the topology map construction process. Masked Language Modeling (MLM) and Single Action Prediction (SAP) agent tasks are used for pre-training. In the MLM task, the input instructions are randomly masked, and the planning module is optimized by recovering the masked words after the mapping-instruction interaction. In the SAP task, random blocks are input to the expert trajectory and their corresponding topology graphs are constructed. The goal of this task is to predict the next teacher action, that is, the subsequent action node of the block trajectory. The loss is defined as follows:

[0082] ;

[0083] In the formula, represents the loss function; t Represents every moment; T represents the number of maximum moments; represents the cross entropy function; Indicates the student's actions; Indicates the overall teacher's actions.

[0084] In the second stage, an interactive demonstrator similar to the Dagger algorithm is used. The demonstrator accesses the real 3D topological map of the ground and selects the observation node with the shortest distance to the final target as the teacher node. Then the overall learning strategy is formulated as:

[0085] ;

[0086] In the formula, Indicates steps t The teacher operation node in represents the loss function; Indicates the instruction set; Represents the topology structure.

[0087] In order to avoid overfitting expert experience, the "student forcing" algorithm is chosen to train the model, sampling the long-term goal of each step by predicting the probability distribution of the score. In each decision link, the topology map is updated, and cross-modal mapping reasoning is used to predict the long-term goal of the agent, and then guide the agent to the target location.

[0088] In the third phase, the agent always runs the mapping-planning-control loop, using the long-term goal sampling strategy in this phase. The agent greedily selects the observation node with the largest prediction score. If the maximum action steps are exceeded or the "stop" node is triggered, the navigation is terminated. Example 1

[0089] Based on the continuous environment, the visual language navigation method based on the topological map provided by the present invention is used to conduct navigation experiments. The schematic diagram of the training environment is shown in FIG. Figure 7 The experimental dataset is the R2R-CE dataset, which is generated by converting the discrete paths of R2R into a continuous environment through HabitatSimulator. It provides 16,844 path instruction pairs of more than 90 visually realistic scenes on the Matterport3D dataset.

[0090] The experiment is divided into three parts: training, verification and testing. The average path length of each trajectory is 9.86m and the average instruction length is 32 words.

[0091] The evaluation indicators used in the experiment include: success rate SR, trajectory length TL, navigation error NE, Oracle success rate OSR and path length weighted success rate SPL.

[0092] Among them, the success rate SR is used to evaluate the probability of successful arrival of navigation, and its definition is as follows:

[0093] ;

[0094] In the formula, N Indicates the number of test data; k Represents any test data; Indicates the docking coordinates of the agent;

[0095] The trajectory length TL is used to measure the step length required for the agent to reach the destination and is defined as follows:

[0096] ;

[0097] In the formula, m represents the target point coordinates, d th Represents the navigation distance of the agent;

[0098] The navigation error NE is used to measure the shortest distance between the agent and the target point, and is defined as follows:

[0099] ;

[0100] In the formula, d m Target node location dk Indicates the actual node position.

[0101] The Oracle Success Rate (OSR) is the success rate of the horizontal Oracle stop strategy.

[0102] The path length weighted success rate SPL is used to measure the acceptance of the navigation success rate and is defined as follows:

[0103] ;

[0104] In the formula, d s Indicates the distance between the starting point and the target point of the current navigation.

[0105] All experiments were conducted based on the PyTorch framework and the Habitat simulator, and were executed on two NVDIA RTX 3090ti. For visual encoding, the ViT-B / 32 version of CLIP was selected as the RGB image encoder. The projected RGB features, depth features, and dimensions of this feature were all 256. The number of layers of the panoramic encoder, text encoder, and cross-modal image encoder were set to 2, 9, and 4, respectively; in the pre-training stage, the pre-trained LXMERT on the R2R-CE dataset was selected to initialize the model, and the AdamW optimizer was used for pre-training 100,000 times, with a batch size of 64 and a learning rate of 5e-5. At this stage, the topology map was constructed offline and exported from the predefined map of the discrete VLN. After that, the discrete path of R2R was used for pre-training, and the Prevalent synthesis instruction was used to enhance the data.

[0106] After training, the weights of the checkpoint with the best model weights are selected to initialize the imitation learning phase. Training is performed by imitation learning, which means that the model tries to imitate the expert behavior in the dataset, so a simple cross entropy loss is applied between the ground truth behavior and the prediction of the agent as the learning goal. AdamW is used as the training optimizer. All agents are trained using progress sampling. The transition from teacher forcing to student forcing is made every 5 epochs, that is, every 3000 iterations. The decay rate is set to 0.75, the learning rate is 1e-5, and the total number of iterations is 15000. For the agent navigating on the graph, the node with the shortest Dijkstra distance to the target is considered as the ground-truth. Low-level control is used to reach the waypoint, and the control instructions are executed step by step. The final best iteration is determined by the best performance of the unseen set in the validation. The results of the experiment are shown in Tables 1 and 2:

[0107] Table 1 Experimental results of various methods on the Val-Seen segmentation set of the R2R-CE dataset

[0108]

[0109] Table 2 Experimental results of various methods on the Val-UnSeen segmentation set of the R2R-CE dataset

[0110]

[0111] Combining Table 1 and Table 2, it can be seen that on the Val-Seen segmentation set, the SR index and SPL index of the present invention are very close to the current best method GridMM. On the Val-UnSeen segmentation set, the OSR index and SPL index of the present invention are 2% and 3% higher than those of GridMM, respectively. This shows that the present invention also has excellent navigation performance.

[0112] Compared with CWP-CMA and CWP-RecBERT commonly used in the field, the performance of the present invention is greatly improved. CWP-CMA and CWP-RecBERT are both models based on waypoint generators tested in continuous environments. CWP-CMA uses the original cross-modal attention mechanism to directly generate actions, while CWP-RecBERT uses the VLN-BERT module to generate path decisions. Although the present invention also uses a waypoint predictor, unlike CWP-CMA and CWP-RecBERT, the present invention dynamically and spontaneously stores waypoints in a topological map, and summarizes each waypoint through an algorithm, so that the present invention can have better backtracking and exploration capabilities in the face of unknown environments. The GASA module also strengthens the visual association between the nodes of the topological map in the model, which plays a vital role in the intelligent agent's selection of nodes that conform to the language description.

[0113] In order to further study the impact of different visual inputs on navigation performance, ablation experiments were set up. The first and fifth groups of experiments used RGB images as input; the second and sixth groups of experiments used depth images as input; the third and seventh groups of experiments used both RGB images and depth images as input. The features of the RGB images and depth images were extracted by two different pre-trained resnet-50 modules, and the two features were fused and passed into the Transformer layer to predict waypoints; the fourth and eighth groups of experiments transformed the RGB encoder into the pre-trained ViT-B / 32 module in the CLIP model, and the depth feature encoder remained the same as the third group of experiments. After that, the two features were linearly converted to the same dimension and fused and passed into the Transformer layer to predict waypoints. Groups 1-4 were tested in the Val-Seen environment set, and groups 5-8 were tested in the Val-UnSeen environment set. The performance data of the tests are shown in Tables 3 and 4 respectively.

[0114] Table 3 Performance data of different visual inputs in the Val-Seen environment set

[0115]

[0116] Table 4 Performance data of different visual inputs in the Val-UnSeen environment set

[0117]

[0118] Combining Table 3 and Table 4, it can be seen that in the val-seen environment, the 4th group of experiments achieved the best performance, and in the val-unseen environment, the 8th group of experiments achieved the best performance. This shows that using RGB images and depth images as input at the same time can improve navigation performance. The reason is that using RGB images and depth images as input at the same time can enable the waypoint predictor to judge the accessibility of waypoints through RGB information and depth information at the same time, thereby achieving better robustness in both visible and invisible environments.

[0119] In order to further study the impact of different instruction spaces under different state spaces, a smile experiment was set up. The 9th and 10th groups of experiments were local spaces, and the 11th and 12th groups of experiments were global spaces. The local space consists of the observation nodes adjacent to the agent, while the global space consists of all observation nodes on the traversal path. In the 9th and 11th groups of experiments, all sub-instructions were blocked, and the tests were conducted in the Val-Unseen environment set and the Val-Unseen environment set, respectively. The performance data of the tests are shown in Table 5.

[0120] Table 5 Performance data of global local space and sub-instruction ablation experiments

[0121]

[0122] As can be seen from Table 5, the overall planning has better performance than the local planning. The reason is that the position can be viewed from a global perspective and a self-correction policy can be provided. However, the local planning requires multiple operations for distant positions, which will result in cumulative deviations, leading to a lot of uncertainty. At the same time, the added sub-instructions are also effective. When the sub-instructions are blocked, the navigation performance has declined to a certain extent. The improvement of the sub-instructions for the global planning is less than that for the local planning, because the local planning makes better use of the high-level semantics by subdividing the instructions into sub-instructions. The global planning pays more attention to the global information and is relatively less affected.

[0123] See also Figure 8 , Figure 8The figure shows the path of the agent in the simulator. The instruction received by the simulator is "go straight past the end of the corridor and the fireplace, leave the room from the door to the left of the fireplace, and stop there." At each step, the image on the left represents the observed RGB image, and the image on the right is the path map. From the first step to the fourth step, the agent moves forward until it observes "fireplace". In the fifth step, it follows the second instruction and moves explicitly to the left. Once the direction is determined, it moves forward without hesitation. Finally, the destination is confirmed and stops at the target location, successfully completing the navigation task.

[0124] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A visual language navigation method based on topological graph, characterized in that: The steps include: Step S1, collecting panoramic images of the current environment observed by the agent at different viewing angles, inputting the panoramic images into a waypoint predictor, and predicting the path points of the agent; Step S2, based on the path points predicted by the waypoint predictor, the path point where the agent is currently located is marked as the current node, and the agent can access the panoramic image observed at the current node; the path points near the current node are marked as observation nodes, indicating the reachable locations near the agent; Mark the path points that the agent has visited as visited nodes; any two adjacent nodes are connected by an edge, and all nodes and edges together form the topological map of the current environment. The topological map is updated in real time according to the position of the agent at different time steps; Step S3, introducing a sub-instruction generation algorithm, converting the original instruction consisting of a long sentence obtained by the agent into a sub-instruction consisting of a short sentence, extracting features of the original instruction and the sub-instruction respectively, and using the extracted two features together as instruction encoding; the specific execution process of the sub-instruction generation algorithm includes the following steps: cleaning by predefined rules, deleting escape characters in the original instruction, and then dividing the cleaned original instruction into multiple sub-instructions by a sentence marker, and finally using a rule set containing heuristic rules to refine the sub-instructions according to the part-of-speech tagging results; "Extracting features from the original instruction and the sub-instruction respectively, and using the two extracted features together as instruction encoding" specifically includes the following process: For feature extraction of the original instruction, we add the position embedding of each word in the original instruction sentence and the type embedding of the text, and then input the embedding information into the multi-layer Transformer model to extract the original instruction features. For feature extraction of sub-instructions, we use the text Transformer module in the CLIP model as the instruction encoder to obtain the sub-instruction features, and use the original instruction features and sub-instruction features together as the instruction encoding. In step S4, the topological map and instruction encoding are used to perform cross-modal fusion at the node level, predict the position of the agent under the topological map, and guide the agent to move to the target location along the specific path required in the original instruction.

2. The visual language navigation method based on topological graph according to claim 1 is characterized in that: In step S1, the viewing angles for panoramic image acquisition are the viewing angles corresponding to the intelligent agent at horizontal heading angles of 0°, 30°, 60°, 90°, 120°, 150°, 180°, 210°, 240°, 270°, 300° and 330°. An RGB image and a depth image are collected at each viewing angle to form a panoramic image.

3. The visual language navigation method based on topological graph according to claim 1 is characterized in that: The waypoint predictor includes a visual encoder, a Transformer model and a nonlinear classifier, wherein the visual encoder is used to encode the collected panoramic image into visual features; The Transformer model uses a self-attention mechanism to model spatial relationships in images, infer adjacent path points, and obtain contextual deep embedding information; the nonlinear classifier is used to project the embedding information output by the Transformer model into probabilities of adjacent path points in space, obtain a heat map representing the probabilities of adjacent path points in space, and use non-maximum suppression to process the heat map to obtain multiple adjacent path points.

4. The visual language navigation method based on topological graph according to claim 3 is characterized in that: The visual encoder includes an RGB feature encoder and a depth feature encoder, wherein the RGB feature encoder is used to extract features from the RGB image to obtain an RGB feature sequence; the depth feature encoder is used to extract features from the depth image to obtain a depth feature sequence; The nonlinear classifier is a multi-layer perceptron.

5. The visual language navigation method based on topological graph according to claim 3 is characterized in that: Each node in the topological map stores visual feature information and location information; each edge stores the Euclidean distance between the two nodes connected to it.

6. The visual language navigation method based on topological graph according to claim 1 is characterized in that: The updating process of the topological map is expressed as: For any new path point, calculate its Euclidean distance to all nodes in the topological map, using a pre-set distance threshold To determine the type of the new waypoint: If the distance between the new path point and all nodes is greater than the distance threshold , then the new path point is classified as a new observation node; If the distance between the new path point and a node is less than the distance threshold , and the node is a visited node, delete the new path point and add an edge between the current node and the visited node; If the distance between the new path point and a node is less than the distance threshold , and the node is an observation node, the observation node is deleted, and the new path point is classified as a new observation node to update the observation node.

7. The visual language navigation method based on topological graph according to claim 1 is characterized in that: Step S4 specifically includes the following steps: Step S41, adding a position code and a navigation step code to each node in the topological map, wherein the position code is to embed the node position into the egocentric view; the navigation step code is to assign a time step for each node to visit; at the same time, a stop node for indicating a stop action is added to the topological map, and the stop node is connected to other nodes; Step S42, sending the node and instruction encoding with added position encoding and navigation step encoding into a multi-layer cross-modal Transformer model for interaction, and predicting the navigation target score of each node; Step S43, shielding the navigation scores of the current node and the visited nodes. The intelligent agent selects the node with the highest navigation score according to the navigation scores of the observation node and the stop node. If an observation node is selected, the shortest path to the target location is calculated using the Dikjstra algorithm. If a stop node is selected, the current navigation is terminated.

8. The visual language navigation method based on topological graph according to claim 7 is characterized in that: Each layer of the multi-layer cross-modal Transformer model consists of a bidirectional cross-attention layer and a self-attention layer for graphics perception, which respectively model the relationship between nodes and instructions and encode the environment layout. The execution process of the self-attention layer for graphics perception is expressed as: ； In the formula, represents the operation process of the self-attention layer for graph perception; X represents nodes; E represents the spatial matrix of all-pair shortest distances obtained from the edges of the topological map, All are learnable matrices; d Indicates the number of nodes; b e Indicates the offset; T Represents matrix transpose; Indicates the use of softmax Function activation; The navigation score of each node in the topological map is expressed as: ； In the formula, Indicates j The navigation score of the node; Indicates that a two-layer feedforward network is used for processing; Represents the output of the multi-layer cross-modal Transformer model j Multimodal feature representation of nodes.

Images