A method and system for visual language navigation based on path space priors

Abstract

The present application relates to a kind of visual language navigation method and system based on path space priori.In the setting that environment is known, the present application is around two kinds of path priori-shortest path priori and general path priori, respectively design the solution of path classification and sequence decision-making;Again for the more difficult setting that environment is unknown, a two-stage exploration-exploitation framework is proposed, i.e.first run exploratory strategy to build high information content environment subgraph, then select appropriate navigation model according to path priori on it.Compared with existing visual language navigation technology, the present application greatly reduces the number of decisions, thereby reducing the possibility of decision-making error;The model of the present application relies on global view when making decisions, and is not easy to mislead because of local confusion;The use of environment topology and path priori of the present application makes the design of specific model very simple, and compared with the complex reinforcement learning model used by existing technology, it has fewer parameters and clearer structure.

Description

TECHNICAL FIELD

[0001] The present application relates to a visual language navigation method, in particular to a method and system for realizing navigation based on path space prior and environment topology classification or sequence decision-making, belonging to the cross field of artificial intelligence, in particular computer vision and natural language processing. TECHNICAL BACKGROUND

[0002] With the explosive growth of data volume and the great improvement of computing power, the new generation of artificial intelligence technology represented by neural networks has developed rapidly and has been gradually applied to various industries, such as intelligent voice assistants, autonomous driving, etc. Early research on artificial intelligence was mostly focused on a single data type, such as visual data and natural language data, while recent cross-modal intelligent research that can integrate multiple different data types has attracted more and more attention, among which the typical representative is the integration of vision and natural language.

[0003] Visual language navigation is the latest technical hotspot in this integration field, which simultaneously integrates vision, language, and related technologies of robots, and thus is more difficult. The basic setting of this task is that an intelligent robot is placed in a real scene environment, which can observe the surrounding environment through the installed camera; the user will describe a route (path) in natural language, and the robot needs to receive and analyze the description, and then move in the environment according to the description, and finally correctly arrive at the target location.

[0004] At present, there have been numerous technical researches in this field, which basically adopt reinforcement learning as the basic framework, modeling the whole problem as an optimization problem of maximizing the return. Although these technologies have greatly improved the performance of this task, recent work analysis points out that the performance of this task has reached a bottleneck period in the past two years, and the improvement brought by various researches is becoming more and more subtle (Wanrong Zhu, Yuankai Qi, Pradyumna Narayana, Kazoo Sone, Sugato Basu, Xin Eric Wang, Qi Wu, Miguel P. Eckstein, William Yang Wang: Diagnosing Vision-and-Language Navigation: What Really Matters. CoRR abs / 2103.16561).

[0005] The research using reinforcement learning has the following two shortcomings:

[0006] (1) They ignore the prior information in the data, especially the spatial prior of the paths. In fact, such prior in the data is significant, e.g. in the Room-to-Room (R2R) data, all the paths are the shortest paths connecting the start and end points. In the improved Room-for-Room (R4R) data, all the paths can also be divided into several segments of the shortest paths. In the larger scale TOUCHDOWN data, the paths are generated based on the automatic routing by the Google navigation engine, which are approximately the shortest paths.

[0007] (2) They basically adopt an exploratory navigation strategy, which does not efficiently utilize the environment topology, i.e. the navigation is completely based on the guess and analysis of the intelligent robot without knowing the global or local map, which is easy to make the robot confused or lost in some positions, resulting in the final failure to reach the destination.

[0008] Although some recent work has begun to consider simultaneous mapping and navigation, trying to optimize subsequent decisions by recording the explored areas, due to their still ignoring the path prior information, the utilization of the map is also very inefficient, and the final performance is not significantly improved. SUMMARY

[0009] In view of the above two problems in the prior art, the present application aims to design a visual language navigation method and system based on path prior and environment topology. In the known environment setting, the present application designs solutions for path classification and sequence decision around two types of path prior—shortest path prior and general path prior. And for the more difficult setting of unknown environment, a two-stage exploration-exploitation framework is proposed, i.e. first run the exploratory strategy to build a high information environment subgraph, and then select the appropriate navigation model according to the path prior. Overall, the present application designs effective visual language navigation schemes for four different situations of prior and information strength.

[0010] The technical scheme adopted by the present application is as follows:

[0011] (I) Environment known, path prior is shortest path prior

[0012] The environment known as the map of the environment in which the intelligent robot operates can be obtained in advance, generally represented as a weighted graph structure. The shortest path prior is that the true path of each navigation is the shortest path from the starting point to the end point. When the environment is known and the prior is the shortest path, the shortest path between any two positions can be calculated in advance according to Dijkstra algorithm or the like before the navigation starts. Considering that the actual environmental distance is basically a real number, it can be reasonably assumed that there is no multiple shortest paths between two points with equal length. Therefore, when the starting point of the navigation is given, each possible end point uniquely corresponds to a shortest path from the starting point. Thus, the navigation problem is equivalent to a shortest path classification problem at this time.

[0013] Solving this classification problem is easier than the original task. The scheme adopted by the present application includes the following steps:

[0014] 1) Calculate the shortest path from the starting point to any point according to the environmental topology graph.

[0015] 2) According to the input natural language description, encode the description into a feature vector using a deep neural network model (first deep neural network model), i.e. obtain a language feature vector.

[0016] 3) For each shortest path calculated in step 1), construct a corresponding path representation sequence, and then encode each path representation sequence into a single feature vector using another deep neural network model (second deep neural network model), i.e. obtain a feature vector for each path.

[0017] 4) Use an additional deep neural network model (third deep neural network model) to score the language feature vector obtained in step 2) and the feature vector of each path obtained in step 3), and give a matching score for each path. Then, normalize the matching score set by Softmax.

[0018] 5) According to the environmental topology graph, find all other paths within a certain distance (such as 3 meters) from the end point of each path (set as p i ) in step 4), and add the scores of these other paths calculated in 4) to the score of the path (i.e. p i ), to obtain the matching score after fusing the scores of nearby paths. This score is more robust than the original score in 4).

[0019] 6) Finally, find the path with the maximum matching score in step 5) as the final predicted path, and let the robot move along the path.

[0020] Wherein, the above step 5) is an optional step, i.e. the step can also be omitted, and step 6) is directly performed after step 4).

[0021] Figure 1 A general flowchart of the above process is given.

[0022] (ii) The environment is known, and the path prior is a general path prior

[0023] The above consideration requires a shortest path prior, which is not always satisfied, such as the R4R dataset. When the shortest path prior is not satisfied, the above framework fails. In order to deal with this more general case, the present application designs a solution for a general path prior. The general path prior is a general path, which does not have the property of the shortest path, but despite this, any general path can always be divided into several connected shortest paths. The correctness of this conclusion is obvious, as long as we assume that the edge between any two adjacent nodes is the shortest path connecting the two nodes (this assumption is obviously true for the navigation scenario), then obviously a path can always be divided according to the continuous edges, where each edge is a shortest path, so the conclusion naturally holds.

[0024] Based on this conclusion, the model in (i) is improved from single classification to multiple sequential classification decisions. Specifically, the following steps are included:

[0025] 1) Calculate the shortest path between any two points according to the environment topology graph.

[0026] 2) According to the input natural language description, use a deep neural network model (first deep neural network model) to encode the description and retain all the corresponding language feature vectors of the intermediate steps for the improved matching score model.

[0027] 3) Initially, the robot is located at the starting point, and all candidate paths at this time are the set of shortest paths from the starting point to all other points. For these candidate paths, construct the corresponding path representation sequence, and then use another deep neural network model (second deep neural network model) to encode each path representation sequence into a single feature vector, i.e. obtain the feature vector of each path.

[0028] 4) Use an additional deep neural network model (third deep neural network model) to score each intermediate step language feature vector obtained in step 2) and each path feature vector obtained in step 3), and then aggregate all the intermediate step scores corresponding to each path to obtain a comprehensive matching score for each path, and then normalize the comprehensive matching score using Softmax.

[0029] 5) Find the path with the highest matching score according to 4) and move the robot along this path to a new position.

[0030] 6) After reaching the new position, determine whether to end or not. If yes, stop. Otherwise, find the shortest paths from the current position to all other positions, concatenate all these shortest paths after the path the robot has already walked, and get a new set of paths as the current candidate paths. Repeat steps 3) to 5) to find the next path and move the robot accordingly.

[0031] 7) Repeat step 6) until stopping.

[0032] It can be seen that the above process changes the one-time shortest path classification in (I) into multiple times of shortest path decision. Figure 2 A flowchart of this process is given.

[0033] (III) Unknown environment

[0034] Both of the above settings assume that the environment is known, which usually occurs in, for example, a long-term running domestic robot in a fixed environment, but there are also some cases where we need the robot to be used in a completely new unknown environment, so the above solution is no longer applicable. In order to reasonably migrate the above solution to the setting of unknown environment, the present application proposes an exploration-exploitation strategy. This strategy operates in two stages:

[0035] 1) First, use any exploration model (or exploration system) in the prior art to perform autonomous exploration in the environment according to the instructions.

[0036] 2) Record the map explored by the exploration model in 1) and construct a high-information subgraph (subtopology graph) of the complete environment, and regard this subgraph as the environment topology graph in (I) or (II), and then apply the method in (I) or (II) to complete the navigation.

[0037] Exploration by borrowing an exploration model in the prior art is because these models themselves do not require map information and can directly receive instructions for operation, and although their own operation may not be able to directly find the target endpoint, but through multiple explorations, there is a high probability that the target point together with the complete path will be included in the explored range, at this time the explored area is constructed into a subgraph and the model in (I) or (II) is used to efficiently utilize the topology and path prior of the environment for re-routing, significantly improving the navigation performance. Whether to use the model in (I) or (II) depends on the path prior, if the shortest path prior can be ensured, (I) is used, otherwise (II) is used. Figure 3 and 4 A flowchart of this process is given.

[0038] The application also provides a visual language navigation system based on path space priori using the method, which comprises:

[0039] a natural language description coding module, which is used for coding the input natural language description into a language feature vector;

[0040] a path coding module, which is used for coding the path representation sequence to obtain a feature vector of each path;

[0041] a description-path matching module, which is used for scoring the language feature vector and the feature vector of each path to obtain a matching score of each path;

[0042] a path selection and execution module, which is used for finding out the path with the largest matching score and making the robot move along the path.

[0043] Compared with the existing visual language navigation technology, the method has the following advantages:

[0044] 1) fewer decisions. The existing visual language navigation technology basically adopts the strategy of step-by-step prediction and movement, which leads to the number of decisions equal to the number of steps of the actual path, while the method of the application only needs one-step decision (classification) to find the predicted path under the shortest path priori, and only needs a small number of decisions under the general priori, greatly reducing the number of decisions and thus reducing the possibility of decision-making errors.

[0045] 2) The model of the application relies on the global view when making decisions, such as classifying according to the complete path under the shortest path priori, and classifying according to the longer partial path under the general priori, while the existing technology basically only makes decisions according to the current state and local neighborhood information, and the decision-making view is limited to the local, which leads to the possibility of getting lost due to local confusion.

[0046] 3) simple model. The use of environment topology and path priori in the application makes the design of specific models very simple, which has fewer parameters and clearer structure compared with the complex reinforcement learning model used in the existing technology. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 a flowchart of the scheme when the environment is known and has the shortest path priori;

[0048] Figure 2 a flowchart of the scheme when the environment is known but only has the general path priori;

[0049] Figure 3 a flowchart of the strategy when the environment is unknown and has the shortest path priori;

[0050] Figure 4Strategy flow chart for general path prior when environment is unknown. DETAILED DESCRIPTION

[0051] In order to more specifically explain the technical details and advantages of the present application, the present application is further described in detail below through examples and drawings.

[0052] 1. Input data

[0053] The input data of the present application is basically consistent with the prior art, involving two types of input, natural language and path.

[0054] For natural language, first, the sentence is tokenized and converted into individual tokens (usually single words), and then a learnable token-feature mapping table is used to find the feature vector of each token, thereby converting the natural language description into a feature sequence. In order to provide a reasonable initialization for token features, the present application uses a pre-trained 300-dimensional GloVe feature to fill the initial mapping table, and the mapping table will be further optimized in the subsequent training process.

[0055] For each path, it is represented as a discrete sequence of positions. Each position the robot will take a panoramic photo, in order to represent this panoramic picture is projected along 36 different orientations to get 36 different orientation pictures, and then use the pre-trained ResNet-152 model on Places-365 data to extract a fixed 2048-dimensional feature vector for each picture, and take the average of the 36 2048-dimensional vectors of the same position as the visual representation of the position. At the same time, in order to further encode the spatial information of each position, the orientation of the robot at each position is additionally calculated, expressed as the horizontal deflection angle φ and the pitch angle θ, and a 4-dimensional vector [sinφ, cosφ, sinθ, cosθ] is generated according to the two angles. This 4-dimensional vector is repeated 32 times to obtain an enhanced 128-dimensional direction representation vector. The 2048-dimensional visual representation vector and the 128-dimensional direction representation vector are connected, and a 2176-dimensional representation vector for each position is obtained. Thus, each path is represented as a sequence containing several 2176-dimensional vectors. Unlike language representation, these path representations are not learnable, which is to reduce the parameter quantity.

[0056] 2. Classification model for shortest path prior

[0057] The classification model proposed in the present application deals with data with the shortest path prior, which mainly includes the following four parts: natural language description encoding module, path encoding module, description-path matching module, and path selection and execution module, which are described below.

[0058] 1) Natural language description encoding module

[0059] Based on the representation sequence generated in step 1 for the preliminary representation of the natural language description, a bidirectional long short-term memory (LSTM) model is used to further encode the sequence. The long short-term memory network in each direction adopts a configuration with a hidden feature dimension of 512, and outputs a single 512-dimensional feature vector after inputting the description representation sequence. The 1024-dimensional feature obtained by splicing the respective vectors of the bidirectional directions is used as the final description feature.

[0060] 2) Path encoding module

[0061] Based on the representation sequence generated in step 1 for the preliminary representation of the path, a bidirectional long short-term memory network is also used for encoding. In the symmetric design adopted in 1), each unidirectional long short-term memory network adopts a hidden feature dimension of 512, and finally produces a 1024-dimensional path representation vector by splicing.

[0062] 3) Description-path matching module

[0063] Based on the 1024-dimensional representations generated in 1) and 2), a two-layer fully connected network using a rectified linear unit (ReLU) as the hidden layer activation function is used to generate the matching score of the two. The input dimension of the first layer of the fully connected network is 2048, i.e., the spliced dimension of two 1024-dimensional, and the output dimension is 1024. After passing through a rectified linear unit non-linear function, the input dimension of the second layer of the fully connected network is 1024, and the output dimension is 1, i.e., a one-dimensional scalar is finally generated as the matching score value of the two.

[0064] 4) Path selection and execution module

[0065] After obtaining the matching score of each candidate path by the description-path matching module in 3), the Softmax normalization is performed, and the path with the highest score is fixedly selected as the final navigation path, and the internal motion module is controlled to move along the selected path. This module is only a fixed execution strategy and does not involve any optimizable part.

[0066] 3、Sequence decision model for general path prior

[0067] The sequence decision model for general prior proposed by the present application and the classification model in step 2 have similar structures, but still have some differences. The main part still includes four parts: natural language description encoding module, path encoding module, description-path matching module, and path selection and execution module. The specific implementation is introduced as follows.

[0068] 1) Natural language description encoding module

[0069] The encoding of natural language description here takes a similar module as in the shortest path prior, i.e. using long short-term memory network. But different from the bidirectional model in the shortest path prior, here it is designed as a long short-term memory network only in the forward direction, and still keeps the hidden feature dimension as 512. This change is to make it easier to calculate the matching score of the path and each intermediate step instruction in the matching module later. Using this model, the preliminary representation sequence of natural language description in step 1 is further encoded, and the encoding features of all intermediate steps are retained. The result is still a feature sequence, and each feature dimension is 512 at this time.

[0070] 2) Path encoding module

[0071] The change to the path encoding module is similar to the change to the natural language encoding module in 1), i.e. using a long short-term memory network only in the forward direction with a hidden feature dimension of 512, in order to maintain symmetry with the module in 1). Using this network, the preliminary representation sequence of the path obtained in step 1 is encoded to obtain a single 512-dimensional representation vector. Unlike 1), here all intermediate step features are not retained, but only the last feature is used. In addition to all candidate paths, a virtual path sequence with a length of 1 is additionally introduced here, and the 2176-dimensional feature in this sequence is a learnable parameter. After encoding this virtual sequence in the aforementioned manner, a 512-dimensional representation is obtained, which is used to represent the option of the system stopping moving, i.e. a stop option is introduced for the candidate path. This representation will also be matched with the description by the matching module in 3) and give a score. If the score of this option is greater than all other paths, the entire navigation process stops.

[0072] 3) Description-path matching module

[0073] Based on the 1024-dimensional representations produced in 1) and 2), a two-layer fully connected network using a rectified linear unit (ReLU) as the hidden layer activation function is used here to produce the matching score of the two. The input dimension of the first layer of fully connected network is 2048, i.e. the spliced dimension of two 1024-dimensional, and the output dimension is 1024. After passing through a rectified linear unit non-linear function, the input dimension of the second layer of fully connected network is 1024, and the output dimension is 1, i.e. a one-dimensional scalar is finally produced as the matching score value of the two.

[0074] The description-path matching described here is divided into two steps. The first step splices the 512-dimensional representation of all intermediate steps obtained in 1) and the 512-dimensional path representation obtained in 2) respectively, and after each splicing, a two-layer fully connected network using a rectified linear unit as the hidden layer activation function is used to calculate the matching score of the path and each intermediate step instruction. The input dimension of the first layer of the fully connected layer is 1024, and the output dimension is 512. After the intermediate rectified linear unit, it is sent to the second fully connected layer, with an input dimension of 512 and an output dimension of 1. The second step aggregates the matching scores of the path and each step instruction obtained before to integrate into a single comprehensive score as the final matching score of the path and the instruction. The integration uses the following formula:

[0075]

[0076] where Aggr soft represents a soft aggregation function, and the corresponding hard aggregation function is to directly take the maximum value of all scores. s represents the final matching score of a certain path and language description, s j is the matching score of the jth step of the path and the sentence, and γ is a hyperparameter, generally set to 1.

[0077] 4) Path selection and execution module

[0078] After obtaining the score of each candidate path in 3), the Softmax normalization is performed, and the path with the highest score is fixedly selected as the next moving path, and the internal motion module is controlled to move along the selected path. If the stop option is selected, the navigation is terminated, otherwise, after moving along the path, a new position is reached to start a new round of decision making. This module is only a fixed execution strategy and does not involve any optimizable part.

[0079] 4, Model training

[0080] The basic deep neural network model is used in the present application, and before actual inference, the parameters need to be trained and optimized on the training data. The training process of each of the above two models is described here.

[0081] 1) Classification model for shortest path prior

[0082] First, the part to be trained needs to be clarified. For the classification model, the parts to be trained include: 1) symbol-representation mapping table of natural language, 2) natural language description encoding module, 3) path encoding module, and 4) description-path matching module. All parameters are optimized together.

[0083] The loss function (objective function) used in the training is the cross-entropy function (Cross-Entropy Loss) commonly used in classification tasks, and the parameters are updated by the Adam optimizer extended from the stochastic gradient descent method.

[0084] 2) Sequential decision-making models for general priors

[0085] Compared to 1), the sequence decision model requires an additional virtual path sequence representing the stopping option for training, while all parameters are still jointly optimized. The training of this model employs a two-stage approach. The first stage is based on Imitation Learning (IL) and uses teacher forcing, meaning that at each step of the decision-making process, a specific teacher decision is used as the target for optimization, and the teacher decision is always used to control the system's movement. To promote the model's most efficient decision-making, this invention designs an optimal teacher decision-making algorithm to generate teacher decisions. The steps of this algorithm are as follows:

[0086] 1) Input a real path P = (n1, n2, ..., nn) t ), where each n i Represents the i-th node (position) on the path, and inputs the topology graph of the environment G = {V, E}, where V represents the set of nodes and E represents the set of edges;

[0087] 2) Based on the topology G of the environment, use Dijkstra's algorithm to calculate the shortest path between any two locations in the environment; and initialize an empty sequence P' and a loop variable i to 1.

[0088] 3) with n i Starting from node P, along n i The path then extends backward, each time extending to a new node n. j The subsequent judgment is based on n. i to n j Is this extended path n? i to n j The shortest path, if and only if n is in P j If there are still nodes after this point, continue extending the path; otherwise, if it is not the shortest path, start from node n. i to n j-1 Add this path to P' and set i to j-1. If it is the shortest path but n j It is already the last node n in P. t Then from n i to n j This path is added to P', and i is set to j;

[0089] 4) Repeat the operation in 3) until i reaches t, then terminate. Finally, the output result is the shortest path sequence P' after path decomposition.

[0090] According to this algorithm, the original complete real path is divided into several decision sub-paths, and the model is trained according to these decisions. Specifically, each decision is still a classification problem in training, so the cross-entropy loss function combined with the Adam optimizer is still used for optimization.

[0091] After optimization using imitation learning, the model has a certain ability to imitate the decision of the teacher, but its own expansion and error correction capabilities are still limited. Therefore, the second stage of reinforcement learning (RL) training is carried out. Reinforcement learning training is different from imitation learning. Each decision is generated by the model's own prediction, and in order to maintain the exploration ability, when selecting a path, all candidate paths are weighted and randomly sampled according to their scores, rather than fixedly selecting the path with the highest score. Reinforcement learning training requires the definition of a reward function. The present application uses the CLS reward function used in the prior art (Vihan Jain, Gabriel Alexander Ku, Ashish Vaswani, Eugene Ie, Jason Baldridge: Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation. ACL(1) 2019: 1862-1872) and uses the classic REINFORCE algorithm to define the objective function. The Adam optimizer is still used to update the parameters.

[0092] 5. Model inference

[0093] The trained model can be used for inference and prediction. As mentioned earlier, considering that in practice there may be both known and unknown environments, the present application designs different strategies to deal with the two situations.

[0094] 1) Known environment

[0095] The environment is known, that is, the topological structure of the environment is known before inference. This is a relatively ideal situation and is consistent with the training process, so the model can directly infer like training. The specific process has been described in the section "SUMMARY OF THE INVENTION" and will not be repeated here.

[0096] 2) Unknown environment

[0097] Environment is unknown, that is, the topology of the environment cannot be known in advance, so the model cannot be directly applied. In order to smoothly adapt to this case, the application designs an exploration-exploitation strategy, the main idea of which is to first explore a small range of environment according to the language description by using an exploratory model, and try to explore the correct path as much as possible, and then apply the model to realize navigation based on the topological subgraph of this exploration. The specific process is also seen in the section of "SUMMARY OF THE INVENTION". This exploration-exploitation strategy is suitable for both classification models and sequence decision models, and the specific use depends on the path prior.

[0098] 6、Experimental results

[0099] In order to fully demonstrate that the technology of the application performs better than the existing technology, experiments are performed on two representative experimental data sets R2R and R4R. The former has a shortest path prior, which is used to verify the classification model; the latter only has a general prior, which is used to verify the sequence decision model. The evaluation index is the navigation success rate (SR). The model is considered successful if it finally reaches a position within 3 meters of the real target location. The success rate is the average success rate of a large number of test data, and obviously the higher the better.

[0100] Tables 1 and 2 respectively show the evaluation results on R2R and R4R. It can be seen that the method of the application significantly exceeds the existing technology on both data sets.

[0101] Table 1 is the navigation success rate of the classification model of the application compared with the best existing technology on the R2R data. A total of two data sets of R2R are evaluated—val_unseen and test_unseen. In the table, "-" indicates that the method does not report the performance on this data set.

[0102] Table 2 is the navigation success rate of the sequence decision model of the application compared with the best existing technology on the R4R data, which is evaluated on the val_unseen data set of R4R.

[0103] Speaker-Follower (Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, Trevor Darrell: Speaker-Follower Models for Vision-and-Language Navigation. NeurIPS 2018: 3318-3329), RCM (Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, Lei Zhang: Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation. CVPR 2019: 6629-6638), Self-Monitoring (Chih-Yao Ma, Jiasen Lu, Zuxuan Wu, Ghassan AlRegib, Zsolt Kira, Richard Socher, Caiming Xiong: Self-Monitoring Navigation Agent via Auxiliary Progress Estimation. ICLR (Poster) 2019), Tactical Rewind (Liyiming Ke, Xiujun Li, Yonatan Bisk, Ari Holtzman, Zhe Gan, Jingjing Liu, Jianfeng Gao, Yejin Choi, Siddhartha S. Srinivasa: Tactical Rewind: Self-Correction via Backtracking in Vision-And-Language Navigation.CVPR 2019:6741-6749), EnvDrop (Hao Tan, Licheng Yu, Mohit Bansal: Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout. NAACL-HLT (1) 2019: 2610-2621), AuxRN (Fengda Zhu, Yi Zhu, Xiaojun Chang, Xiaodan Liang: Vision-Language Navigation With Self-Supervised Auxiliary Reasoning Tasks. CVPR 2020: 10009-10019), Active Perception (Hanqing Wang, Wenguan Wang, Tianmin Shu, Wei Liang, Jianbing Shen: Active Visual Information Gathering for Vision-Language Navigation. ECCV (22) 2020: 307-322), VLN-BERT (Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, Dhruv Batra: Improving Vision-and-Language Navigation with Image-Text Pairs from the Web. ECCV (6) 2020: 259-274), SSM (Hanqing Wang, Wenguan Wang, Wei Liang, Caiming Xiong, Jianbing Shen: Structured Scene Memory for Vision-Language Navigation. CoRR abs / 2103.03454 (2021)), PTA (Federico Landi, Lorenzo Baraldi, Marcella Cornia, Massimiliano Corsini, Rita Cucchiara: Perceive, Transform, and Act: Multi-Modal Attention Networks for Vision-and-Language Navigation.CoRRabs / 1911.12377 (2019)), EGP (Zhiwei Deng, Karthik Narasimhan, Olga Russakovsky: Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation. NeurIPS 2020), OAAM (Yuankai Qi, Zizheng Pan, Shengping Zhang, Anton van den Hengel, Qi Wu: Object-and-Action Aware Model for Visual Language Navigation. ECCV (10) 2020: 303-317).

[0104] Table 1

[0105]

[0106] Table 2

[0107]

[0108] Other embodiments of the application:

[0109] 1) The long short-term memory network can be replaced by a more powerful sequence encoding model such as BERT, and the more powerful sequence encoding model can bring better performance.

[0110] 2) The description-path matching module can be replaced by a more powerful cross-modal matching model such as multi-modal BERT, and the more powerful cross-modal matching model can also bring better performance.

[0111] Based on the same inventive concept, another embodiment of the application provides an electronic device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the application.

[0112] Based on the same inventive concept, another embodiment of the application provides a computer readable storage medium (such as ROM / RAM, magnetic disk, optical disk), which stores a computer program, and the computer program is executed by a computer to realize each step of the method of the application.

[0113] The above examples are only used to illustrate the technical solutions of the present application but not to limit the present application, and the ordinary skilled in the art can modify or equivalently replace the technical solutions of the present application without departing from the principles and scope of the present application, and the protection scope of the present application should be subject to the description of the claims.

Claims

1. A method for visual language navigation based on path space priors, the method comprising: The method is suitable for the case that the environment is known and the path prior is the shortest path prior, and comprises the following steps: 1) calculating the shortest paths from the starting point to any point according to the environment topology graph; 2) encoding the input natural language description into a language feature vector by using a first deep neural network model; 3) constructing a corresponding path representation sequence for each shortest path calculated in step 1), and then encoding each path representation sequence into a single feature vector by using a second deep neural network model to obtain the feature vector of each path; 4) scoring the language feature vector obtained in step 2) and the feature vector of each path obtained in step 3) by using a third deep neural network model to obtain the matching score of each path; 5) finding the path with the maximum matching score as the final predicted path, and making the robot move along the final predicted path.

2. The method of claim 1, wherein, Step 4) also includes finding all other paths within a certain distance of the end point of each path p i from the environmental topology map, adding the matching scores of these other paths to the matching score of the path p i to obtain the matching score after fusing the matching scores of the nearby paths.

3. The method of claim 1, wherein, The first deep neural network model and the second deep neural network model are bidirectional long short-term memory networks or BERT networks; and the third deep neural network model is a two-layer fully connected network using a rectified linear unit as the hidden layer activation function.

4. A method for visual language navigation based on path space priors, the method comprising: The method is suitable for the case that the environment is known and the path prior is the general path prior, and comprises the following steps: 1) calculating the shortest paths between any two points according to the environment topology graph; 2) encoding the input natural language description by using a first deep neural network model, and retaining the language feature vectors corresponding to all intermediate steps; 3) initially, the robot is located at the starting point, and at this time, all candidate paths are the shortest path set from the starting point to all other points; constructing a corresponding path representation sequence for these candidate paths, and then encoding each path representation sequence into a single feature vector by using a second deep neural network model to obtain the feature vector of each path; 4) scoring each intermediate step language feature vector obtained in step 2) and the feature vector of each path obtained in step 3) by using a third deep neural network model, and then aggregating the matching scores of all intermediate steps corresponding to each path to obtain the comprehensive matching score of each path; 5) finding the path with the maximum comprehensive matching score in step 4) to make the robot move along the path to a new position; 6) after reaching the new position, determining whether to end, and if not, finding the shortest paths from the current position to all other positions, concatenating all these shortest paths after the path already walked by the robot to obtain a new path set as the current candidate paths, and then repeating steps 3) to 5) to find the next path and correspondingly move the robot; 7) repeating step 6) until stopping.

5. The method of claim 4, wherein, In step 3), a virtual path sequence is additionally introduced in addition to all candidate paths, and the representation obtained by encoding the virtual path sequence is used to represent the option of stopping moving, that is, a stop option is introduced for the candidate paths.

6. The method of claim 4, wherein, The first deep neural network model and the second deep neural network model are forward long short-term memory networks or BERT networks; and the third deep neural network model is a two-layer fully connected network using a rectified linear unit as the hidden layer activation function.

7. A visual-language navigation method based on path space priors, characterized in that, The method is suitable for the case that the environment is unknown, and comprises the following steps: According to the instruction, autonomous exploration is performed in the environment by using any of the exploratory models; A map explored by the exploratory model is recorded, and a sub-topological map of a complete environment is constructed, the sub-topological map is taken as an environment topological map, and the method in any of claims 1-6 is used for navigation.

8. A path space prior based visual language navigation system employing the method of any one of claims 1 to 7, characterized by, Comprise: A natural language description coding module, configured to code an input natural language description into a language feature vector; A path coding module, configured to code a path representation sequence to obtain a feature vector of each path; A description-path matching module, configured to score the language feature vector and the feature vector of each path to obtain a matching score of each path; A path selection and execution module, configured to find a path with the largest matching score and make the robot move along the path.

9. An electronic device, comprising: Comprise a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, the computer program comprises instructions for executing the method in any of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, when the computer program is executed by a computer, the method in any of claims 1-7 is realized.

Images