Distance-aware transformer visual language navigation method

Abstract

The application discloses a visual language navigation method based on distance perception of a Transformer, and belongs to the technical field of visual language cross-modal technology.The algorithm is realized in the following manner: firstly, visual information, instruction information and memory structure of a perceivable region of an intelligent agent are initialized, then a scene memory updating module based on a graph data structure is provided, and exploration information in a navigation process is further fused by combining a language visual multi-modal pre-training model, so that the perception ability of the intelligent agent to the environment is strengthened; a distance-based progress monitor is provided to compress the action space of each step decision in the navigation process, so as to reduce the operation resources and speed up the model training; a dynamic distance fusion module is provided to integrate distance information into action decision, so that the algorithm can perform global exploration while taking into account the exploration path length, and the efficiency of the navigation task is improved.The visual language navigation method based on distance perception of the Transformer provided by the application can obviously improve the exploration efficiency of the scene memory algorithm while ensuring a good navigation success rate.

Description

Technical Field

[0001] This invention relates to the field of visual language cross-modal technology, specifically to a distance-aware Transformer visual language navigation method. Background Technology

[0002] Visual Language Navigation (VLN) aims to train an agent to reach a target location by combining natural language instructions and visual information observed by the agent in an unstructured, invisible environment by performing a series of actions.

[0003] Visual language navigation tasks require intelligent agent algorithms to have comprehensive capabilities in natural language understanding, visual environment perception, multimodal feature alignment, and reasonable policy decision-making.

[0004] Most VLN navigation algorithms primarily use a sequence-to-sequence (Seq2Seq) framework [1], utilizing a long short-term memory (LSTM) neural network to model the temporal state of the agent's navigation, which is used to process the language and image information flow during the navigation process. This operation prevents the agent from directly accessing historical information in the navigation process, such as the spatial layout of visited locations and previous decisions. Another type of agent algorithm stores the memory of explored scenes in a graph structure during the navigation process, thereby expanding the local action space to the global action space, helping the agent evaluate all navigable locations, and thus possessing a powerful ability to correct errors in a timely manner.

[0005] The existing method has the following two problems:

[0006] 1. The rapid expansion of the global action space will significantly reduce the convergence speed of the model. On the other hand, too many candidate positions will cause the model to consume a lot of GPU resources during training.

[0007] 2. The algorithms generally lack backtracking constraints, which easily leads to a large number of invalid and repeated paths, resulting in high navigation path lengths and reduced navigation efficiency. Summary of the Invention

[0008] This invention proposes a distance-aware Transformer-based visual language navigation method. It enhances the agent's environmental perception by providing a scene memory update module based on graph data structures and integrating exploration information from the navigation process with a multimodal pre-trained language vision model. Furthermore, it reduces computational resources and accelerates model training by providing a distance-based progress monitor to compress the action space of each decision step during navigation. Finally, it incorporates distance information into action decisions through a dynamic distance fusion module, enabling the algorithm to consider path length while conducting global exploration. This approach balances thorough exploration and timely error correction while further reducing unnecessary selection of distant candidate locations, thereby improving the efficiency of language vision navigation tasks.

[0009] To achieve this objective, the present invention adopts the following technical solution:

[0010] A distance-aware Transformer-based visual language navigation method is provided, characterized by the following steps:

[0011] S1 initializes navigation based on the navigation task and visual information in the perceived environment. The main component initialized is the scene memory structure. Navigation module, navigation status Navigation command information ;

[0012] S2, based on the scene memory structure at the current navigation progress t. Extract visual information of the current location Distance information Based on the visual and spatial information of the candidate positions, a global action space corresponding to the current position is constructed.

[0013] S3, using a progress monitor The global action space is filtered, and the top n candidate positions with the highest evaluation progress are selected as the actual action space in the action decision-making process.

[0014] S4, this visual information Command information Distance information The input is a distance-aware Transformer navigation module with a dynamic distance fusion module, which updates the agent's navigation state. Output the distance-weighted action probability distribution corresponding to the current position. And select the candidate node with the highest probability as the next action;

[0015] S5, update the scene memory structure based on the visual features observed after the action is performed. arrive ;

[0016] S6. Repeat S2-S5 until the agent algorithm determines that the navigation task is completed or the maximum number of steps K is reached.

[0017] As a preferred method, navigation initialization is performed using the following approach:

[0018] A1, Construct a directed graph based on the visual features of the observable range at the starting point. , where nodes Corresponding to the revelation position, edge The spatial relative positional relationship between positions u and s, in this case, mainly refers to the spatial positional relationship between the starting position and the candidate position;

[0019] A2, replaces the parameters of the randomly initialized Transformer model in the navigation module with the parameters of the pre-trained language model;

[0020] A3, based on natural language instruction text The sentence will be segmented according to the sentence structure of the original instruction using the [SEP] marker, and a [CLS] marker will be added at the beginning position to handle loop states during navigation. The preprocessed sequence will then be processed through a multi-layer Transformer network to obtain the initial instruction information as shown in formula (1). With navigation status :

[0021]

[0022] As a preferred option, memory structure based on scenario Construct the global action space corresponding to the current progress t, as shown in formula (2):

[0023] In formula (2), This represents the number of locations the agent has visited at the current progress point.

[0024] This represents the candidate positions that are directly connected to the visited position u, i.e., a subset of the action space at the current progress.

[0025] Representative scene memory structure Stored visual and spatial relative location information of candidate locations.

[0026] Preferably, through a progress monitor The global action space is filtered as follows (3):

[0027]

[0028] In formula (3), This represents the navigation state of the agent in the previous step;

[0029] Visual information representing the k-th candidate position.

[0030] The final action space obtained after filtering is

[0031] As a preferred option, a progress monitor Integrating the navigation state of the agent from the previous step With the currently observed visual information The current navigation progress is estimated as follows (4):

[0032]

[0033] In formula (4), Represents the Sigmoid activation function;

[0034] , This is the parameter matrix of a learnable fully connected neural network.

[0035] As a preferred option, n = 20.

[0036] As a preferred option, distance information One-hot encoding is used to encode the distance between the candidate position and the current position as a 20-dimensional 0-1 vector.

[0037] As a preferred option, the distance-aware Transformer navigation module based on the dynamic distance fusion module receives visual information. Command information Distance information Then, based on the following steps, an action decision is made at progress position t and the current navigation state is updated:

[0038] B1 calculates the overall attention score for language, visual, distance information, and navigation state vector, and the language vector. It also calculates the attention score and attention weight for all language features.

[0039] B2 uses the attention weights corresponding to language and vision to obtain language and vision weighted information respectively, and performs cross-pattern matching by dot product and concatenates it with the navigation state vector output by the last layer of Transformer to map to a new state representation;

[0040] B3, concatenates spatial information onto the newly generated navigation state vector and maps it to the final navigation state representation;

[0041] B4 uses visual and distance-related attention scores as the action probability distribution and distance weight distribution, respectively.

[0042] B5 combines the action probability distribution with the distance weight distribution to obtain a distance-weighted action probability distribution.

[0043] As a preferred approach, the agent algorithm is trained using a hybrid learning model that combines reinforcement learning and imitation learning, and optimized using the following formula (5):

[0044]

[0045] In formula (5), The action obtained by the algorithm based on probability sampling; The reward corresponding to the action; This represents the correct action; The coefficients represent the imitation learning loss function. The coefficients represent the distance loss function; The coefficients represent the loss function corresponding to the progress monitor.

[0046] As a preferred embodiment, the intelligent agent algorithm scene memory update specifically involves: in navigation step t, from... After selecting a candidate position u, the algorithm adds a new node to the scene memory structure to represent u. When the agent reaches position u, The navigable node corresponding to u will be deleted and added to the navigable node added after reaching position u. At the same time, the visual information and spatial relative information in the graph will be updated according to the navigation state vector.

[0047] As a preferred option, K = 15.

[0048] The present invention has the following beneficial effects:

[0049] 1. By providing a scene memory update module based on graph data structure and combining language vision multimodal pre-trained models to integrate exploration information during navigation, the agent's ability to perceive the environment is enhanced.

[0050] 2. By providing a distance-based progress monitor, the action space for each decision step in the navigation process is compressed, reducing computational resources and accelerating model training;

[0051] 3. By providing a dynamic distance fusion module, distance information is integrated into action decision-making, enabling the algorithm to consider the path length while conducting global exploration, thereby improving the efficiency of navigation tasks. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly described below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0053] Figure 1 This is a diagram illustrating the implementation steps of the distance-aware Transformer-based visual language navigation method provided by the present invention.

[0054] Figure 2 This is a system architecture diagram of the distance-aware Transformer-based visual language navigation method of the present invention; Detailed Implementation

[0055] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0056] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual images. They should not be construed as limiting the scope of this patent. To better illustrate the embodiments of the present invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual dimensions of the product. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0057] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "inner," and "outer" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present patent. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0058] In the description of this invention, unless otherwise explicitly specified and limited, the term "connection" or similar designation indicating a connection between components should be interpreted broadly. For example, it can refer to a fixed connection, a detachable connection, or an integral part; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can refer to the internal communication between two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0059] This invention provides a distance-aware Transformer-based visual language navigation method, such as... Figure 1 This includes the following steps:

[0060] S1 initializes navigation based on the navigation task and visual information from the perceived environment, primarily initializing the scene memory structure. Navigation module, navigation status Navigation command information For scene memory structure Construct a directed graph based on the visual features of the observable range at the starting point. , where nodes Corresponding to the starting position, edge The spatial relative positional relationship between positions u and s mainly refers to the spatial relationship between the starting position and the candidate position. For the navigation module, the parameters of the randomly initialized Transformer model in the navigation module are replaced with the parameters of the pre-trained language model (this invention uses PREVALENT pre-trained model parameters). For navigation status and navigation command information, the parameters are based on the natural language command text. The sequence is segmented according to the sentence structure of the original instruction using the [SEP] segmentation marker, and a [CLS] marker is added at the beginning position to handle loop states during navigation. The preprocessed sequence is then passed through a multi-layer Transformer network to obtain the initial instruction information. With navigation status ;

[0061] S2, based on the scene memory structure at the current navigation progress t. Extract visual information of the current location Distance information Based on the visual and spatial information of the candidate positions, a global action space corresponding to the current position is constructed.

[0062]

[0063] In formula (2), This represents the number of locations the agent has visited at the current progress point.

[0064] This represents the candidate positions that are directly connected to the visited position u, i.e., a subset of the action space at the current progress.

[0065] Representative scene memory structure Stored visual and spatial relative location information of candidate locations.

[0066] S3, using a progress monitor global action space The process involves screening and selecting the top n candidate positions with the highest evaluation progress as the action space in the action decision-making process. (n represents the size of the action space after filtering, which is an important parameter affecting the convergence speed and computational resource consumption during algorithm training. Through repeated experimental data comparison, n=20 can accelerate the algorithm's convergence speed while ensuring navigation success rate and reducing computational resource consumption, enabling a model that requires 8 Tesla V100 graphics cards to train can be trained on a single 2080Ti graphics card.) The progress monitor mainly utilizes the previous navigation state. The visual features currently observed The current navigation progress is evaluated as shown in formula (4):

[0067]

[0068] In formula (4), Represents the Sigmoid activation function;

[0069] , This is the parameter matrix of a learnable fully connected neural network.

[0070] S4, this visual information Command information Distance information The input is a distance-aware Transformer navigation module with a dynamic distance fusion module, which updates the agent's navigation state. Output the distance-weighted action probability distribution corresponding to the current position. And select the candidate node with the highest probability as the next action, as shown in the following formula (6):

[0071]

[0072] Specifically, this application uses and They represent The state vector and language vector of the k-th attention head and the output of the l-th layer in the multi-layer Transformer structure. The attention scores of all language features are as follows: Formula (7):

[0073]

[0074] Subsequently, the model averages the scores of all attention heads (K = 12) and uses a Softmax function to obtain the overall attention weights for the language features, as shown in the following formula (8):

[0075]

[0076] Similarly, the visual feature attention score and weight are respectively and The distance feature attention score and weight are respectively and .

[0077] Next, the model performs a weighted summation of the input text features and visual features to obtain the weighted features, as shown in the following formula (9):

[0078]

[0079] Subsequently, the model performs cross-pattern matching between the weighted language and visual features using dot product pairs and... The navigation state vector output by the last layer The mapping to the new state representation is as follows (10):

[0080]

[0081] Finally, the model incorporates directional features. The vector is concatenated to the newly generated navigation state vector and mapped to the final navigation state representation, as shown in formula (11):

[0082]

[0083] Then, the model uses the distance feature attention score and the visual feature attention score to weight the distance-weighted action probability distribution, as shown in the following formula (12):

[0084]

[0085] This paper uses reinforcement learning and imitation learning to train the agent. In reinforcement learning, this paper mainly uses the A2C algorithm, where the agent, at each step t of navigation, calculates the probability of the action output by the agent. The distance from the corresponding location to the task endpoint is used as the reward for the corresponding action. In imitation learning, the agent navigates along a real trajectory by following the correct actions at each position and calculates the cross-entropy loss for each decision. To train the distance-aware module, we compute the shortest path distance vector from each candidate position in the scene memory graph to the current navigation position. and the distance weight vector output by the model. The dot product is summed as part of the training loss to help the agent comprehensively consider the combined effects of language, vision, and distance features on action decisions.

[0086] Specifically, the loss function for training the agent can be expressed as:

[0087]

[0088] In formula (5), The action obtained by the algorithm based on probability sampling; The reward corresponding to the action; This represents the correct action; The coefficients represent the imitation learning loss function; The coefficients represent the distance loss function; The coefficients corresponding to the progress monitor loss function; This represents the actual progress of the navigation task; This represents the estimated task progress obtained from the progress monitor.

[0089] S5, update the scene memory structure based on the visual features observed after the action is performed. arrive Specifically: in navigation step t, from After selecting a candidate position u, the algorithm adds a new node to the scene memory structure to represent u. When the agent reaches position u, The navigable node corresponding to u will be deleted and added to the navigable node added after reaching position u. At the same time, the algorithm updates the visual and spatial information in the map based on the landmarks and action-related text features in the language instructions and the current navigation state, as shown in the following formulas (13) and (14):

[0090]

[0091]

[0092] in The current navigation status. and These are the representations of landmarks and actions in the text, used to analyze the landmark and action features that the agent is interested in from the instructions. Next, the model concatenates and maps the navigation state with the text representations of landmarks and actions to obtain the landmark perception state and the action perception state, as shown in formula (15):

[0093]

[0094] Next, for the nodes and edges in the scene memory structure, the stored visual and spatial information are updated using the attention mechanism, as shown in formulas (16) and (17):

[0095]

[0096]

[0097] Subsequently, to comprehensively consider the visual and spatial information corresponding to nodes and edges in the graph, this invention employs an iterative update method based on long-term inference. This method iteratively updates visual and orientation features by exchanging messages between nodes in the form of trainable functions. Specifically, in the s-th iteration, for each node u, the visual information... , Update as follows: (18)(19)

[0098]

[0099]

[0100] The update function It is implemented using a gated recurrent unit (GRU). After S aggregation iterations, the model further improves visual and orientation features, i.e. , It is achieved by capturing information within the S-hop neighborhood of node u.

[0101] S6. Repeat steps S2-S5 until the agent algorithm determines that the navigation task is complete, or the maximum number of steps K is reached. (Based on existing research experience, the maximum number of steps K is chosen as 15.)

[0102] To verify the performance of this application in the language vision navigation task, the model performance was evaluated using the Room-to-Room dataset. The dataset consists of 14,025 to 1,020,2349 navigation tasks from the training set, validation set (known environment), and validation set (unknown environment). The experimental process primarily focuses on the following five evaluation metrics to assess the performance of the VLN algorithm:

[0103] (1) Success Rate (SR): This refers to the percentage of navigation tasks in the total number of tasks in which the agent stops within 3 meters of the target position. This metric is the most direct indicator for evaluating the navigation performance of an agent.

[0104] (2) Navigation error (NE) refers to the shortest distance between the agent's stopping position and the target position.

[0105] (3) Trajectory length (TL) is the average total length of the navigation trajectory of an agent. To some extent, it can reflect the efficiency of the agent's navigation.

[0106] (4) Global success rate (OR) refers to the proportion of navigation tasks in which the agent has a position distance of less than 3 meters from the target position during the navigation process.

[0107] (5) Path-weighted success length (SPL)

[31] is a trade-off between SR and TL.

[0108] In the implementation method, the learning rate of the model is fixed throughout the training process. The AdamW optimizer was used. For data, a hybrid dataset of room-to-room and augmented data from PREVALENT was chosen to train the model. Visual information in the environment was encoded by a ResNet-152 network pre-trained on the Places365 dataset. The training batch size was set to 8, and model training was performed on a single NVIDIA 2080Ti GPU.

[0109] This application compares the performance of sequence-to-sequence models and scene memory-based models, and the comparison results are shown in Table 1 below. Compared to navigation decisions based on the local action space of the current position (rows 1-4 in Table 1) and scene memory-based models (rows 5-7 in Table 1), Table 1 shows that the scene memory-based agent algorithm has a significantly higher navigation success rate. However, due to the lack of appropriate constraints on the exploration process, the navigation path length TL is too long, significantly reducing the path length-weighted success rate (SPL). The distance-aware Transformer visual language navigation method provided in this application effectively maintains the original success rate of such algorithms while significantly reducing the navigation path length TL and navigation error NE, thereby significantly improving the path length-weighted success rate (SPL) of the scene memory-based agent and greatly improving navigation efficiency.

[0110]

[0111] Table 1

[0112]

[0113] Continued from Table 1

[0114] Meanwhile, ablation experiments were conducted on the proposed algorithm on this dataset. The experimental results are shown in rows 8-10 of Table 1 below. Here, w / o means without, w / o distance fusion means removing the distance fusion module, so that the agent can directly select candidate positions to perform actions without considering distance restrictions; w / o distance fusion and action filtering means removing the distance fusion and action filtering modules, that is, without considering distance, no further filtering is performed on the global action space.

[0115] As shown in Table 1, overall, on both the validation set (known environment) and the validation set (unknown environment), the Transformer model initialized with a pre-trained model and a global action space and progress monitor (i.e., without distance fusion and action filtering modules) outperforms previous state-of-the-art models in both SR and OR. However, it requires extensive exploration in unfamiliar environments. The global action filter improves the model's performance in NE, TL, and SPL on the validation set (unknown environment), indicating that controlling the size of the global action space is useful for the agent to effectively explore unseen environments. Furthermore, the proposed dynamic fusion module significantly reduces the length of the navigation path, further improving exploration efficiency. Although these two modules incur some performance loss, the improved SPL measurement performance means they still contribute to efficient exploration.

[0116] This application also compares the impact of using visual language pre-trained initialization models on navigation performance; the experimental results are shown in Table 2.

[0117]

[0118] Table 2 shows a performance comparison of the model with random initialization and with PREVALENT initialization for 50,000 iterations. It can be seen that the model initialized using the pre-trained model has a significant advantage in key metrics, demonstrating the effectiveness of introducing the pre-trained model.

[0119] It should be stated that the above-described specific embodiments are merely preferred embodiments of the present invention and the technical principles employed. Those skilled in the art should understand that various modifications, equivalent substitutions, and variations can be made to the present invention. However, such variations, as long as they do not depart from the spirit of the present invention, should be within the scope of protection of the present invention. Furthermore, some terminology used in this specification and claims is not limiting, but merely for ease of description.

Claims

1. A distance-aware Transformer visual language navigation method, characterized by the following steps: include: S1 initializes navigation based on the navigation task and visual information from the perceived environment, primarily initializing the scene memory structure. Navigation module, navigation status Navigation command information ; S2, based on the scene memory structure at the current navigation progress t. Extract visual information of the current location Distance information Based on the visual and spatial information of the candidate positions, a global action space corresponding to the current position is constructed. ; S3, using a progress monitor global action space The process involves screening and selecting the top n candidate positions with the highest evaluation progress as the action space for the action decision-making process. , where n is a positive integer greater than 0; S4, this visual information Command information Distance information The input is a distance-aware Transformer navigation module with a dynamic distance fusion module, which updates the agent's navigation state. Output the distance-weighted action probability distribution corresponding to the current position. And select the candidate node with the highest probability as the next action; S5, based on the visual features and action space observed after the action is performed. Update scene memory structure arrive ; S6, repeat S2-S5 until the agent algorithm determines that the navigation task is completed, or the maximum number of steps K is reached; Step S3 includes evaluating the results using the following formula: in, This represents the navigation state of the agent in the previous step; Visual information representing the k-th candidate position; The final action space obtained after filtering is ; The distance-aware Transformer navigation module based on the dynamic distance fusion module described in step [the step] receives visual information. Command information Distance information Then, based on the following steps, an action decision is made at progress position t and the current navigation state is updated, including: B1 calculates the overall attention score for language, vision, distance information and navigation state vector, and the attention score and attention weight for language vector and all language features, respectively. B2 uses the attention weights corresponding to language and vision to obtain language and vision weighted information respectively, and performs cross-pattern matching by dot product and concatenates it with the navigation state vector output by the last layer of Transformer to map to a new state representation; B3, concatenates spatial information onto the newly generated navigation state vector and maps it to the final navigation state representation; B4 uses visual and distance-related attention scores as the action probability distribution and distance weight distribution, respectively. B5 combines the action probability distribution with the distance weight distribution to obtain a distance-weighted action probability distribution.

2. The visual language navigation method based on distance-aware Transformer according to claim 1, characterized in that, Step S1 also includes: A1, Construct a directed graph based on the visual features of the observable range at the starting point. , where nodes Corresponding to the revelation position, edge The spatial relative positional relationship between positions u and s, in this case, mainly refers to the spatial positional relationship between the starting position and the candidate position; A2, replaces the parameters of the randomly initialized Transformer model in the navigation module with the parameters of the pre-trained language model; A3, based on natural language instruction text The sequence will be segmented according to the sentence structure of the original instruction using the [SEP] segmentation marker, and a [CLS] marker will be added at the beginning position to handle loop states during navigation; the preprocessed sequence will then pass through a multi-layer Transformer network to obtain the initial instruction information as follows. With navigation status : 。 3. The visual language navigation method based on distance-aware Transformer according to claim 1, characterized in that, Step S2 also includes constructing the global action space corresponding to the current position using the following formula. : in, This represents the number of locations the agent has visited at the current progress point. This represents the candidate positions that are directly connected to the visited position u, i.e., a subset of the action space at the current progress. Representative scene memory structure Stored visual and spatial relative location information of candidate locations.

4. The visual language navigation method based on distance-aware Transformer according to claim 1, characterized in that, Also includes: The progress monitor Integrating the navigation state of the agent from the previous step With the currently observed visual information The current navigation progress is estimated as follows: in, Represents the Sigmoid activation function; , This is the parameter matrix of a learnable fully connected neural network.

5. The visual language navigation method based on distance-aware Transformer according to claim 1, characterized in that, n = 20。 6. The visual language navigation method based on distance-aware Transformer according to claim 1, characterized in that, Distance information One-hot encoding is used to encode the distance between the candidate position and the current position as a 20-dimensional 0-1 vector.

7. The visual language navigation method based on distance-aware Transformer according to claim 1, characterized in that, The agent algorithm is trained using a hybrid learning model that combines reinforcement learning and imitation learning, and optimized using the following formula: in, The action obtained by the algorithm based on probability sampling; The reward corresponding to the action; This represents the correct action; The coefficients represent the imitation learning loss function. The coefficients represent the distance loss function; The coefficients corresponding to the progress monitor loss function; This represents the probability of the action output by the intelligent agent; This represents the estimated task progress obtained from the progress monitor. This represents the actual progress of the navigation task; Shortest path distance vector; The distance weight vector represents the output of the model, T represents the total number of steps in the navigation task, and i represents the label of each element in the vector when the two vectors are used for the inner product operation.

8. The visual language navigation method based on distance-aware Transformer according to claim 1, characterized in that, Step S5 includes navigation step t, from After selecting a candidate position u, a new node is added to the scene memory structure to represent u; when the agent reaches position u... The navigable node corresponding to u will be deleted and added to the navigable node after reaching position u; at the same time, the visual information and spatial relative information in the graph will be updated according to the navigation state vector.

Images