A visual language navigation method based on reinforcement end point alignment improved VLN-BERT

Abstract

This invention belongs to the fields of deep learning and robot control technology, and provides a visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT for robot visual language navigation tasks. First, path-instruction pairs are extracted from the VLN dataset, and panoramic image sequences and natural language instructions in the path are embedded to obtain a preprocessed dataset. Based on the three-stage pre-training of VLN-BERT, the model is pre-trained through an enhanced endpoint alignment task to enhance the model's visual language alignment of path endpoints. The pre-training sequence is: general language foundation, visual foundation, action foundation, and enhanced endpoint alignment. The model, after four stages of pre-training, is fine-tuned through a path selection task to enable path selection capabilities. This invention enhances the model's visual language alignment of path endpoints by incorporating the enhanced endpoint alignment task into the three-stage pre-training process of VLN-BERT, thereby improving the navigation success rate of the agent in real-world environments.

Description

Technical Field

[0001] This invention belongs to the fields of deep learning and robot control technology. Specifically, it is a visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT, which is used for the visual language navigation task of robots. Background Technology

[0002] A long-term goal of artificial intelligence is to create an intelligent agent that can perceive its environment through visual information, communicate with humans using natural language, move freely in real environments, and perform complex tasks. With the significant progress made by deep learning in fields such as computer vision, natural language processing, and robot control and decision-making, visual language navigation has been proposed as a fundamental task for achieving this goal.

[0003] Based on understanding human natural language, intelligent agents act in real-world environments according to visual information and natural language instructions provided by humans, ultimately completing navigation tasks. For example, given the natural language instruction "Leave the bedroom, walk through the kitchen in the opposite direction of the photo on the wall, turn right at the long white counter, and stop when you pass the two chairs," the agent first needs to know it is in the bedroom and leave; then it needs to identify the photo on the wall and walk through the kitchen in the opposite direction; next, it needs to identify the long white counter and turn right; finally, it needs to identify the two chairs and stop at the appropriate location. Throughout this process, the agent needs to constantly compare surrounding visual information with the natural language instructions, distinguish between completed and incomplete instructions, and make decisions based on the current visual information and incomplete instructions. However, aligning and fusing information from two different modalities—visual and linguistic—and making decisions and navigating in complex environments presents a significant challenge for intelligent agents. The VLN-BERT model proposes using rich image-text pairs obtained from the internet to learn the visual foundation, thereby improving model performance by enhancing the model's multimodal information processing capabilities.

[0004] In recent years, significant progress has been made in vision-and-language pretraining (VLP) models based on the Transformer architecture and self-supervised learning objectives similar to BERT. As a result, many VLN models use pre-trained vision-and-language models as the base model and then fine-tune them to extract vision-and-language features.

[0005] The existing background technology still has the following shortcomings:

[0006] 1. Visual Language Navigation (VLN) requires massive amounts of data to fine-tune pre-trained models, thereby improving their generalization ability in complex and ever-changing real-world environments. However, due to the high costs associated with rendering 3D environments and labeling data, datasets for visual language navigation are relatively scarce. Therefore, efficiently utilizing existing data is crucial for improving the performance of visual language navigation models. Several pre-training tasks have been proposed to help models adapt to visual language navigation tasks; instruction trajectory matching is used to improve the model's ability to align panoramic image sequences with natural language instructions; action prediction helps the model predict actions based on current visual language information; shuffling loss is used to improve the model's temporal reasoning ability; and other methods have been proposed to enhance the model's spatial awareness.

[0007] 2. The pre-training task mentioned above did not pay additional attention to the alignment between the path endpoint in VLN and its corresponding text in the natural language instructions. In actual navigation tasks, humans place greater emphasis on the endpoint. In most cases, humans can accept the outcome that the agent takes some incorrect intermediate paths but arrives at the correct endpoint, but they generally find it difficult to accept the outcome that the agent fails to arrive at the correct endpoint. The natural language instructions in VLN describe in detail how the agent reaches the endpoint step by step from the starting point. If the agent cannot accurately reach the endpoint, then the navigation will definitely fail, and all the previous precise descriptions will be invalid. Therefore, for VLN tasks, the model should place greater weight on the alignment between the path endpoint and its corresponding text than on the alignment between other viewpoints and their corresponding text, and the penalty for the agent failing to arrive at the correct endpoint should be greater than the penalty for the agent taking incorrect intermediate paths but arriving at the correct endpoint. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention provides a visual language navigation method based on enhanced endpoint alignment-improved VLN-BERT, thereby resolving the issues in the prior art.

[0009] The technical solution of this invention is as follows:

[0010] A visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT, implemented using the VLN-BERT model, includes the following steps:

[0011] Step 1: Perform data preprocessing; First, extract path-instruction pairs from the VLN dataset, and then embed the panoramic image sequence and natural language instructions in the path to obtain the preprocessed dataset;

[0012] Step 2: Data interaction; The preprocessed data is interacted with through Co-TRM. Co-TRM uses a two-Transformer encoder structure to realize data interaction;

[0013] Step 3: Perform four-stage pre-training; In addition to the original three-stage pre-training of VLN-BERT based on general language foundation, vision foundation and action foundation, a fourth stage is added. The fourth stage is to pre-train the model for the task of strengthening endpoint alignment, which is used to enhance the model's visual and language alignment of the path endpoint; The pre-training order is general language foundation, vision foundation, action foundation, and strengthening endpoint alignment.

[0014] Step 4: Fine-tune the model using a path selection task; Fine-tune the model that has been pre-trained through four stages using a path selection task to enable the model to select paths.

[0015] Step 5: Perform route navigation and destination confirmation; use the fine-tuned model to perform actual route navigation, and continuously interact with and verify the actual environment during the navigation process.

[0016] Furthermore, in step 1, before pre-training and fine-tuning the model, it is necessary to preprocess the information in the VLN dataset. The VLN dataset usually consists of many path-instruction pairs, where the path is represented by a sequence of panoramic images. The information preprocessing methods include language information processing and visual information processing.

[0017] Furthermore, the language information processing method in step 1 is as follows:

[0018] Given a natural language instruction X = [x1, x2, ..., x L It is first transformed into an embedding vector Ω = [[CLS],ω1,ω2,...,ω] through word embedding and position embedding. L ,[SEP]];

[0019] [CLS] and [SEP] are two special markers used to represent global information and separate different sentences, respectively. ω is the word corresponding to each natural language instruction x in the embedding vector. Then, the embedding vector is input into the Transformer encoder for attention calculation.

[0020] Furthermore, the visual information processing method in step 1 is as follows:

[0021] Each path T = [P1, P2, ..., P M It contains M panoramic images; VLN-BERT first uses a pre-trained Faster R-CNN to extract the region features R from each panoramic image. i =[r1,r2,...,r k Then, for each panoramic image, VLN-BERT embeds the panoramic image index and the spatial information of each region; finally, these embeddings are merged to obtain the visual embedding for each region:

[0022]

[0023] in, It is a panoramic image index embedding. For the spatial information of the region, W s The spatial information of the region is mapped to a high-dimensional space; [IMG] is added as a special label before the feature of each panoramic image to extract global visual information; the final visual information is:

[0024]

[0025] Furthermore, the detailed process in step 2 is as follows:

[0026] The model structure of this method is the same as that of VLN-BERT. The preprocessed visual and linguistic information interacts through Co-TRM. Co-TRM uses two Transformer encoder structures to process visual and linguistic information respectively.

[0027] Unlike the Transformer encoder, Co-TRM uses the query vector computed by the visual encoder and the key and value vectors computed by the language encoder when calculating multi-head attention for the visual stream; and it uses the query vector computed by the language encoder and the key and value vectors computed by the visual encoder when calculating multi-head attention for the language stream. By exchanging the key and value vectors of the visual encoder and the language encoder, Co-TRM achieves the interaction between visual and language information. The calculation process is as follows:

[0028]

[0029] Where Q, K, and V represent the query vector, key vector, and value vector, respectively, d K This represents the dimension of the key vector; this method takes [IMG] as input to Co-TRM and [CLS] as the corresponding output. As global information, it is processed by matrix dot product and then input into the linear mapping layer, finally outputting the similarity score:

[0030]

[0031] Furthermore, the pre-training of the general language foundation in step 3 is as follows:

[0032] The language flow model is pre-trained using text information extracted from Wikipedia and BookCorpus, giving the model a general language foundation. The pre-training method is the same as BERT, which is a masked language model MLM and a next sentence prediction NSP task. MLM first randomly masks some words in the text, then predicts the masked words based on the context information, and finally pre-trains the model using the cross-entropy loss between the true value and the predicted value.

[0033] In each sequence, 15% of the words are randomly masked, with 80% of the words replaced by the [MASK] tag, 10% of the words replaced by random words, and 10% of the words left unchanged. NSP selects two sentences to allow the model to determine whether they are related. The two sentences have a 50% probability of being related and a 50% probability of being randomly selected.

[0034] Furthermore, the visual foundation pre-training in step 3 is as follows:

[0035] The model is pre-trained using the Conceptual Captions dataset, which contains approximately 3.3 million image-text pairs and is automatically obtained from the Internet. This allows the model to learn visual fundamentals from diverse Internet data, enabling multimodal masking (MMM) and multimodal alignment prediction (MAP). MMM first randomly masks some words and image regions, then simultaneously predicts the masked words and image regions based on both visual and linguistic context. Finally, the model is pre-trained using their respective loss functions.

[0036] When predicting masked image regions, MMM does not directly predict the feature values ​​of the region, but instead predicts the semantic distribution of the region. The model is pre-trained by minimizing the KL divergence between the true distribution and the predicted distribution. When predicting masked words, MMM uses the same loss function as MLM. In MAP, positive samples are correct image-text pairs, and negative samples are formed by randomly replacing the images or text of positive samples. This method first performs a matrix dot product between the output of the visual stream and the output of the language stream, and then outputs a binary variable as the final result through linear mapping, i.e., whether the image and text are aligned.

[0037] Furthermore, the pre-training for the basic movements in step 3 is as follows:

[0038] The path-instruction pairs from the VLN dataset are used to learn the action basis, allowing the model to learn commonly used actions in visual language navigation, including but not limited to "go forward", "turn right" and "stop", with the pre-training method being MMM.

[0039] Furthermore, the enhanced endpoint alignment pre-trained model in step 3 is as follows:

[0040] The Enhanced Endpoint Alignment task improves the model's focus on the endpoint by strengthening the alignment between the endpoint panorama and the end-of-instruction text. For each path-instruction pair, the Enhanced Endpoint Alignment task creates three incorrect paths: the first incorrect path removes the endpoint from the correct path; the second incorrect path randomly replaces the endpoint with another viewpoint from the correct path; and the third incorrect path randomly adds a viewpoint after the endpoint of the correct path. Based on one correct path and three incorrect paths, the Enhanced Endpoint Alignment task uses a three-stage pre-trained model to score the natural language instruction, and finally uses cross-entropy loss to enhance the model's visual-linguistic alignment of the path endpoint. The implementation of the Enhanced Endpoint Alignment task is as follows:

[0041]

[0042]

[0043] Loss = -logp[0]

[0044] Where T is the navigation path and P is the panoramic view of the viewpoint;

[0045] Score by formula The calculation is performed, where p is a vector and Loss is the loss function for the enhanced endpoint alignment task.

[0046] Furthermore, the path selection fine-tuning method in step 4 is as follows:

[0047] After the four-stage pre-training process, the model has a general language foundation and action foundation, and the alignment of the path endpoint is enhanced; therefore, similar to VLN-BERT, this method samples one correct path and three incorrect paths from the paths generated by beam search, and supervises the model to select the correct path through cross-entropy loss.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] 1. This invention obtains a dataset on visual and language navigation through a pre-trained dataset, and improves the visual and language navigation method of VLN-BERT based on enhanced endpoint alignment, thereby improving the navigation success rate of the agent in the real environment; using VLN-BERT as the base model, it also enables the model to have a general language foundation and action foundation.

[0050] 2. This invention incorporates the enhanced endpoint alignment task into the three-stage pre-training process of VLN-BERT, forming a four-stage pre-training process. This enhances the model's visual-linguistic alignment of path endpoints and improves the agent's navigation success rate in real-world environments. Therefore, the new model can better complete navigation tasks in real-world environments. Attached Figure Description

[0051] Figure 1 This is a flowchart of the steps of the present invention.

[0052] Figure 2 This is a flowchart of the visual language navigation process based on VLN-BERT of this invention.

[0053] Figure 3 This is a structural diagram of the VLN-BERT model of this invention.

[0054] Figure 4 This is a diagram of the improved four-stage pre-training process of VLN-BERT according to the present invention.

[0055] Figure 5 This invention enhances the endpoint alignment task graph.

[0056] Figure 6 This is a schematic diagram of the pre-training sequence in step 2 of the present invention.

[0057] Figure 7 This is a schematic diagram of the visual language navigation principle of the present invention. Detailed Implementation

[0058] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples; the following examples are used to illustrate the present invention, but should not be used to limit the scope of the present invention.

[0059] Example: Figure 1-7 As shown, this invention provides a visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT. This method can enhance VLN-BERT's alignment of the path endpoint in navigation with its corresponding text in natural language instructions by strengthening the endpoint alignment pre-training task, thereby improving the model's navigation success rate.

[0060] VLN-BERT is implemented based on the Vi LBERT model architecture and uses rich image-text pairs obtained from the internet to learn the visual foundation, thereby improving the model's performance. VLN-BERT assumes that the agent has already explored the environment before performing the VLN task and stores navigation and panoramic information in memory for easy access. The visual language navigation process based on VLN-BERT is as follows: Figure 2As shown, for a given natural language instruction, the agent first generates N paths using a beam search based on a greedy policy-based instruction following model, according to environmental information. Then, it uses a pre-trained VLN-BERT to align and score the natural language instruction with all generated paths. Finally, it selects the path with the highest score as the final path and navigates along that path. VLN-BERT proposes a three-stage pre-training process, pre-training the model in the order of general language foundation, visual foundation, and action foundation.

[0061] This solution proposes an enhanced endpoint alignment pre-training task for VLN-BERT, which improves the model's focus on the endpoint by strengthening the alignment between the endpoint panorama and the end-of-instruction text. Building upon the three-stage pre-training process of VLN-BERT, this enhanced endpoint alignment pre-training task strengthens the model's visual-linguistic alignment of path endpoints.

[0062] The specific steps of this method are as follows:

[0063] Step 1: Perform data preprocessing; First, extract path-instruction pairs from the VLN dataset, and then embed the panoramic image sequence and natural language instructions in the path to obtain the preprocessed dataset;

[0064] Before pre-training and fine-tuning the model, the VLN dataset needs to be preprocessed. The VLN dataset usually consists of many path-instruction pairs, where the path is represented by a sequence of panoramic images. The information processing methods include language information processing and visual information processing.

[0065] The language information processing method is as follows:

[0066] Given a natural language instruction X = [x1, x2, ..., x L It is first transformed into an embedding vector Ω = [[CLS],ω1,ω2,...,ω] through word embedding and position embedding. L ,[SEP]];

[0067] [CLS] and [SEP] are two special markers used to represent global information and separate different sentences, respectively; then, the embedding vector is input into the Transformer encoder for attention calculation.

[0068] Visual information processing methods are as follows:

[0069] Each path T = [P1, P2, ..., P M This method contains M panoramic images; similar to VLN-BERT, it first uses a pre-trained Faster R-CNN to extract the region features R from each panoramic image. i ;

[0070] R i =[r1,r2,...,r k ];

[0071] Then, for each panoramic image, this method embeds the panoramic image index and the spatial information of each region; finally, these embeddings are merged to obtain the visual embedding of each region:

[0072]

[0073] in, It is a panoramic image index embedding. For the spatial information of the region, W s The spatial information of the region is mapped to a high-dimensional space; [IMG] is added as a special label before the feature of each panoramic image to extract global visual information; the final visual information is:

[0074]

[0075] By using the pre-trained dataset, we obtained datasets on vision and language navigation, which played a very important role in efficiently utilizing existing data to improve the performance of vision and language navigation models. Using VLN-BERT as the base model also enabled the model to have a general language foundation and action foundation, enhancing the model's generalization ability and reliability.

[0076] Step 2: Data interaction; The preprocessed data is interacted with through Co-TRM. Co-TRM uses a two-Transformer encoder structure to realize data interaction;

[0077] like Figure 3 As shown, the preprocessed visual and linguistic information interacts through Co-TRM; Co-TRM uses two Transformer encoder structures to process visual and linguistic information respectively;

[0078] Unlike the Transformer encoder, Co-TRM uses the query vector computed by the visual encoder and the key and value vectors computed by the language encoder when calculating multi-head attention for the visual stream; and it uses the query vector computed by the language encoder and the key and value vectors computed by the visual encoder when calculating multi-head attention for the language stream. By exchanging the key and value vectors of the visual encoder and the language encoder, Co-TRM achieves the interaction between visual and language information. The calculation process is as follows:

[0079]

[0080] Where Q, K, and V represent the query vector, key vector, and value vector, respectively, d K This represents the dimension of the key vector; this method takes [IMG] as input to Co-TRM and [CLS] as the corresponding output. As global information, it is processed by matrix dot product and then input into the linear mapping layer, finally outputting the similarity score:

[0081]

[0082] Step 3: Perform four-stage pre-training; Based on the three-stage pre-training of VLN-BERT, the model is pre-trained by strengthening the endpoint alignment task, which enhances the model's visual and linguistic alignment of the path endpoint. The pre-training order is general language foundation, visual foundation, action foundation, and enhanced endpoint alignment.

[0083] Among them, the common language foundation:

[0084] The language flow model is pre-trained using text information extracted from Wikipedia and BookCorpus, giving the model a general language foundation. The pre-training method is the same as BERT, which is a masked language model MLM and a next sentence prediction NSP task. MLM first randomly masks some words in the text, then predicts the masked words based on the context information, and finally pre-trains the model using the cross-entropy loss between the true value and the predicted value.

[0085] In each sequence, 15% of the words are randomly masked, with 80% of the words replaced by the [MASK] tag, 10% of the words replaced by random words, and 10% of the words left unchanged. NSP selects two sentences to allow the model to determine whether they are related. The two sentences have a 50% probability of being related and a 50% probability of being randomly selected.

[0086] Visual basics:

[0087] The model is pre-trained using the Conceptual Captions dataset, which contains approximately 3.3 million image-text pairs and is automatically obtained from the Internet. This allows the model to learn visual fundamentals from a wide variety of Internet data. The pre-training method is the same as ViLBERT, which includes Mask Multimodal Modeling (MMM) and Multimodal Alignment Prediction (MAP). MMM first randomly masks some words and image regions, and then predicts the masked words and image regions based on both visual and linguistic context. Finally, the model is pre-trained using their respective loss functions.

[0088] When predicting masked image regions, MMM does not directly predict the feature values ​​of the region, but predicts the semantic distribution of the region. It pre-trains the model by minimizing the KL divergence between the true distribution and the predicted distribution. When predicting masked words, MMM uses the same loss function as MLM. In MAP, positive samples are correct image-text pairs, and negative samples are formed by randomly replacing the images or text of positive samples. Vi LBERT first performs a matrix dot product of the output of the visual stream and the output of the language stream, and then outputs a binary variable as the final result through a linear mapping, i.e., whether the image and text are aligned.

[0089] Basic movements:

[0090] The path-instruction pairs from the VLN dataset are used to learn the action basis, allowing the model to learn commonly used actions in visual language navigation, including but not limited to "go forward", "turn right" and "stop", with the pre-training method being MMM.

[0091] Strengthen endpoint alignment:

[0092] The enhanced endpoint alignment task improves the model's focus on the endpoint by strengthening the alignment between the endpoint panorama and the end-of-instruction text. The implementation of the enhanced endpoint alignment task is as follows: Figure 5 As shown; for each path-instruction pair, the reinforcement endpoint alignment task creates three incorrect paths. The first incorrect path removes the endpoint from the correct path; the second incorrect path randomly replaces the endpoint with another viewpoint from the correct path; and the third incorrect path randomly adds a viewpoint after the endpoint of the correct path. Based on one correct path and three incorrect paths, the reinforcement endpoint alignment task uses a model that has been pre-trained in three stages to score the natural language instructions. Finally, cross-entropy loss is used to enhance the model's visual-linguistic alignment of the path endpoints. The implementation of the reinforcement endpoint alignment task is as follows:

[0093]

[0094]

[0095] Loss = -logp[0]

[0096] Where T is the navigation path and P is the panoramic view of the viewpoint;

[0097] Score by formula The calculation is performed, where p is a vector and Loss is the loss function for the enhanced endpoint alignment task.

[0098] By incorporating the enhanced endpoint alignment task into the three-stage pre-training process of VLN-BERT, a four-stage pre-training process is formed, which enhances the model's visual-linguistic alignment of path endpoints and improves the agent's navigation success rate in real-world environments; enabling the VLN-BERT model to better complete navigation tasks in real-world environments.

[0099] Step 4: Fine-tune using a path selection task; fine-tune the model that has been pre-trained through four stages using a path selection task to enable the model to select paths.

[0100] After the four-stage pre-training process, the model has a general language foundation and action foundation, and the alignment of the path endpoint is enhanced. Similar to VLN-BERT, this method samples one correct path and three incorrect paths from the paths generated by beam search, and supervises the model to select the correct path through cross-entropy loss.

[0101] Step 5: Perform route navigation and destination confirmation; use the fine-tuned model to perform actual route navigation, and continuously interact with and verify the actual environment during the navigation process.

[0102] The embodiments of the present invention are given for the purposes of illustration and description. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT, implemented using the VLN-BERT model, characterized in that, It includes the following steps; Step 1: Perform data preprocessing; First, extract path-instruction pairs from the VLN dataset, and then embed the panoramic image sequence and natural language instructions in the path to obtain the preprocessed dataset; Step 2: Data interaction; The preprocessed data is interacted with through Co-TRM, which uses a two-Transformer encoder structure to achieve data interaction; Step 3: Perform four-stage pre-training; In addition to the original three-stage pre-training of VLN-BERT, namely general language foundation, visual foundation and action foundation, a fourth stage is added. The fourth stage is to train the model using the enhanced endpoint alignment task to enhance the model's visual language alignment of path endpoints; The pre-training order is general language foundation, visual foundation, action foundation and enhanced endpoint alignment. The enhanced endpoint alignment pre-trained model in step 3 is as follows: The Enhanced Endpoint Alignment task improves the model's focus on the endpoint by strengthening the alignment between the endpoint panorama and the end-of-instruction text. For each path-instruction pair, the Enhanced Endpoint Alignment task creates three incorrect paths: the first incorrect path removes the endpoint from the correct path; the second incorrect path randomly replaces the endpoint with another viewpoint from the correct path; and the third incorrect path randomly adds a viewpoint after the endpoint of the correct path. Based on one correct path and three incorrect paths, the Enhanced Endpoint Alignment task uses a three-stage pre-trained model to score the natural language instruction, and finally uses cross-entropy loss to enhance the model's visual-linguistic alignment of the path endpoint. The implementation of the Enhanced Endpoint Alignment task is as follows: ； in, For navigation path, A panoramic view of the given viewpoint; From the formula Perform calculations. For a vector, To enhance the loss function for the endpoint alignment task; Step 4: Fine-tune the model using a path selection task; Fine-tune the model that has undergone four stages of pre-training using a path selection task to enable the model to select paths. Step 5: Perform route navigation and destination confirmation; use the fine-tuned model to perform actual route navigation, and continuously interact with and verify the actual environment during the navigation process.

2. The visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT as described in claim 1, characterized in that: In step 1, before pre-training and fine-tuning the model, it is necessary to preprocess the information in the VLN dataset. The VLN dataset consists of many path-instruction pairs, where the path is represented by a sequence of panoramic images. The information preprocessing methods include language information processing and visual information processing.

3. The visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT as described in claim 2, characterized in that, The language information processing method in step 1 is as follows: Given a natural language instruction It is first converted into an embedding vector through word embedding and position embedding. ; in and These are two special markers, used to represent global information and to separate different sentences, respectively. It is the word corresponding to each natural language instruction x in the embedded vector; Then, the embedding vector is fed into the Transformer encoder for attention computation.

4. The visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT as described in claim 2, characterized in that, The visual information processing method in step 1 is as follows: Each path Contains M panoramic images Similar to VLN-BERT, a pre-trained Faster R-CNN is first used to extract region features from each panoramic image. Then, for each panoramic image, embed the panoramic image index and the spatial information of each region; Finally, these embeddings are merged to obtain the visual embedding for each region: ； in, It is a panoramic image index embedding. For the spatial information of the region, Mapping the spatial information of a region to a high-dimensional space; using a special label to extract global visual information. Added before the features of each panoramic image; the final visual information is: 。 5. The visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT as described in claim 1, characterized in that, The detailed process in step 2 is as follows: The preprocessed visual and linguistic information interacts through Co-TRM; Co-TRM uses two Transformer encoder structures to implement the interaction of visual and linguistic information respectively. Unlike the Transformer encoder, Co-TRM uses the query vector computed by the visual encoder and the key and value vectors computed by the language encoder when calculating multi-head attention for the visual stream; and it uses the query vector computed by the language encoder and the key and value vectors computed by the visual encoder when calculating multi-head attention for the language stream. By exchanging the key and value vectors of the visual encoder and the language encoder, Co-TRM achieves the interaction between visual and language information. The calculation process is as follows: ； in, These represent the query vector, key vector, and value vector, respectively. This represents the dimension of the key vector; it will be input into Co-TRM. and Corresponding output and This information is used as global information, and after performing a matrix dot product, it is input into a linear mapping layer, ultimately outputting a similarity score: 。 6. The visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT as described in claim 1, characterized in that, The pre-training of the general language foundation in step 3 is as follows: The language flow model is pre-trained using text information extracted from Wikipedia and BookCorpus, giving the model a general language foundation. The pre-training method is the same as that of the VLN-BERT model, which is a masked language model (MLM) and a next-sentence NSP prediction task. MLM first randomly masks some words in the text, then predicts the masked words based on the context information, and finally pre-trains the model using the cross-entropy loss between the true and predicted values. In each sequence, 15% of the words are randomly masked. Of the masked words, 80% are replaced with the [MASK] tag, 10% are replaced with random words, and 10% remain unchanged. NSP selects two sentences to allow the model to determine whether they are related. The two sentences have a 50% probability of being related and a 50% probability of being randomly selected.

7. The visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT as described in claim 1, characterized in that, The visual pre-training in step 3 is as follows: The model is pre-trained by automatically acquiring the ConceptualCaptions dataset containing approximately 3.3 million image-text pairs from the Internet, allowing the model to learn visual fundamentals from diverse Internet data, enabling multimodal mask modeling (MMM) and multimodal alignment prediction (MAP). MMM first randomly covers some words and image regions, then predicts the covered words and image regions based on both visual and linguistic context, and finally pre-trains the model using their respective loss functions. When predicting masked image regions, MMM does not directly predict the feature values ​​of the region, but instead predicts the semantic distribution of the region. The model is pre-trained by minimizing the KL divergence between the true distribution and the predicted distribution. When predicting masked words, MMM uses the same loss function as MLM. In MAP, positive samples are correct image-text pairs, and negative samples are formed by randomly replacing the images or text of positive samples. The outputs of the visual stream and the language stream are then matrix-dot productd, and a binary variable is output as the final result through linear mapping, indicating whether the image and text are aligned.

8. The visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT as described in claim 1, characterized in that, The basic pre-training for the movements in step 3 is as follows: The path-instruction pairs from the VLN dataset are used to learn the action basis, allowing the model to learn commonly used actions in visual language navigation, including but not limited to "go forward", "turn right" and "stop", with the pre-training method being MMM.

9. The visual language navigation method based on enhanced endpoint alignment to improve VLN-BERT as described in claim 1, characterized in that, The path selection fine-tuning method in step 4 is as follows: After the four-stage pre-training process, the model has a general language foundation and action foundation, and the alignment of the path endpoint is enhanced. Similar to VLN-BERT, a correct path and three incorrect paths are sampled from the paths generated by beam search, and the model is supervised to select the correct path through cross-entropy loss.

Images