A robot visual language navigation method based on direction perception learning

Abstract

The present application relates to the technical field of deep learning and robot control, and provides a robot visual language navigation method based on direction perception learning, which comprises image text embedding of equipment, embedding operation of images and corresponding instruction texts in an R2R data set, alignment and fusion of visual language information in the subsequent process, four-stage pre-training of VLN-BERT, so that the intelligent agent obtains the ability to perform basic navigation actions, and finally performs path selection fine-tuning, so that the intelligent agent with general visual language understanding ability can perform subsequent navigation tasks through path selection. The present application can improve the direction perception ability of the intelligent agent, thereby enhancing the alignment ability of the intelligent agent to natural language instructions and visual information, and finally improving the navigation performance. Therefore, the present model can better complete the navigation task in a real environment.

Description

Technical Field

[0001] This invention relates to the fields of deep learning and robot control technology, and more specifically, to a robot visual language navigation method based on direction-aware learning. By incorporating direction-aware learning into the model, the method improves the agent's direction perception ability, thereby enhancing the agent's ability to align natural language instructions and visual information, and ultimately improving navigation performance. Background Technology

[0002] One of the future goals of artificial intelligence is to enable intelligent agents to perceive and understand their surroundings through visual, auditory, and other information. Based on this understanding, they can communicate with humans using natural language and ultimately perform various complex tasks in real-world environments. For example, when we give a robot the instruction "Bake a cake for me," the robot can autonomously move to the corresponding area, including the kitchen, living room, and bedroom, and complete subsequent tasks such as detecting and picking up relevant items, processing ingredients, and operating the oven. With the significant progress made in deep learning in fields such as computer vision and natural language processing, developing robots capable of autonomous learning and performing complex tasks has gradually become one of the key research areas in artificial intelligence.

[0003] However, the agent's orientation perception problem has not been considered in the Vision-and-Language Navigation (VLN) task. General image-text learning methods cannot meet the agent's subsequent needs for performing navigation tasks. In navigation tasks, the agent has multiple action choices, including but not limited to "turn left, turn right, move forward, move backward, stop," etc. For certain instructions, with the starting and ending positions remaining unchanged, the agent has a chance to successfully reach the target destination even if it chooses an action different from the actual navigation process, or even an action in the opposite direction. However, during navigation, errors in orientation perception may lead to unreasonable action choices, causing the agent to make incorrect navigation actions, collect interfering environmental visual information, and thus cause problems such as the agent taking detours or deviating from the expected destination direction. This increases the navigation time and may even significantly lead to navigation task failure. Figure 7 As shown, various navigation problems are caused by errors in orientation perception:

[0004] In this diagram, the green path represents the correct path, while the blue and purple paths are the agent's actual navigation paths. Node 1 and Node 2 represent the locations where the agent makes action choices during navigation. On different navigation paths, the agent's perception of direction differs at a given action selection node, leading to different action choices. Therefore, when making navigation action decisions, the agent needs to select the most reasonable action from a vast action space. Choosing the correct direction greatly aids in making the correct action selection. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a robot visual language navigation method based on direction-aware learning. This method enhances the agent's direction perception ability by incorporating direction-aware learning into the VLN-BERT visual language multimodal alignment pre-training process, thereby strengthening the agent's alignment ability with natural language commands and visual information, ultimately improving navigation performance. This solves the problem of reduced navigation success rate of pre-trained models in invisible or unfamiliar environments in existing technologies.

[0006] The technical solution of this invention is as follows:

[0007] A robot visual language navigation method based on direction-aware learning is characterized by the following steps: image and text embedding, where images and corresponding instruction texts in an R2R dataset are embedded to facilitate subsequent alignment and fusion of visual language information; four-stage pre-training, incorporating direction-aware learning into the three-stage pre-training process of VLN-BERT to improve the agent's direction-awareness capability; and path selection fine-tuning, enabling the agent, which already possesses general visual language understanding capabilities, to perform subsequent navigation tasks through path selection.

[0008] Furthermore, by embedding images and text, the images and corresponding instruction texts in the R2R dataset are embedded separately, which facilitates the subsequent alignment and fusion of visual language information.

[0009] Furthermore, in the VLN task, the visual information that the agent needs to process is a series of image information that is continuous in time and space, and the linguistic information is a text description with navigation instructions. When embedding images, similar to VLN-BERT, this method combines the corner position information of the image segment, the area of ​​the image segment, the agent's orientation and the sine and cosine values ​​of the pitch angle, and the relevant information of the next position, transforming them into a 2048-dimensional input vector. The image segment index information is also transformed into another 2048-dimensional input vector. A pre-trained Faster R-CNN network is used to extract features from the image, and the resulting features are combined with the two vectors mentioned above to form the visual embedding result. When embedding linguistic information, this method uses the same embedding method as BERT, adding the word embedding vector and the position embedding vector to obtain the final linguistic embedding result.

[0010] Furthermore, in the R2R dataset, for each path, there are three natural language instruction descriptions. After analyzing the natural language instructions in the R2R dataset, this invention filters and organizes the key words in the instructions that determine the agent's action selection based on orientation into the following words: left, right, up, down, forward, and stop. These words can effectively help the agent perceive orientation during natural language instruction understanding and subsequent navigation, and also largely determine the agent's action selection.

[0011] Table 1. Partial Command Contents for R2R Datasets

[0012]

[0013] Furthermore, a four-stage pre-training process is implemented, incorporating orientation awareness learning into the three-stage pre-training process of VLN-BERT, thereby improving the agent's orientation awareness capability.

[0014] Furthermore, the four-stage pre-training process includes: Stage 1, where:

[0015] The ViLBERT model was trained using a text dataset extracted from the Internet, including Wikipedia and BooksCorpus. The model was trained using a masked language model (MLM) and next sentence prediction (NSP) to enable word prediction and contextual understanding, thus giving it general language understanding capabilities.

[0016] Furthermore, MLM first randomly masks the words in the text and then performs backpropagation based on contextual information. During pre-training, 15% of the words in the input text are masked. The masking operation includes: 80% probability of replacing the original word with a specific word representation [MASK], 10% probability of randomly replacing the original word with another word, and 10% probability of leaving the original word unchanged. Finally, the model parameters are updated using cross-entropy loss. The loss function of MLM is shown below:

[0017]

[0018] In the formula, N represents the total number of words to be masked, and w i C refers to the i-th masked word. i It is the context information of the i-th masked word, P(w i |C i ) represents the probability distribution predicted by the model for the masked words.

[0019] Furthermore, the NSP task allows the model to learn the relationships between different sentences in the input text, enhancing its performance on subsequent downstream tasks. In NSP, the model determines the relationship between adjacent sentences in the input, using a binary network to determine whether a contextual relationship exists between two input sentences. For each input sentence pair, the model outputs its corresponding binary probability distribution, representing the likelihood that the sentence pair is continuous or discontinuous. The loss function for the NSP task is shown below:

[0020]

[0021] In the formula, M is the total number of statement pairs, and y j The actual labels for the j-th statement pair include: y j =0 indicates that the input statement pairs are not consecutive, y j =1 indicates that the input statement pairs are consecutive, P(y j |S j ) is the probability output of the model for the j-th statement pair.

[0022] The loss function for the first stage can be expressed as a weighted sum of the MLM loss function and the NSP loss function, with the hyperparameter λ representing the relative influence of the two loss functions. The loss function for the first stage is shown in the following equation:

[0023] L1 = L MLM +λL NSP

[0024] Furthermore, the four-stage pre-training process includes, in the second stage:

[0025] Building upon the ViLBERT model's language understanding capabilities, the model is trained across visual language modes using the Conceptual Captions text-image dataset. By leveraging Masked Multi-modal Modeling (MMM) and Multi-modal Alignment Prediction (MAP), the model can infer local images of the mask based on panoramic image information and perform matching and alignment between image content understanding and natural language descriptions, thus enabling the model to possess general visual language understanding capabilities.

[0026] Furthermore, MMM mimics the MLM training process in BERT by randomly masking images and text. It then uses global image information and contextual text information to perform backward reasoning on the masked portion. During parameter updates, it trains using the loss functions corresponding to image and text predictions respectively. For the image mask prediction part, Vi LBERT does not directly output the image features of the masked portion, but rather its semantic distribution. It is trained by minimizing the KL divergence between the predicted and true semantic distributions. For the text information mask prediction part, the process is the same as BERT's prediction of masked text, using a cross-entropy loss function to train the predicted and true text.

[0027] Furthermore, in the MAP task, correct text-image pairs are used as positive samples, and new text-image pairs generated by randomly replacing text or images are used as negative samples. The output vectors obtained using the initial input [IMG] of the image input sequence and the initial feature vocabulary [CLS] of the text input sequence are used as the final image prediction output and text prediction output. The ViLBERT model finally performs a dot product operation between the text prediction output and the image prediction output, and uses a binary network to determine whether the information in the text output and the image output match.

[0028] Furthermore, the four-stage pre-training process includes the third stage, in which:

[0029] The Vi LBERT model, which already possesses general visual language understanding capabilities, is trained using the R2R dataset. The MMM and MAP methods are used to enable the model to learn basic actions in navigation, thus giving the model general visual language understanding and navigation capabilities.

[0030] Furthermore, the four-stage pre-training process includes: Stage 4, in which:

[0031] For the general VLN-BERT model, a direction-aware learning method is used to further enhance the model's ability to perceive direction in navigation commands, thereby improving its performance on navigation tasks. During the visual-language multimodal alignment pre-training process, by learning directional word mask prediction for the agent, the agent can more efficiently and accurately align the connection between directional words and visual information during actual navigation. Figure 6 A schematic diagram of the direction-aware learning principle, where the colored parts are the directional words in the mask.

[0032] Furthermore, in the direction-aware learning process for locative words, this invention employs a pre-training method for mask prediction learning of locative words. The following mask prediction method is used for locative words in natural language instructions: replacing the locative word with a special identifier [MASK] with a 50% probability; and keeping the locative word unchanged with a 50% probability. The masked locative words are predicted backwards using the model's learning of global visual image sequences and contextual instructions. Finally, the model parameters are updated using backpropagation using cross-entropy loss and the KL divergence between the predicted image output and the true output. The loss function for direction-aware learning is shown in the following equation:

[0033]

[0034] L Image =KL(Pre||Real)

[0035] L DPL =L Image +L Language

[0036] Furthermore, in L language In the formula, N represents the total number of locative words to be masked, and w i C refers to the i-th masked directional word. i It is the context information of the i-th masked word, P(w i |C i ) represents the probability distribution for the prediction of masked directional words in MLM; in L Image In the formula, KL represents the KL divergence, and Pre and Real represent the semantic distributions of the predicted and real image outputs, respectively; in L... DPL In the formula, L DPL This represents the loss function for direction-aware learning.

[0037] Furthermore, path selection fine-tuning is performed. After the four-stage pre-training process described above, path selection is used as a downstream task to fine-tune the model, enabling it to acquire path selection capabilities in addition to its general visual language navigation capabilities. For each path-instruction pair, the path selection task uses beam search to randomly generate three incorrect paths that do not match the actual data. "+" indicates the actual path, and "-" indicates the incorrect path. Cross-entropy loss is used for training, as shown in the following formula:

[0038]

[0039] in, and The word embedding vector of the special marker [CLS] used for classification task in the i-th path and the image feature vector obtained by the VLN-BERT model from the first image fragment [I MG] are represented by W, which is a learnable parameter matrix used to calculate the path score.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] The visual-language navigation method based on direction-aware learning proposed in this invention can improve the agent's direction-awareness ability, thereby enhancing the agent's ability to align natural language instructions and visual information, ultimately improving navigation performance. Therefore, this model can better complete navigation tasks in real-world environments. Attached Figure Description

[0042] Figure 1 This diagram illustrates the main steps of a robot visual language navigation method based on direction-aware learning, as described in this invention.

[0043] Figure 2 This is a flowchart illustrating the four-stage pre-training process of the present invention.

[0044] Figure 3 This is a logic diagram of the four-stage pre-training of the Vi LBERT model for cross-modal training of visual language according to the present invention.

[0045] Figure 4 This is a logic diagram of the four-stage pre-training process of this invention to add navigation capabilities to the Vi LBERT model.

[0046] Figure 5 This is a logic diagram of the four-stage pre-training of the Vi LBERT model for language training according to the present invention.

[0047] Figure 6 This is a schematic diagram illustrating the four-stage pre-training direction-aware learning principle of the present invention.

[0048] Figure 7 These are various navigation problems caused by errors in direction perception. Detailed Implementation

[0049] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0050] This invention provides a robot visual language navigation method based on direction-aware learning, comprising:

[0051] S1. Image and text embedding: Embedding operations are performed on the images and corresponding instruction texts in the R2R dataset to facilitate the subsequent alignment and fusion of visual language information.

[0052] S2. Perform four-stage pre-training, adding orientation awareness learning to the three-stage pre-training process of VLN-BERT, thereby improving the agent's orientation awareness ability;

[0053] S3. Fine-tune the execution path selection: Enable the intelligent agent, which already has general visual language understanding capabilities, to perform subsequent navigation tasks through path selection.

[0054] Example 1: In this embodiment of the invention, the direction perception ability of the agent is improved by adding direction perception learning during the visual language multimodal alignment pre-training process of VLN-BERT, thereby enhancing the agent's ability to align natural language instructions and visual information, and ultimately improving navigation performance.

[0055] The method involves the following steps: First, image and text embedding is performed. To facilitate the subsequent alignment and fusion of visual and linguistic information, the images and corresponding instruction texts in the R2R dataset are embedded separately. The visual information that the agent needs to process is a series of image information that is continuous in time and space, and the linguistic information is a text description with navigation instructions. During the embedding operation, similar to VLN-BERT, DPL combines the corner position information of the image segment, the area of ​​the image segment, the agent's orientation, the sine and cosine values ​​of the pitch angle, and the relevant information of the next position to convert it into a 2048-dimensional input vector. The image segment index information is converted into another 2048-dimensional input vector. Image features are extracted through a pre-trained Faster R-CNN network, and the result is combined with the above two vectors to form the visual embedding result. When embedding the language, DPL uses the same embedding method as BERT. The final language embedding result is equal to the sum of the word embedding vector and the position embedding vector.

[0056] After the steps are completed, this embodiment of the invention performs a four-stage pre-training of the model. The first stage involves training the ViLBERT model in a single language modality using a text information dataset extracted from the Internet, including Wikipedia and BooksCorpus. Furthermore, a Masked-language-Model (MLM) and Next-Sentence-Prediction (NSP) are used to enable the model to perform word prediction and contextual understanding, giving it general language understanding capabilities. MLM first randomly encodes the words in the text, and then, based on contextual information, reverse-infers the masked words. In this embodiment's pre-training process, 15% of the words in the input text are masked. This masking includes: 80% probability of replacing the original word with a feature word representation [MASK]; 10% probability of randomly replacing the original word with another word; and 10% probability of leaving the original word unchanged. Finally, the model parameters are updated via backpropagation using cross-entropy loss. The loss function formula for MLM is:

[0057]

[0058] Where N represents the total number of words masked, w i C refers to the i-th masked word. i It is the context information of the i-th masked word, P(w i |C i ) represents the probability distribution predicted by the model for the masked words.

[0059] The NSP task allows models to learn the relationships between different sentences in the input text, enhancing their performance on subsequent downstream tasks. In NSP, the model first determines the relationship between preceding and following sentences, then uses a binary network to determine whether there is contextual association between the input sentences. The model provides a corresponding binary probability distribution for each input sentence pair to describe whether the two sentences are continuous or discontinuous. The loss function for the NSP task is as follows:

[0060]

[0061] Where M is the total number of statement pairs, y j It is the real label of the j-th statement pair, which includes y j =0 indicates that the input statement pairs are not consecutive, y j =1 indicates that the input statement pairs are consecutive, P(y j |S j ) is the probability output of the model for the j-th statement pair.

[0062] The loss function for the first stage can be expressed as a weighted sum of the MLM loss function and the NSP loss function, with the hyperparameter λ representing the relative influence of the two loss functions. The loss function for the first stage is shown in the formula:

[0063] L1 = L MLM +λL NSP

[0064] This invention, based on the ViLBERT model's language understanding capabilities, employs a concept-captioned image-text hybrid dataset for cross-modal visual language training. The image-text hybrid dataset includes the ConceptualCaptions text-image dataset. Through Masked Multi-modal Modeling (MMM) and Multi-modal Alignment Prediction (MAP), the model identifies and aligns masked regions, thus enabling it to possess general visual language understanding capabilities.

[0065] The MMM model simulates the MLM training process in BERT, employing a random masking method to cover images and text. It comprehensively considers global graph and contextual information to achieve inverse inference of the mask. Furthermore, it combines the relevant loss functions for image and text prediction to achieve inverse inference of the mask. For image mask prediction, ViLBERT outputs not the image features in the mask, but rather their semantic distribution within the model. It then trains by minimizing the KL divergence between the predicted and true semantic distributions. For text information masking prediction, this embodiment uses the BERT model's method for predicting masked text, specifically training the model with the prediction results and actual data using the cross-entropy loss function.

[0066] In the MAP task, appropriate text-image pairs are used as positive samples, and new text-image pairs are formed as negative samples by randomly replacing text or images. This method uses the output vector obtained by combining the start input (IMG) of the image input sequence with the start feature word (CLS) of the text input sequence as the image prediction result and the text prediction result, respectively. Finally, the ViLBERT model performs a dot product operation on the text prediction result and the image prediction result, and a binary neural network determines the degree of matching between the text output and the image output.

[0067] This invention trains the ViLBERT model, which already possesses general visual language understanding capabilities, using the R2R dataset and employing the MMM and MAP methods to enable the model to learn basic actions in navigation. This model possesses both general visual language understanding and navigation capabilities.

[0068] This invention employs a direction-aware learning method on the general VLN-BERT model, further enhancing the model's ability to perceive direction in navigation commands and improving its performance in navigation tasks. During the visual-language multimodal alignment pre-training process, by learning directional word mask prediction for the agent, the agent can more efficiently and accurately align the connection between directional words and visual information in actual navigation tasks.

[0069] Table 1 shows a portion of the R2R dataset mentioned in this embodiment of the invention. The R2R dataset serves as a benchmark for researching visual navigation tasks, primarily focusing on how machine learning models can understand natural language instructions and navigate in complex indoor environments. This dataset consists of images of multiple rooms and human-designed navigation paths, with the goal of training models to find routes from one room to another based on given language descriptions. For each path, the R2R dataset contains three natural language instruction descriptions. After analyzing the natural language instructions in the R2R dataset, this invention categorizes the actions of the agent based on orientation into several key steps: left, right, up, down, forward, and stop. These words effectively aid the agent's orientation perception during natural language instruction understanding and subsequent navigation, significantly influencing the agent's action selection. In the process of learning orientation perception using these words, this embodiment of the invention employs a pre-training method for mask prediction learning of the orientation words. The following masking prediction method was used for directional words in natural language instructions: 50% probability of replacing the directional word with a special identifier [MASK]; 50% probability of leaving the directional word unchanged. The model learns from the global visual image sequence and contextual instructions to predict the masked directional words in reverse. Then, the model parameters are updated using backpropagation through cross-entropy loss and the KL divergence between the predicted and actual outputs. The loss function for direction-aware learning is shown in the formula:

[0070]

[0071] L Image =KL(Pre||Real)

[0072] L DPL =L Image +L Language

[0073] N represents the total number of locative words being masked, w i C refers to the i-th masked directional word. i It is the context information of the i-th masked word, P(w i |C i) represents the probability distribution of the predicted location words in MLM; KL represents the KL divergence; Pre and Real represent the semantic distributions of the predicted image output and the real image output, respectively; L DPL This represents the loss function for direction-aware learning. Table 1 shows some instructions from the R2R dataset.

[0074] Finally, this embodiment of the invention fine-tunes the execution path selection. After the above four-stage pre-training process, path selection is used as a downstream task to fine-tune the model, enabling it to acquire path selection capabilities in addition to its general visual language navigation capabilities. For each path-instruction pair, the path selection task uses a beam search method to randomly generate three erroneous paths that do not match the actual data. "+" indicates the actual path, and "-" indicates the incorrect path. Cross-entropy loss is used for training.

[0075]

[0076]

[0077] in, and The word embedding vector of the special marker [CLS] used for classification task in the i-th path and the image feature vector obtained by the VLN-BERT model from the first image fragment [I MG] are represented by W, which is a learnable parameter matrix used to calculate the path score.

[0078] In this embodiment of the invention, the method improves the agent's orientation perception ability by incorporating orientation perception learning into the visual-language multimodal alignment pre-training process of VLN-BERT, thereby enhancing the agent's alignment ability with natural language instructions and visual information, and ultimately improving navigation performance.

[0079] The embodiments of the present invention are given for the purposes of illustration and description. Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A robot visual language navigation method based on direction-aware learning, characterized in that, The robot vision-language navigation method based on the VLN-BERT model follows these steps: S1. Image and text embedding: Embedding operations are performed on the images and corresponding instruction texts in the R2R dataset to facilitate the subsequent alignment and fusion of visual language information. S2. Perform four-stage pre-training, adding orientation awareness learning to the three-stage pre-training process of VLN-BERT, thereby improving the agent's orientation awareness ability; The four-stage pre-training process includes: Stage 4, in which: For the general VLN-BERT model, a direction-aware learning method is used to enhance the model's ability to perceive direction in navigation commands, thereby improving the model's performance on navigation tasks. During the pre-training process for visual-language multimodal alignment, mask prediction learning of directional words is performed on the agent, enabling the agent to more efficiently and accurately align the relationship between directional words and visual information during actual navigation. In the direction-aware learning process for directional words, a pre-training method of mask prediction learning is adopted. The following mask prediction method is used for directional words in natural language commands: With a 50% probability, the directional word is replaced with a special identifier [MASK], and with a 50% probability, the directional word remains unchanged. The model learns from global visual image sequences and contextual instructions to predict the masked directional word in reverse. Finally, the model parameters are updated using backpropagation based on cross-entropy loss and the KL divergence between the predicted and actual image outputs. The loss function for direction-aware learning is shown in the following equation: ； ； ； exist In the formula, The total number of locative words that are masked. Refers to the i-th masked location word. It is the contextual information of the i-th masked word. This represents the probability distribution for the prediction of masked directional words in MLM; In the formula, KL represents the KL divergence, and Pre and Real represent the semantic distributions of the predicted and real image outputs, respectively; In the formula, The loss function represents direction-aware learning; S3. Fine-tune the execution path selection: Enable the intelligent agent, which already has the ability to understand general visual language, to perform subsequent navigation tasks through path selection. The execution path selection fine-tuning, after a four-stage pre-training process, uses path selection as a downstream task to fine-tune the model, enabling it to acquire path selection capabilities in addition to general visual language navigation capabilities. For each path-instruction pair, the path selection task uses beam search to randomly generate three incorrect paths that do not match the actual data. The "+" subscript indicates the actual path, and the "-" subscript indicates the incorrect path. Training is performed using cross-entropy loss, as shown in the following formula: ； ； Where S is the path score and f is a function. and This represents the word embedding vector of the special marker [CLS] used for classification in the i-th path and the image feature vector obtained from the first image fragment [IMG] using the VLN-BERT model. is a learnable parameter matrix.

2. The robot visual language navigation method based on direction-aware learning as described in claim 1, characterized in that, The image and text embedding method includes: Image embedding method: The edge information of the image data, image segments, agent orientation and the sine and cosine values ​​of the pitch angle, and the relevant information of the next position are combined together and transformed into a 2048-dimensional input vector. The image segment index information is transformed into another 2048-dimensional vector. The pre-trained Faster R-CNN network is used to perform feature extraction on the image features. The features and the two vectors mentioned above together form the visual embedding effect. The text embedding method is the same as BERT, which adds the word embedding vector and the position embedding vector to obtain the embedding result.

3. The robot visual language navigation method based on direction-aware learning as described in claim 2, characterized in that, The four-stage pre-training process includes: Stage 1, in which: The ViLBERT model was trained using a language monomodal method with text information datasets extracted from the Internet, including Wikipedia and BooksCorpus. MLM and NSP were used to enable the model to perform word prediction and contextual understanding, giving it general language understanding capabilities.

4. The robot visual language navigation method based on direction-aware learning as described in claim 3, characterized in that, The four-stage pre-training process includes: Stage 2, in which: Building upon the ViLBERT model's language understanding capabilities, the model is trained across visual language modalities using the Conceptual Captions text-image dataset. By leveraging MMM and MAP, the model can infer local images of masks based on panoramic image information and perform matching and alignment between image content understanding and natural language descriptions, thus enabling the model to possess general visual language understanding capabilities.

5. The robot visual language navigation method based on direction-aware learning as described in claim 4, characterized in that, The four-stage pre-training process includes: Stage 3, in which: The ViLBERT model, which already possesses general visual language understanding capabilities, was trained using the R2R dataset. The MMM and MAP methods were used to enable the model to learn basic actions in navigation, thus giving the model general visual language understanding and navigation capabilities.

Images