Visual language navigation method based on action improved pre-training

Abstract

The invention discloses a visual language navigation method based on action improved pre-training. Comprising the following steps: S1, constructing a model which comprises an instruction encoder, an observation encoder, a history encoder and a cross-modal encoder and is used for realizing alignment and fusion of visual and language information; s2, pre-training the model by adopting three pre-training tasks of mask action word modeling, orientation discrimination modeling and scene enhancement modeling; and S3, performing fine tuning on the pre-trained model under the R2R data set by using imitation learning and reinforcement learning. According to the method, the understanding capability of the robot on action words and the alignment capability on visual information and orientation details are improved through three pre-training tasks, so that the action selection capability of the robot in the navigation process is improved, and finally the navigation success rate and the navigation efficiency of the robot in the real navigation process are improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of visual language navigation, and particularly relates to a visual language navigation method based on action improved pre-training. BACKGROUND

[0002] One long-term goal of artificial intelligence is to achieve embodied intelligence, that is, robots can interact with the environment and perform various complex tasks. Vision-and-Language Navigation (VLN) is a key step towards embodied intelligence. Since the natural language instructions provided by humans contain multiple navigation actions, at each step of selecting an action, the robot needs to judge the executed instructions and unexecuted instructions according to the historical information and the current visual observation, and then select the most appropriate navigation action for the current step. Once the robot incorrectly aligns the current visual information with the orientation details in the instructions, it will select the wrong navigation action, which may lead to the failure of the entire navigation task. Therefore, improving the alignment of visual information and the corresponding orientation details in the instructions plays an important role in the robot selecting the correct navigation action.

[0003] With the development of visual language pre-training, multi-modal models such as LXMERT, CLIP, and GPT-4 are used as models for vision-and-language navigation. In addition, corresponding pre-training tasks have been proposed to enhance the perception and decision-making ability of robots. However, these pre-training tasks only roughly utilize visual information and instructions to supervise the action selection of robots, without focusing on the alignment of visual information and orientation details in the instructions. In vision-and-language navigation, the robot needs to rely on visual and orientation information to issue correct control commands. If the robot only focuses on one aspect and ignores the other aspect, it is likely to make a wrong action, which may lead to the failure of the entire navigation task.

[0004] Therefore, a visual language navigation method based on action improved pre-training is urgently needed. SUMMARY

[0005] To solve the defects existing in the prior art, the application provides a visual language navigation method based on action improved pre-training.

[0006] To solve the above technical problems, the application provides the following technical solutions:

[0007] The application provides a visual language navigation method based on action improved pre-training, comprising the following steps:

[0008] S1, constructing a model, the model comprising an instruction encoder, an observation encoder, a history encoder, and a cross-modal encoder, for realizing the alignment and fusion of visual and language information;

[0009] S2, pre-training the model by using three pre-training tasks of mask action word modeling, orientation discrimination modeling and scene enhancement modeling;

[0010] S3, fine-tuning the pre-trained model under the R2R dataset by using imitation learning and reinforcement learning.

[0011] Preferably, the instruction encoder encodes the instruction information in the step S1.

[0012] The observation encoder encodes the panoramic information observed in the current step.

[0013] The history encoder encodes the panoramic information observed by the robot in each step before the current step as the historical information.

[0014] The cross-modal encoder encodes the instruction information, the observed panoramic information and the historical information at the same time.

[0015] Preferably, the mask action word modeling in the step S2 comprises the following steps:

[0016] S211, filtering out commonly used action words from the R2R dataset, the action words including navigation direction words and specific action words;

[0017] S212, randomly covering the action words by using the same mask probability as the BERT model in the pre-training stage, so that the model predicts the covered words according to the uncovered context and navigation trajectory;

[0018] S213, updating the model parameters by using a cross-entropy loss function; the cross-entropy loss function is as follows:

[0019]

[0020] Wherein, M represents a set of mask word positions, ω m represents a covered word, ω \m represents an uncovered word, and T represents a navigation trajectory.

[0021] Preferably, the orientation discrimination modeling in the step S2 comprises the following steps:

[0022] S221, for the correct view, keeping the angle information unchanged and randomly replacing the correct picture with other pictures, or keeping the picture information unchanged and randomly replacing the correct angle with an incorrect angle;

[0023] S222, adding the pseudo-label generated in the step S221 to the end of the observation information to obtain updated observation information, and the updated observation information is used as the action space of the robot.

[0024] S223, under the updated observation information, the robot aligns the picture in the visual information and the corresponding angle with the position details in the instruction information, and selects the correct action from the action space added with the pseudo label;

[0025] S224, the cross-entropy loss in the action selection process is calculated to update the model parameters;

[0026]

[0027] wherein, represents the optimal action, X represents the instruction information, H t represents the historical information, O' t represents the updated observation information.

[0028] Preferably, the scene enhancement modeling in step S2 comprises the following steps:

[0029] S231, image information of each position in the simulation environment is obtained;

[0030] S232, for each position of the candidate view, the image-to-image generation mechanism of the Stable-Diffusion-v1-4 model is used, the original image is disturbed in the latent space by setting appropriate generation intensity parameters, so as to realize the local style change of the image while retaining the room structure and spatial layout high-level semantic information; the generated new image is guided by the prompt word to generate a new image which is the same in semantic structure but different in visual style from the original image; Figure One

[0031] S233, the generated image is used to replace the original image, and the model selects the action according to the updated panoramic information;

[0032] S234, the cross-entropy loss of the scene enhancement modeling is calculated to update the model parameters, and the cross-entropy loss function is as follows:

[0033]

[0034] wherein, O″ t is the observation information replaced by the generated image.

[0035] Compared with the prior art, the present application has the following beneficial effects:

[0036] The present application improves the understanding ability of the robot to action words and the alignment ability of the robot to visual information and position details through three pre-training tasks, thereby improving the action selection ability of the robot in the navigation process, and finally improving the navigation success rate and navigation efficiency of the robot in the real navigation process. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 ​is a structural block diagram of a model in the present application;

[0038] Figure 2 is a schematic diagram of a position discrimination modeling task construction in the present application;

[0039] Figure 3 is a comparison diagram of image generation before and after a scene enhancement modeling task in the present application. DETAILED DESCRIPTION

[0040] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to explain and illustrate the present application, and are not used to limit the present application.

[0041] The embodiment provides a visual language navigation method based on action improved pre-training, and the method comprises the following steps:

[0042] S1, a model is constructed, the model comprises an instruction encoder, an observation encoder, a history encoder and a cross-modal encoder, and is used for realizing alignment and fusion of visual and language information.

[0043] In the embodiment, the instruction encoder encodes instruction information. The instruction encoder maps the instruction information provided by a human to a high-dimensional space, for subsequent multi-modal information fusion. Before embedding the instruction, the instruction encoder adds a [CLS] mark at the beginning of each instruction and a [SEP] mark at the end of the instruction, respectively representing global information and the end of the instruction. After applying word embedding and position embedding, the instruction is input into a BERT model to extract language features, and the final result is represented as X=[x CLS ,x1,x2,…,x L , x SEP ].

[0044] In the embodiment, the observation encoder encodes panoramic information observed at a current step, and the panoramic information is composed of 36 discrete views, and the shooting positions include three pitch angles (-30°, 0°, 30°) and twelve heading angles (-180°, -150°, -120°, -90°, -60°, -30°, 0°, 30°, 60°, 90°, 120°, 150°). The panoramic information is represented as wherein O t represents panoramic information, represents picture information of each view, represents corresponding angle information, and the angle information includes a pitch angle and a heading angle

[0045] At each navigation step, the observation encoder first extracts visual features of each view using the ViT-B / 16 model, and then extracts the [CLS] token in it to represent the global information of each view. These [CLS] tokens, together with the zero vector representing the "stop" action, constitute the total visual features. Since the angle information of each view is different, the observation encoder maps the angle information of each view to a high-dimensional space as the position encoding of each view. Finally, the visual features and the position encoding are added to form the final panoramic feature, and the specific calculation process is represented as:

[0046]

[0047]

[0048] wherein, represents the visual features of each view, represents the total visual features, W v and W g represent the weights of the linear layer, represents the visual features of the [CLS] token of each view (i.e., the global visual features of each view), represents the angle information of each view, F t represents the panoramic feature.

[0049] In this embodiment, the history encoder encodes the panoramic information observed by the robot at each step before the current step as historical information. First, the panoramic features of each step before the current step are averaged into a token h k , k ∈ {1, 2,..., t-1} using a multi-layer Transformer encoder. Then, these asynchronous tokens are encoded using a time-series Transformer encoder. Finally, a [CLS] token is added at the beginning to form the final history vector, denoted as H t = [h CLS , h1, h2,..., h t-1 ].

[0050] In this embodiment, the cross-modal encoder encodes the instruction information, the observed panoramic information and the historical information simultaneously. The cross-modal encoder consists of a language encoder and a visual encoder. Each encoder contains a cross-attention layer, a self-attention layer and a feedforward network. The cross-attention layer realizes the interaction between visual information and language information by exchanging key and value vectors between the two modalities. The calculation process is as follows:

[0051]

[0052] After the cross-modal encoder outputs the result, [CLS] tag in the output and the panoramic features Performing Hadamard product, we get And input into a two-layer feedforward neural network to calculate the logits of each view and get Q′ t Finally, Q′ t It is input into the Softmax layer to calculate the probability, and the action with the highest probability is selected as the navigation action of the robot's current step. The calculation process is as follows:

[0053]

[0054]

[0055] Among them, P t is the probability vector, is the optimal action selected.

[0056] S2. The model is pre-trained using three pre-training tasks: masked action word modeling, orientation discrimination modeling, and scene enhancement modeling.

[0057] In this embodiment, masked action word modeling includes the following steps:

[0058] S211, screening out commonly used action words from the R2R dataset, where the action words include navigation direction words and specific action words;

[0059] S212. In the pre-training phase, the action words are randomly masked using the same masking probability as the BERT model, allowing the model to predict the masked words based on the unmasked context and navigation trajectory.

[0060] S213. Update the model parameters using the cross entropy loss function. The cross entropy loss function is as follows:

[0061]

[0062] Where M represents the set of masked vocabulary positions, ω m Indicates the masked word, ω \m represents unmasked words, and T represents the navigation trajectory.

[0063] In this embodiment, if Figure 2 As shown, the orientation recognition modeling in step S2 includes the following steps:

[0064] S221: For the correct view, keep the angle information unchanged and randomly replace the correct image with another image; or keep the image information unchanged and randomly replace the correct angle with an incorrect angle;

[0065] S222: The pseudo label generated in step S221 The added to the end of the observation information, get the updated observation information, the updated observation information as the action space of the robot;

[0066] S223, under the updated observation information, the robot aligns the picture in the visual information and the corresponding angle with the orientation details in the instruction information, and selects the correct action from the action space added with the pseudo label;

[0067] S224, calculate the cross-entropy loss in the action selection process to update the model parameters;

[0068]

[0069] wherein, represents the optimal action, X represents the instruction information, H t represents the historical information, O′ t represents the updated observation information.

[0070] In the embodiment, the step S2 of scene enhancement modeling comprises the following steps:

[0071] S231, obtain image information of each position in the simulation environment;

[0072] S232, for each position of the candidate view, an image-to-image generation mechanism of the Stable-Diffusion-v1-4 model is used, the original image is disturbed in the latent space by setting appropriate generation intensity parameters, so as to realize local style change of the image while preserving the room structure and spatial layout high-level semantic information; at the same time, in order to control the content change range in the generation result, the model is guided to generate new images which are the same in semantic structure but different in visual style by using prompt words such as “keep the original room type and layout, but change the colors and shapes of the furniture” and the like; Figure One

[0073] S233, use the generated image to replace the original image, and let the model select the action according to the updated panoramic information. The generation effect is shown in Figure 3 Compared with the original image, the details such as the colors and shapes of the furniture and the content of the painting in the new image are changed. Such fine-grained differences can effectively alleviate the problem of over-emphasizing specific scene features during the training stage of the model, and promote the model to focus on the overall semantic information;

[0074] S234, calculate the cross-entropy loss of the scene enhancement modeling to update the model parameters, and the cross-entropy loss function is as follows:

[0075] ​

[0076] wherein O" t is the observation information replaced by the generated image.

[0077] S3, using imitation learning and reinforcement learning, fine-tuning the pre-trained model under the R2R dataset. For imitation learning, the model is trained using the instruction-trajectory pairs annotated by humans in the R2R dataset, and the model parameters are updated by calculating the cross-entropy loss to backpropagate, so that the model can learn the action strategy embodied by humans in the indoor navigation task. For reinforcement learning, the A2C (Advantage Actor-Critic) algorithm is used to further optimize the navigation strategy, and the reward function is constructed by setting the reward mechanism.

[0078] Finally, it should be noted that: the above only for the preferred embodiments of the present application, and not for the purpose of limiting the present application, although the foregoing embodiments of the present application have been described in detail, for those skilled in the art, it still can be modified, or part of the technical features of the equivalent replacement of the technical solutions recorded in the foregoing embodiments. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included within the scope of protection of the present application.

Claims

1. A visual language navigation method based on action-improved pre-training, characterized in that: The following steps are involved: S1. Build a model, which includes an instruction encoder, an observation encoder, a history encoder, and a cross-modal encoder to achieve alignment and fusion of visual and language information; S2, pre-train the model using three pre-training tasks: masked action word modeling, orientation discrimination modeling, and scene enhancement modeling; S3. Use imitation learning and reinforcement learning to fine-tune the pre-trained model on the R2R dataset.

2. The visual language navigation method based on action-improved pre-training according to claim 1, characterized in that: In step S1, the instruction encoder encodes instruction information; The observation encoder encodes the panoramic information observed in the current step; The history encoder encodes the panoramic information observed by the robot at each step before the current step as history information; The cross-modal encoder simultaneously encodes instruction information, observed panoramic information and historical information.

3. The visual language navigation method based on action-improved pre-training according to claim 2, characterized in that: The masked action word modeling in step S2 includes the following steps: S211, screening out commonly used action words from the R2R dataset, where the action words include navigation direction words and specific action words; S212. In the pre-training phase, the action words are randomly masked using the same masking probability as the BERT model, allowing the model to predict the masked words based on the unmasked context and navigation trajectory. S213. Update the model parameters using the cross entropy loss function. The cross entropy loss function is as follows: Where M represents the set of masked vocabulary positions, ω m Indicates the masked word, ω \m represents unmasked words, and T represents the navigation trajectory.

4. The visual language navigation method based on action-improved pre-training according to claim 1, characterized in that: The orientation discrimination modeling in step S2 includes the following steps: S221: For the correct view, keep the angle information unchanged and randomly replace the correct image with another image; or keep the image information unchanged and randomly replace the correct angle with an incorrect angle; S222: The pseudo label generated in step S221 Add it to the end of the observation information to obtain the updated observation information, and the updated observation information is used as the robot's action space; S223. Based on the updated observation information, the robot aligns the image and the corresponding angle in the visual information with the orientation details in the instruction information, and selects the correct action from the action space with the pseudo-label added. S224, calculate the cross entropy loss in the action selection process to back-propagate and update the model parameters; in, represents the optimal action, X represents the instruction information, H t Represents historical information, O′ t Indicates updated observation information.

5. The visual language navigation method based on action-improved pre-training according to claim 1, characterized in that: The scene enhancement modeling in step S2 includes the following steps: S231, obtaining image information of each position in the simulation environment; S232. For each candidate view at each position, the image-to-image generation mechanism of the Stable-Diffusion-v1-4 model is used to perturb the original image's latent space by setting appropriate generation strength parameters. This allows for local style changes in the image while preserving high-level semantic information about the room structure and spatial layout. The model is then guided by prompt words to generate a new image that is semantically consistent with the original image but has a different visual style. S233, using the generated image to replace the original image, and letting the model select an action based on the updated panoramic information; S234. Calculate the cross entropy loss of scene enhancement modeling to update the model parameters. The cross entropy loss function is as follows: Among them, O t is the observation information replaced by the generated image.

Images