A visual language navigation method based on follower model

Abstract

The present invention discloses a visual language navigation method based on a follower model. The follower model consists of an encoder and a decoder. Both the encoder and the decoder adopt a long short-term memory network structure. The encoder is used to preprocess visual image features, and the encoder is used to predict the next time step action. The follower model also includes a cross-modal contrast learning module and a visual variational autoencoder module. The follower model, the cross-modal contrast learning module and the visual variational autoencoder module together constitute an intelligent agent navigation model. The cross-modal contrast learning module is used to learn two cross-modal information, language and vision. The visual variational autoencoder module includes a visual encoder and a visual decoder, and is used to learn the distribution of visual image features. The loss function of the intelligent agent navigation model consists of Loss1, Loss2 and Loss3. Loss1 is the loss function of the visual variational autoencoder module, Loss2 is the loss function of the follower model, and Loss3 is the loss function of the cross-modal contrast learning module.

Description

Technical Field

[0001] This invention belongs to the field of intelligent algorithms and uses a deep learning approach to solve the visual language navigation problem in the interdisciplinary field of robotics and artificial intelligence. Specifically, it relates to a visual language navigation method based on a follower model. Background Art

[0002] Enable the robot to understand natural language instructions, and combine visual information in the real environment to perform corresponding actions to move to the specified destination. This simple navigation task is called Vision-and-Language Navigation (VLN) [1] VLNs are a crucial task, connecting language, vision, and navigation in unstructured, invisible environments. In practical terms, a robot that can clearly understand human language and intelligently move in a real-world environment can minimize human manipulation and supervision, performing repetitive manual labor or dangerous tasks for humans, such as bridge inspections and firefighting. Scientifically, exploring how robots can interpret human language and perceive visual environments can promote the interdisciplinary development of related fields such as computer vision, natural language processing, and robotics.

[0003] In the vision and language navigation (VLN) task, the room-to-room (R2R) task was the first to be noticed by researchers. Most of the research work on VLN is based on the R2R task and R2R dataset. The basic framework of these works is the sequence-2-sequence model, which focuses on using different innovative methods to improve the performance of the model. These include methods based on exploration strategies, methods based on data enhancement, and methods based on pre-trained models.

[0004] Regarding the exploration strategy-based approach, the focus of related research on exploration strategies is to enable the model to find an effective and fast path from the starting point to the end point. Ma et al. [2] A module is proposed to evaluate the entire process of the agent going to the goal. Based on this, Ma et al. [3] A “regret” module for moving forward or backward and a “progress marker” module to help the agent decide the next direction are designed. Although all current methods use beam search to make local action decisions or score the entire trajectory, Ke et al. [4] proposed a frontier-aware search (FAST) method with a backtracking navigator. When the agent finds itself lost, the FAST navigator can explicitly backtrack using asynchronous search. Huang et al. [5]Two auxiliary subtasks are defined: cross-modal alignment (CMA) and next visual scene (NVS), so that the visual and textual representations of an agent learned in a specific environment can be transferred to other environments. Zhu et al. [6] Four self-supervised auxiliary reasoning tasks are introduced to utilize additional training signals obtained from semantic information to help the agent understand the environment and tasks, thereby improving model performance.

[0005] Regarding data augmentation-based methods, Fried et al. [7] proposed a speaker-follower model for data augmentation and reasoning in supervised learning. Hong et al. [8] The benchmark R2R dataset is enriched with sub-commands and their corresponding paths, which can better provide sufficient semantic information for the agent during training. Agarwal et al. [9] proposed an “ongoing” speaker model that generates navigation instructions in two steps: first, a series of discrete visual landmarks are selected along the trajectory using a hard attention mechanism, and then language is generated conditioned on these landmarks. Unlike traditional data augmentation methods, Parvaneh et al.

[10] An efficient algorithm is proposed to generate counterfactual examples and add them to the training process, thereby improving the agent's ability to navigate in new environments. There is another application of counterfactual thinking.

[11] A model-agnostic method called adversarial path sampler is introduced to sample paths to gradually optimize the agent's navigation policy. Yu et al.

[12] They addressed the scarcity of data in the R2R task and eliminated the bias in the dataset by performing random walk data augmentation. By doing so, they reduced the generalization error and enabled the model to outperform the baseline model in unseen environments. An et al.

[13] A module named Neighbor View Enhancement Model is designed to adaptively fuse visual context from neighbor views at both global and local levels. Liu et al.

[14] proposed a random environment mixing method to decompose and reconstruct the environment and the corresponding path to construct a brand new environment as training data, narrowing the performance gap between visible and invisible environments and improving the overall performance. Sun et al.

[15] They point out that depth images as a valuable signal source for navigation have not been fully explored and are therefore often ignored in previous studies. Therefore, they propose an adaptive instance normalization module and a transfer attention module based on depth images to address this issue.

[0006] Regarding the method based on pre-training models, the general feature representation obtained by the pre-training model can be applied to various downstream tasks. This method has been verified in many research fields. A powerful pre-trained backbone network can effectively improve the performance of downstream tasks, such as image recognition tasks in the field of computer vision and question answering tasks in the field of natural language processing. In the field of VLN, Li et al.

[16] Through the pre-trained language model (BERT

[17] and GPT-3

[18] ) and random sampling to train the agent to improve the generalization performance of the model in unseen environments. Hao et al.

[19] A universal pre-trained navigator based on vision and language is developed, which is pre-trained with image, language and action triplets and fine-tuned on the R2R task to improve the generalization performance of the model in unknown environments. Due to the scarcity of training data in VLN tasks, Majumdar et al.

[20] Trying to use a large amount of data crawled from the Internet to solve this problem, proposed the VLN-BERT model, pre-trained the image-text pairs in the network and then fine-tuned it on the R2R task to significantly improve the performance of the VLN task. Hong et al.

[21] Proposed a multimodal BERT model The model has time-aware recursive functions, which provide richer information for the agent. Qi et al.

[22] We propose an object- and room-based sequential BERT model (ORIST) that improves language implementation by encoding visual and instruction inputs at the same fine-grained level (i.e., objects and words). The trained model can identify the relative orientation of navigation locations and the room types of the current and final navigation destinations.

[0007] The baseline model of the present invention is the speaker-follower model [7] The follower model in the image is embedded in the panoramic visual space. [1] The following describes the specific structure of the panoramic action space and follower model:

[0008] (1) Panoramic visual (action) space. In the follower model, the agent is allowed to directly execute high-level action instructions. That is, the agent can observe the complete panoramic visual space every time and then choose the next location to go to according to the high-level action instructions.

[0009] Compared with the ordinary visual space (the agent can only see the picture from the camera fixed on it), the panoramic visual space [7]Allows the agent to fully perceive the surrounding visual environment and directly execute high-level action instructions. In the panoramic visual space, the agent will observe all 360-degree panoramic visual information of the agent's surrounding environment before executing each high-level action instruction. For ease of processing, at the current time step t, the panoramic visual space is discretized into 36 perspectives (where horizontal rotation is divided into 12 parts according to the heading angle, and forward, upward and downward are divided into 3 parts according to the elevation angle, divided into 30 degrees). Each discretized perspective i (represented by the visual image of the perspective) will then be encoded into the corresponding encoded feature vector v t,i (i.e., visual image features). When the agent is at a certain position, due to the obstruction of certain objects in the panoramic visual space, the agent can only choose the direction that is not blocked to move. The agent's navigable direction J at each position (also represented by the visual image of the perspective) will also be encoded into the corresponding encoded feature vector U t,j (i.e. visual image features. Note that u t,j and v t,i Different in meaning, V t,i is all the coded information in the panoramic visual space, U t,j The visual image features in the panoramic visual space are extracted using a pre-trained ConvNet model (the original dataset already contains the extracted complete visual image features, so there is no need to extract them again). When the agent is at a certain position in the panoramic visual space, it can also obtain a four-dimensional directional feature vector Z related to its own posture. four =[sinψ,cosψ,sinθ,cosθ], where ψ is the heading angle and θ is the elevation angle. The final encoded feature vector v t,i and u t,j It is composed of visual image features and four-dimensional direction features, namely v t,i =[v t,i ,z four ],u t,j =[u t,j ,z four In order to allow the agent to decide when to stop while performing the navigation task, a special "STOP" encoding feature vector is introduced, represented by u0 = 0. When the agent believes that it has reached the destination, it can execute the "STOP" action to stop the navigation task.

[0010] (2) Specific structure of follower model

[0011] The follower model adopts the classic sequence-to-sequence structure, which consists of two parts: encoder and decoder. Figure 1 shown.

[0012] The basic structure of encoder and decoder is Long Short-Term Memory (LSTM)

[23] When performing a navigation task, a navigation path is first taken from the dataset (with a sentence describing the navigation path and all visual image features seen in the panoramic visual space through which the navigation path passes). The encoder is responsible for processing the sentence (that is, a sentence describing a path that the agent is required to perform navigation, expressed in English, for example: "go to the corner, then turn left."). The encoder encodes each English word in the input sentence to obtain the encoded language feature matrix Where N is the total number of words, d c is the feature dimension of each word after encoding.

[0013] (3) Preprocessing of visual image features of the follower model decoder

[0014] Before inputting the visual image features into the decoder at each time step t, the visual image features are preprocessed, that is, the memory vector h of the previous time step t-1 of the decoder is first t-1 With all visual image features v t,i Perform attention mechanism calculation, as shown in formula (2-1) and formula (2-2):

[0015] a t，i =(W1h t-1 +b1) T (W2v t，i +b2)#(2-1)

[0016]

[0017] where h t-1 is the memory vector of the decoder at the previous time step t-1, v t,i is the visual image feature of the panoramic visual space corresponding to the view angle i at the current time step. W1 and W2 are the corresponding parameter matrices, and b1 and b2 are the corresponding bias terms. t,i That is, the attention weights of different perspectives i calculated at the current time step t.

[0018] After obtaining the attention weights of different perspectives i, the attention weights are combined with the original v t,iMultiply the corresponding numbers and then add them together, as shown in formula (2-3):

[0019]

[0020] where v t,att is the attention feature vector calculated from the visual image features of all viewpoints i in the panoramic visual space at the current time step t.

[0021] (4) The follower model decoder part predicts the next time step action

[0022] The obtained v t,att and the visual image feature u of the predicted navigation direction at the previous time step t-1 t Perform splicing to obtain the splicing vector x t =[u t ,v t,att ], splicing vector x t Then input it into the decoder to reduce the dimension into the memory vector h of the current time step t t , then h t Perform attention calculation on the language feature matrix ctx output by the encoder before, obtain the attention weights of different words in the sentence, and then perform weighted summation on the different words of the original language feature matrix ctx according to the attention weights to obtain the weighted content vector ctx w , and finally ctx w and h t Splice them together to get h′ t , and then processed by a linear transformation function and tanh function to obtain the output memory vector of the decoder at the current time step t As shown in formula (2-4) and formula (2-5):

[0023] h′ t =[ctx w , h t ]#(2-4)

[0024]

[0025] Finally, the probability value of each navigable direction j at the current time step t is calculated using the product method of corresponding vector elements, as shown in Equations (2-6) and (2-7):

[0026]

[0027]

[0028] The probability value p j The larger the value is, the more likely the agent should go to the navigable direction j at the current time step t. Then according to pj Select the visual image feature u that should be used for the navigation direction at the current time step t t+1 (That is, the agent selects a navigable direction j and moves toward the corresponding location in that direction.) In particular, when the agent believes that it has reached the vicinity of the destination, it can execute the "STOP" encoding feature vector u0=0 to stop the current navigation task.

[0029] About contrastive learning method

[0030] Contrastive learning is a self-supervised learning model, a subtype of the unsupervised learning paradigm. Unlike conventional supervised learning, self-supervised learning's most significant feature is that it doesn't require manually annotated real-world labels. Instead, it directly uses raw input data as supervision to train the model and extract effective feature representations from the data. The trained model is then fine-tuned based on specific downstream tasks.

[0031] Contrastive learning generally involves processing raw data samples into positive and negative examples. The model then maps the raw, positive, and negative examples into the same latent space using the same encoding method for comparison. This approach reduces the distance between the raw data and positive examples in the latent space, while simultaneously increasing the distance between the raw data and negative examples. This allows the model to learn good feature representations of the raw data, helping improve the performance of downstream tasks. Therefore, a key issue in contrastive learning is how to properly construct positive and negative sample pairs.

[0032] The general paradigm of contrastive learning is to construct the input data into positive samples and negative samples, learn an encoder f, input the positive samples and negative samples into the encoder and encode them into the feature representation f(x) of the latent space, so that the encoder f satisfies formula (2-8):

[0033] score(f(x),f(x + ))>>score(f(x),f(x-))#(2-8)

[0034] Where x is the original data, x + is a positive example similar to x, x - is a negative example that is dissimilar to x. score(·) is a metric function or indicator used to measure the similarity between two data points. A commonly used metric function is the vector inner product. If the vector inner product is used to calculate the similarity between two data points, the loss function to be optimized by the comparative learning model is shown in Equation (2-9):

[0035]

[0036] From the above formula, we can see that one sample x corresponds to one positive sample x + and N-1 negative samples x - Formally, this loss function is equivalent to the conventional cross-entropy function, which is equivalent to performing an N-class classification task, where positive examples are classified as the correct category and negative examples are classified as the wrong category. The contrastive learning loss function shown above is generally referred to as the InfoNCE loss in the literature.

[0037] About the Variational Autoencoder Model

[0038] Variational Auto-Encoder (VAE)

[24] VAE is an unsupervised learning generative model. Given a set of data samples, VAE learns to fit the distribution function of these samples based on the input samples. The VAE then samples this distribution function and decodes and outputs new data samples, making the generated data diverse. VAE is developed based on the auto-encoder (AE).

[0039] Although AE can reconstruct input data, it cannot generate new data, while VAE can. The biggest difference between VAE and AE is that VAE introduces the idea of ​​probability distribution, which turns the original mapping relationship into a sampling process that obeys a certain probability distribution. The feature vector obtained by AE after encoding the input sample is a specific value, while VAE first obtains a probability distribution function that fits the input sample after encoding, and then samples a new feature vector from the distribution. The feature vector is then sent to the decoder for decoding, and finally outputs new data. The structure of VAE is as follows: Figure 2 shown.

[0040] Currently, VLN remains a burgeoning research field. Since the release of the Room-to-Room (R2R) dataset, numerous datasets and research works have been proposed for VLN tasks. However, many challenges remain. First, the model must effectively learn the alignment and matching relationship between two different modalities: language and vision. This allows the robot to understand natural language commands from humans and the visual information of the real environment, and to associate key visual features and appropriate actions in the real scene with the commands to achieve navigation. Second, there is the problem of data scarcity. It is often difficult to collect sufficient real-world data from real visual environments to fully reflect the visual diversity of the real world. This not only affects model training but also makes it difficult for the model to learn the optimal matching relationship between language and vision in interactive environments, thus affecting navigation performance. Finally, there is the issue of generalization performance. Due to the scarcity of real visual data, even if a model trained with limited training data has good navigation performance, its navigation performance will significantly degrade when deployed in an unknown, previously unobserved environment. In general, VLN is a new research direction that has only developed in recent years and is still a relatively difficult and challenging task. Summary of the Invention

[0041] The purpose of the present invention is to overcome the deficiencies in the prior art and provide a visual language navigation method based on a follower model.

[0042] The purpose of the present invention is achieved through the following technical solutions:

[0043] A visual language navigation method based on a follower model. The follower model consists of an encoder and a decoder, both of which adopt a long short-term memory network structure. The encoder is used to preprocess visual image features, and the decoder is used to predict the next time step action. The follower model also includes a cross-modal contrastive learning module and a visual variational autoencoder module. The follower model, the cross-modal contrastive learning module and the visual variational autoencoder module together constitute an intelligent agent navigation model. The cross-modal contrastive learning module is used to learn two cross-modal information, language and vision. The visual variational autoencoder module includes a visual encoder and a visual decoder, which are used to learn the distribution of visual image features.

[0044] The loss function of the intelligent agent navigation model consists of Loss1, Loss2 and Loss3. Loss1 is the loss function of the visual variational autoencoder module, which includes visual feature reconstruction loss and KL divergence to measure distribution similarity; Loss2 is the loss function of the follower model, and an additional hyperparameter that increases over time is added to constrain the intelligent agent navigation model to prevent the navigation error of the intelligent agent navigation model from accumulating too much over time. Loss3 is the loss function of the cross-modal contrastive learning module, which adopts the contrastive learning loss function.

[0045] Furthermore, the cross-modal contrastive learning module can shorten the distance between similar cross-modal information in the latent space and increase the distance between dissimilar cross-modal information in the latent space, so that the intelligent agent navigation model can effectively encode language information and visual information.

[0046] Furthermore, the visual variational autoencoder module first encodes the input visual image features to obtain a fitting distribution function, and decodes visual image features similar to the input visual image features from the fitting distribution function to increase the diversity of visual information during training and improve the generalization performance of the intelligent agent navigation model in unseen visual environments.

[0047] Furthermore, the visual encoder in the visual variational autoencoder module encodes the attention feature vector v t,att , output the memory vector h encoded at the current time step t e,t , then use h e,t Calculate the mean and logarithmic variance, as shown in formula (4-1) and formula (4-2):

[0048] μ t =W3h e，t +b3#(4-1)

[0049]

[0050] where μ t is the mean, is the logarithmic variance, W3 and W4 are the corresponding weight matrices, b3 and b4 are the corresponding bias terms; then a noise vector ε is randomly sampled from the high-dimensional standard Gaussian distribution n(0,i) t , calculate the sample memory vector to be input into the visual decoder As shown in formula (4-3):

[0051]

[0052] Input into the visual decoder, and after decoding, a new visual feature representation vector is reconstructed The loss function of the visual variational autoencoder module at each time step T is Loss t,1 It consists of two parts, namely the attention feature vector v t,att The reconstruction loss and the KL divergence loss of the visual encoder fitting distribution function are shown in formula (4-4):

[0053]

[0054]

[0055] Where d is the attention feature vector v t,att The number of dimensions, It is the d m The mean of the dimension, It is the d m dimensional variance, Loss1 is the Loss of all time steps t,1 The smaller the value of Loss1, the better the visual variational autoencoder module is trained.

[0056] Furthermore, in the follower model, the reconstructed and the visual image feature u of the predicted navigation direction at the previous time step t-1 t Perform splicing to obtain the splicing vector Splicing vector x t Then input it into the decoder of the follower model;

[0057] Calculate the cross entropy loss Loss′ between the predicted action and the true action label at each time step t t,2 ; When calculating Loss′ t,2 When , for Loss′ at different time steps t,2 , multiplied by an increasing weight γ related to the time step T t ; Weight γ t The calculation method is the same as multiplying the weight γ t Loss t,2 As shown in formula (4-6) and formula (4-7):

[0058]

[0059] Loss t，2 =γ t Loss' t，2 #(4-7)

[0060]

[0061] where l episode is the maximum number of time steps allowed by the agent navigation model when making action predictions, with hyperparameter β = 0.5, and Loss′t,2 is the original cross entropy loss, Loss t,2 is the weighted cross entropy loss, Loss2 is the Loss of all time steps t,2 The smaller the value of Loss2, the better the training of the agent navigation model.

[0062] Furthermore, in the cross-modal contrastive learning module, visual information refers to the memory vector h at each time step t encoded by the visual encoder. e,t The visual feature matrix H is composed of the spliced ​​together, and the language information refers to the language feature matrix ctx obtained after encoding by the encoder of the original follower model.

[0063] Furthermore, during the training of the agent navigation model, a small batch training method is adopted, and the number of data samples input into the agent navigation model each time is batch = 100; when a batch of data samples is completely processed in the agent navigation model, the language feature matrix ctx and the visual feature matrix H can be obtained, as follows:

[0064] Align the language feature matrix ctx and the visual feature matrix H, as shown in Equations (4-9) and (4-10):

[0065] Q'=ctx T ctx#(4-9)

[0066] K'=H T H#(4-10)

[0067] Then, the aligned language feature matrix Q′ and visual feature matrix K′ are subjected to the softmax function for each row d i Normalize and get W Q and W K , W Q and W K Each row of d i As shown in formula (4-11) and formula (4-12):

[0068]

[0069]

[0070] Then the matrix W Q Perform the product of each row element with the matrix Q′ and then sum it to get the vector q; K Perform the product of each row element with the matrix K′ and then sum it to get the vector k; each dimension d of q and k j (Indicates the d j Dimension) as shown in formula (4-13) and formula (4-14):

[0071]

[0072]

[0073] Vector q is equivalent to the original sample in the contrastive learning paradigm, called original sample q, vector k is equivalent to the positive sample in the contrastive learning paradigm, called positive sample k, and there is also a queue Q randomly initialized by the standard normal distribution at the beginning of training for storing negative samples equivalent to the contrastive learning paradigm. The length is K, and K is the number of negative samples stored in queue Q. Whenever a batch of data samples completes the calculation of all loss functions in the intelligent agent navigation model, the data samples of the batch are stored in the head of queue Q, and the same number of old data samples stored in the tail of queue Q are deleted at the same time, so as to achieve the purpose of dynamically updating queue Q as the training process progresses;

[0074] Calculate the similarity score between the original sample q and the positive sample k pos =q T k, calculate the similarity score between the original sample q and the negative sample queue Q neg =q T Q, then score pos and score neg Splicing them together, we get the similarity vector score where index value 0 is the positive example similarity and other index values ​​are the negative example similarity total =[score pos ,score neg ];

[0075] The loss function of the contrastive learning model is converted into a classification cross entropy loss function Loss3 for a classification task with K+1 categories, as shown in formula (4-15):

[0076]

[0077] in It is the real d k The probability value of the class, the smaller the value of Loss3, the better the training of the cross-modal contrastive learning module.

[0078] The present invention also provides a visual language navigation device, comprising:

[0079] The follower model unit is used to preprocess the visual image features through the encoder and predict the next time step action through the decoder;

[0080] Cross-modal contrastive learning module, used to learn cross-modal information of language and vision;

[0081] A visual variational autoencoder module for learning the distribution of visual image features;

[0082] The intelligent agent navigation model solving unit is used to solve the intelligent agent navigation model using the corresponding loss functions of the follower model unit, cross-modal contrastive learning module and visual variational autoencoder module.

[0083] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the follower model-based visual language navigation method when executing the program.

[0084] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the visual language navigation method based on a follower model.

[0085] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0086] The cross-modal contrastive learning module added by the present invention learns the matching relationship between language instructions and cross-modal information of visual image sequences during training, shortens the distance between similar cross-modal information in the latent space, and increases the distance between dissimilar cross-modal information in the latent space, so that the model can encode language and visual information more effectively.

[0087] The visual variational autoencoder module added by the present invention can learn the probability distribution of visual image sequences during training and reconstruct similar new visual image sequences. Even if the same data samples are input each time during training, since the input to the action prediction decoder is a reconstructed similar but not identical visual feature representation obtained through probability distribution sampling, the effect is equivalent to increasing the diversity of limited data samples. Therefore, during the training process, the model can reasonably infer unseen visual information that is different from but similar to the observed visual information based on the known visual image information, thereby improving the generalization performance of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is a structural diagram of the follower model in the prior art.

[0089] Figure 2 It is a structural diagram of the variational autoencoder VAE in the prior art.

[0090] Figure 3 It is a schematic diagram of the contrast-visionVAE-follower structure of the intelligent agent navigation model of the present invention.

[0091] Figure 4 Schematic diagram of the structure of the visual variational autoencoder module visionVAE in the present invention.

[0092] Figure 5 This is a schematic diagram of the computational process of the cross-modal contrastive learning module. DETAILED DESCRIPTION

[0093] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0094] This embodiment targets the room-to-room (R2R) task in visual language navigation (VLN) and is based on the original follower model. [7] , provides a visual language navigation method, based on the follower model, the cross-modal contrastive learning module (Cross Modal Contrastive Learning, CMCL) and the visual variational autoencoder module (visionVAE) together form an agent navigation model. For the sake of simplicity, the following content will refer to the agent navigation model as the contrast-visionVAE-follower model, the cross-modal contrastive learning module as the CMCL model, and the visual variational autoencoder module as the visionVAE.

[0095] (1) In order to learn the matching relationship between language and vision (language is one modality and vision is another modality), a cross-modal contrastive learning module (CMCL) is added to the agent navigation model. This module shortens the distance between similar cross-modal information in the latent space and increases the distance between dissimilar cross-modal information in the latent space, enabling the agent navigation model to more effectively encode language and visual information.

[0096] (2) A new visual variational autoencoder module (visionVAE) is added before the data is input to the decoder of the original follower model to learn the distribution of visual image features. VisionVAE first encodes the input visual image features to obtain a fitted distribution function, and then decodes new visual image features similar to the input visual image features from the fitted distribution function to increase the diversity of visual information during training and improve the generalization performance of the intelligent navigation model in unseen visual environments;

[0097] (3) The loss function of the improved agent navigation model consists of Loss1, Loss2, and Loss3. Loss1 is the loss function of visionVAE, which includes the visual feature reconstruction loss and the KL divergence that measures the distribution similarity. Loss2 is the loss function of the original follower model, but with a new hyperparameter γ that increases over time. t The purpose is to constrain the agent's navigation model and prevent the model's navigation error from accumulating too much over time. Loss3 is the loss function of CMCL, which uses the classic contrastive learning loss function.

[0098] The framework of the agent navigation model (contrast-visionVAE-follower) proposed in this embodiment is as follows Figure 3 shown.

[0099] Specifically, the process of building the contrast-visionVAE-follower model is as follows:

[0100] The contrast-visionVAE-follower model proposed in this example consists of three main components: the encoder and decoder of the original follower model, a newly added visual variational autoencoder module (visionVAE, which consists of a visual encoder and a visual decoder), and a cross-modal contrastive learning module (CMCL). The final loss function also consists of three parts: Loss1, Loss2, and Loss3.

[0101] (1) Newly added vision variational autoencoder module (visionVAE)

[0102] In the contrast-visionVAE-follower model, the newly added visionVAE follows the basic idea of ​​VAE and is used to learn the fitting distribution function of the visual image features of the input data samples to increase the diversity of visual image features during training and improve the generalization performance of the contrast-visionVAE-follower model in never-seen visual environments (i.e., unknown data other than the input data set). The structure of visionVAE in the contrast-visionVAE-follower model is as follows: Figure 4 shown.

[0103] (101) Preprocessing of visual image features (attention mechanism calculation)

[0104] The preprocessing of visual image features in the contrast-visionVAE-follower model is basically the same as that in the original follower model. The only difference is that in the original follower model, the memory vector h of the decoder at the previous time t-1 is used. t-1 With all visual image features v t,i The attention mechanism is calculated, and in the contrast-visionVAE-follower model, the memory vector h of the previous time t-1 of the visual encoder is replaced e,t-1 With all visual image features v t,i Perform attention mechanism calculation.

[0105] (102) visionVAE part

[0106] The visual encoder in visionVAE then encodes the attention feature vector v t,att , output the memory vector h encoded at the current time step t e,t (h e,t In addition to being used in the decoding process of the visual decoder, it will also be used in CMCL), and then use h e,t Calculate the mean and logarithmic variance, as shown in formula (4-1) and formula (4-2):

[0107] μ t =W3h e，t +b3#(4-1)

[0108]

[0109] where μ t is the mean, is the logarithmic variance, W3 and W4 are the corresponding weight matrices, and b3 and b4 are the corresponding bias terms. Then, in order to allow the gradient to be back-propagated, the "reparameterization" strategy commonly used in VAE is adopted to randomly sample a noise vector ε from the high-dimensional standard Gaussian distribution N(0,I) t , calculate the sample memory vector to be input into the visual decoder As shown in formula (4-3):

[0110]

[0111] Input into the visual decoder, and after decoding, a new visual feature representation vector is reconstructed The loss function of visionVAE at each time step t is Loss t,1 It consists of two parts, namely the attention feature vector v t,attThe reconstruction loss and the KL divergence loss of the visual encoder fitting distribution function are shown in formula (4-4):

[0112]

[0113]

[0114] Where d is the visual feature representation vector v t,att The number of dimensions, It is the d m The mean of the dimension, It is the d m dimensional variance. Loss1 is the Loss of all time steps t,1 The smaller the value of Loss1, the better the visionVAE is trained.

[0115] (2)Follower model part

[0116] The follower model structure in the contrast-visionVAE-follower model remains basically unchanged. and the visual image feature u of the predicted navigation direction at the previous time step t-1 t Perform splicing to obtain the splicing vector Splicing vector x t It is then input into the decoder of the original follower model, and the calculation process for predicting the action at the current time step is the same as that of the original follower model.

[0117] Calculate the cross entropy loss Loss′ between the predicted action and the true action label at each time step t t,2 When calculating Loss′ t,2 When , for Loss′ at different time steps t,2 , multiplied by an increasing weight γ related to the time step t t Because as the time step t increases, the deviation of the predicted action performed by the agent will gradually accumulate. If the later action is not predicted accurately, the agent will deviate further from the correct destination. Therefore, the weight γ t It can be used to penalize the agent for inaccurate predictions during training. t The calculation method is the same as multiplying the weight γ t Loss t,2 As shown in formula (4-6) and formula (4-7):

[0118]

[0119] Loss t，2 =γt Loss' t，2 #(4-7)

[0120]

[0121] where l episode is the maximum number of time steps allowed by the model when making action predictions, hyperparameter β = 0.5, Loss′ t,2 is the original cross entropy loss, Loss t,2 is the weighted cross entropy loss, Loss2 is the Loss of all time steps t,2 The smaller the value of Loss2, the better the contrast-visionVAE-follower model is trained.

[0122] (3) New Cross-Modal Contrastive Learning (CMCL) module

[0123] (301) Cross-modal Information Alignment in CMCL

[0124] In the contrast-visionVAE-follower model, in order to learn the matching relationship between language and vision (language is one modality and vision is another modality), the newly added CMCL is used to shorten the distance between similar cross-modal information in the latent space and increase the distance between dissimilar cross-modal information in the latent space, so that the contrast-visionVAE-follower model can encode language and visual information more effectively. This module mainly refers to a significant work in contrastive learning, namely, the work by He Kaiming et al.

[25] Proposed MoCo method.

[0125] The MoCo method believes that contrastive learning can be viewed as a process in which a training encoder performs a search for matching information in a dynamically changing dictionary. It proposes improving the method of using a memory bank to using a dynamically changing queue. This queue functions similarly to a memory bank and is also used to store feature representations in a dataset. The difference is that the data features stored in this queue are dynamically updated. During each training iteration, the queue is enqueued, and a batch of data features encoded by the current encoder is stored in the queue. At the same time, the data features of the batch with the longest storage time in the queue are deleted. Overall, the total number of data features stored in the queue remains unchanged for each training iteration, and the data features in the queue are updated as the training process progresses. In this way, we can flexibly set queues of different sizes according to needs, avoiding the problem of occupying a large amount of GPU memory.

[0126] CMCL is trained synchronously with the training of the entire contrast-visionVAE-follower model. In CMCL, visual information refers to the memory vector h at each time step t encoded by the visionVAE’s visual encoder. e,t The visual feature matrix H is composed of the spliced ​​together, and the language information refers to the language feature matrix ctx obtained after encoding by the encoder part of the original follower model. The calculation process of CMCL is as follows Figure 5 shown.

[0127] In actual training, a small batch training method is used, and the number of data samples input to the contrast-visionVAE-follower model at a time is batch = 100. After a batch of data samples is completely processed in the contrast-visionVAE-follower model, the matrices ctx and H are obtained. Since the last dimension of the matrix ctx is not the same as the last dimension of the matrix H, they need to be aligned first to make them equal, as shown in Equations (4-9) and (4-10):

[0128] Q'=ctx T ctx#(4-9)

[0129] K'=H T H#(4-10)

[0130] Then use the softmax function on the matrix Q′ and matrix K′ for each row d i (Indicates the d i Normalization is performed to obtain W Q and W K , W Q and W K Each row of d i (Indicates the d i (row) as shown in formula (4-11) and formula (4-12):

[0131]

[0132]

[0133] Then the matrix W Q Perform the product of each row element with the matrix Q′ and then sum it to get the vector q; K Perform the product of each row element with the matrix K′ and then sum it to get the vector k. Each dimension d of q and k j (Indicates the d j Dimension) as shown in formula (4-13) and formula (4-14):

[0134]

[0135]

[0136] Vector q is equivalent to the original sample in the contrastive learning paradigm, called original sample q, vector k is equivalent to the positive sample in the contrastive learning paradigm, called positive sample k, and there is also a queue Q randomly initialized by the standard normal distribution at the beginning of training for storing negative samples equivalent to the contrastive learning paradigm. The length is K, and K is the number of negative samples stored in queue Q. Whenever a batch of data samples completes the calculation of all loss functions in the intelligent agent navigation model, the data samples of the batch are stored in the head of queue Q, and the same number of old data samples stored at the tail of queue Q are deleted, so as to achieve the purpose of dynamically updating queue Q as the training process progresses.

[0137] Calculate the similarity score between the original sample q and the positive sample k pos =q T k, calculate the similarity score between the original sample q and the negative sample queue Q neg =q T Q, then score pos and score neg Splicing them together, we get the similarity vector score where index value 0 is the positive example similarity and other index values ​​are the negative example similarity total =[score pos ,score neg ].

[0138] (302) Loss function of CMCL part

[0139] Since the similarity of the positive examples is all in the first position in the vector, contrastive learning hopes to train the model so that the similarity value of the first position in the vector is large and the similarity values ​​of other positions are small. Therefore, the contrastive learning loss function can be converted into a classification cross entropy loss function Loss3 for a classification task with K+1 categories, as shown in formula (4-15):

[0140]

[0141] in It is the real d k The smaller the value of Loss3, the better the CMCL training.

[0142] (4) The final loss function of the entire contrast-visionVAE-follower model

[0143] The loss function Loss of the contrast-visionVAE-follower model consists of three parts: Loss1, Loss2, and Loss3, as shown in formula (4-16):

[0144] Loss=Loss1+Loss2+Loss3#(4-16)

[0145] During the training process, we hope that the Loss value is as small as possible.

[0146] Specifically, this embodiment uses the training set in the R2R dataset to perform end-to-end training on the contrast-visionVAE-follower model on the Matterport3D simulator, and uses the visible test set and invisible test set in the R2R dataset to test the navigation performance of the model.

[0147] (1) Matterport 3D Simulator

[0148] The experiments require training in the Matterport3D simulator. The Matterport3D simulator is a new large-scale visual reinforcement learning simulation environment that can be applied to scientific research and development of agents based on the R2R dataset

[26] . The panoramic action space in the Matterport3D simulator allows the agent to fully perceive the surrounding visual environment and directly execute high-level action commands.

[0149] (2) Dataset used in the experiment

[0150] The dataset used in this example is the R2R dataset. The R2R dataset consists of 7,189 paths sampled from the Matterport 3D simulator navigation map. Each path consists of 5 to 7 discrete viewpoints, and the average path length of all paths is 10 meters. Each path is accompanied by three human-language instruction sentences, totaling approximately 21,500 instructions, with each instruction averaging 29 words. The R2R dataset is divided into a training set, a validation set, and a test set. The validation set is further divided into a visible validation set and an invisible validation set. The data samples in the visible validation set appear in the training set, while the data samples in the invisible validation set never appear in the training set. All data samples in the test set have never appeared in the training set or validation set. The training set and visible validation set contain 61 scenes, 14,025 instruction data in the training set, and 1,020 instruction data in the visible validation set. The invisible validation set contains 11 scenes and 2,349 instructions, and the test set contains 18 scenes and 4,173 instructions. This experiment only uses the training set and the validation set.

[0151] (3) Training and testing

[0152] When training the model, the agent is given a natural language instruction from the R2R dataset. As input, where L is the length of the natural language instruction, x i is the word vector of each word in the natural language instruction. At the same time, the agent will also obtain the initial RGB image o0 corresponding to the current instruction by observing it in the simulator. The obtained initial RGB image is determined by the initial posture of the agent, that is, s0 =<v0,ψ0,θ0> , where s0 is the initial pose, v0 is the 3D position, ψ0 is the heading angle, and θ0 is the elevation angle. The agent must perform a series of actions 〈s0,a0,s1,a1,…,s T ,a T > to change its own posture or position, where each action a is executed t will cause the agent to go to the new state s t+1 = <v t+1 ,ψ t+1 ,θ t+1 > and make new observations to generate new RGB images o t+1 . When the agent performs a special "STOP" action, the current navigation task ends. If the executed action sequence finally allows the agent to move to a valid range close enough to the target position v*, the navigation task is considered successful, otherwise it fails. In the experiment, when training the model, one batch of data is input each time to update the parameters. After inputting the current batch of data for training and updating the parameters, the navigation performance is immediately tested on the complete visible validation set and the invisible validation set. This training is iterated 20,000 times to obtain the final trained model.

[0153] Multiple groups of control experiments were conducted on the basic follower model and the improved contrast-visionVAE-follower model in the Matterport3D simulator, and the experimental results and the performance of the improved model were analyzed.

[0154] (4) Performance evaluation indicators

[0155] The model performance evaluation indicators selected in this embodiment are as follows:

[0156] (1) Navigation error (NE): defined as the shortest straight-line distance between the agent’s last stopping position and the target position;

[0157] (2) Success rate (SR): In a specific navigation task, if the shortest straight-line distance between the agent's final stopping position and the target position is less than a threshold of 3 meters, the task is considered successful. When multiple navigation tasks are performed, the ratio of the number of successful navigation tasks to the total number of navigation tasks is the success rate.

[0158] Specifically, the experimental results of the follower model and the contrast-visionVAE-follower model obtained after multiple sets of experiments are shown in Table 1:

[0159] Table 1 Experimental results of this embodiment

[0160]

[0161]

[0162] In the model code, various random number initializations can be manually set using a random seed. This ensures the reproducibility of experimental results. Experiments with the follower model and the contrast-vision VAE-follower model with the same random seed parameter values ​​serve as a control experiment. When the random seed parameter is not manually set, it is randomly assigned by the computer.

[0163] In summary, after multiple sets of experimental analysis, the results show that the navigation success rate of the contrast-visionVAE-follower model in the visible verification dataset and the invisible verification dataset is improved compared with the baseline follower model, and the navigation error is reduced. On the one hand, the added cross-modal contrast learning module learns the matching relationship between language instructions and cross-modal information of visual image sequences during training, shortens the distance between similar cross-modal information in the latent space, and widens the distance between dissimilar cross-modal information in the latent space, so that the model can more effectively encode language and visual information. On the other hand, the added visual variational autoencoder module can learn the probability distribution of visual image sequences during training and reconstruct similar new visual image sequences. Even if the same data samples are input each time during training, since the input to the action prediction decoder is a reconstructed similar but not identical visual feature representation obtained by sampling the probability distribution, the effect is equivalent to increasing the diversity of limited data samples, so that the model can reasonably infer unseen visual information that is different but similar to the observed visual information based on the known visual image information during training, thereby improving the generalization performance of the model.

[0164] Preferably, an embodiment of the present application further provides a specific implementation of an electronic device capable of implementing all steps of the follower model-based visual language navigation method in the above embodiment, and the electronic device specifically includes the following contents:

[0165] Processor, memory, communications interface, and bus;

[0166] Among them, the processor, memory, and communication interface communicate with each other through the bus; the communication interface is used to realize information transmission between related devices such as server-side devices, metering devices, and user-side devices.

[0167] The processor is used to call the computer program in the memory, and when the processor executes the computer program, all the steps in the visual language navigation method based on the follower model in the above embodiment are implemented.

[0168] An embodiment of the present application also provides a computer-readable storage medium that can implement all the steps of the visual language navigation method based on the follower model in the above embodiment. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements all the steps of the visual language navigation method based on the follower model in the above embodiment.

[0169] The various embodiments in this specification are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other. Each embodiment focuses on the differences between the other embodiments. In particular, the hardware + program embodiments are generally similar to the method embodiments, so their description is relatively simple. For relevant portions, refer to the description of the method embodiments.

[0170] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0171] Although the present application provides method operation steps such as embodiments or flowcharts, more or fewer operation steps may be included based on conventional or non-creative work. The order of steps listed in the embodiments is only one way of executing the steps among many steps and does not represent the only execution order. When an actual device or client product is executed, it can be executed in the order shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0172] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0173] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0175] The present invention is not limited to the embodiments described above. The above description of the specific embodiments is intended to describe and illustrate the technical solutions of the present invention. The above specific embodiments are merely illustrative and not restrictive. Without departing from the scope of the present invention and the scope of protection of the claims, those skilled in the art may make various specific modifications based on the teachings of the present invention, all of which fall within the scope of protection of the present invention.

[0176] References

[0177] [1]Anderson P,Wu Q,Teney D,et al.Vision-and-language navigation:Interpreting visually-grounded navigation instructions in real environments[C] / / Proceedings of the IEEE conference on computer vision and patternrecognition.2018:3674-3683.

[0178] [2]Ma C Y,Lu J,Wu Z,et al.Self-monitoring navigation agent viaauxiliary progress estimation[J].arXiv preprint arXiv:1901.03035,2019.

[0179] [3]Ma C Y,Wu Z,AlRegib G,et al.The regretful agent:Heuristic-aidednavigation through progress estimation[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2019:6732-6740.

[0180] [4]Ke L,Li X,Bisk Y,et al.Tactical rewind:Self-correction viabacktracking in vision-and-language navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2019:6741-6749.

[0181] [5]Huang H,Jain V,Mehta H,et al.Transferable representation learningin vision-and-language navigation[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.2019:7404-7413.

[0182] [6]Zhu F,Zhu Y,Chang X,et al.Vision-language navigation with self-supervised auxiliary reasoning tasks[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition.2020:10012-10022.

[0183] [7]Fried D,Hu R,Cirik V,et al.Speaker-follower models for vision-and-language navigation[J].Advances in Neural Information Processing Systems,2018,31.

[0184] [8]Hong Y,Rodriguez-Opazo C,Wu Q,et al.Sub-instruction aware vision-and-language navigation[J].arXiv preprint arXiv:2004.02707,2020.

[0185] [9]Agarwal S,Parikh D,Batra D,et al.Visual landmark selection forgenerating grounded and interpretable navigation instructions[C] / / CVPRworkshop on Deep Learning for Semantic Visual Navigation.2019,2.

[0186]

[10] Parvaneh A,Abbasnejad E,Teney D,et al.Counterfactual vision-and-language navigation:Unravelling the unseen[J].Advances in Neural InformationProcessing Systems,2020,33:5296-5307.

[0187]

[11] Fu T J,Wang X E,Peterson M F,et al.Counterfactual vision-and-language navigation via adversarial path sampler[C] / / European Conference onComputer Vision.Springer,Cham,2020:71-86.

[0188]

[12] Yu F,Deng Z,Narasimhan K,et al.Take the scenic route:Improvinggeneralization in vision-and-language navigation[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition Workshops.2020:920-921.

[0189]

[13] An D,Qi Y,Huang Y,et al.Neighbor-view enhanced model for visionand language navigation[C] / / Proceedings of the 29th ACM InternationalConference on Multimedia.2021:5101-5109.

[0190]

[14] Liu C,Zhu F,Chang X,et al.Vision-language navigation with randomenvironmental mixup[C] / / Proceedings of the IEEE / CVF International Conferenceon Computer Vision.2021:1644-1654.

[0191]

[15] Sun Q,Zhuang Y,Chen Z,et al.Depth-Guided AdaIN and ShiftAttention Network for Vision-And-Language Navigation[C] / / 2021 IEEEInternational Conference on Multimedia and Expo(ICME).IEEE,2021:1-6.

[0192]

[16] Li X,Li C,Xia Q,et al.Robust navigation with language pretrainingand stochastic sampling[J].arXiv preprint arXiv:1909.02244,2019.

[0193]

[17] Vaswani A,Shazeer N,Parmar N,et al.Attention is all you need[J].Advances in neural information processing systems,2017,30.

[0194]

[18] Brown T,Mann B,Ryder N,et al.Language models are few-shotlearners[J].Advances in neural information processing systems,2020,33:1877-1901.

[0195]

[19] Hao W,Li C,Li X,et al.Towards learning a generic agent forvision-and-language navigation via pre-training[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition.2020:13137-13146.

[0196]

[20] Majumdar A,Shrivastava A,Lee S,et al.Improving vision-and-language navigation with image-text pairs from the web[C] / / EuropeanConference on Computer Vision.Springer,Cham,2020:259-274.

[0197]

[21] Hong Y,Wu Q,Qi Y,et al.A Recurrent Vision-and-Language BERT forNavigation.arXiv2021[J].arXiv preprint arXiv:2011.13922.

[0198]

[22] Qi Y,Pan Z,Hong Y,et al.Know what and know where:An object-and-room informed sequential bert for indoor vision-language navigation[J].arXive-prints,2021:arXiv:2104.04167.

[0199]

[23] Cheng J,Dong L,Lapata M.Long short-term memory-networks formachine reading[J].arXiv preprint arXiv:1601.06733,2016.

[0200]

[24] Kingma D P,Welling M.Auto-encoding variational bayes[J].arXivpreprint arXiv:1312.6114,2013.

[0201]

[25] He K,Fan H,Wu Y,et al.Momentum contrast for unsupervised visualrepresentation learning[C] / / Proceedings of the IEEE / CVF conference oncomputer vision and pattern recognition.2020:9729-9738.

[0202]

[26] Chang A,Dai A,Funkhouser T,et al.Matterport3d:Learning from rgb-ddata in indoor environments[J].arXiv preprint arXiv:1709.06158,2017.

Claims

1. A visual language navigation method based on a follower model. The follower model consists of an encoder and a decoder. Both the encoder and the decoder use a long short-term memory network structure. The encoder is used to preprocess visual image features, and the decoder is used to predict the next time step action. The characteristics are: It also includes a cross-modal contrastive learning module and a visual variational autoencoder module. The follower model, the cross-modal contrastive learning module and the visual variational autoencoder module together constitute an intelligent agent navigation model. The cross-modal contrastive learning module is used to learn two cross-modal information, language and vision; the visual variational autoencoder module includes a visual encoder and a visual decoder, which are used to learn the distribution of visual image features; The loss function of the agent navigation model is given by 、 and composition, is the loss function of the visual variational autoencoder module, including the visual feature reconstruction loss and the KL divergence that measures the distribution similarity; It is the loss function of the follower model, and an additional hyperparameter that increases over time is added to constrain the agent navigation model to prevent the navigation error of the agent navigation model from accumulating too much over time. It is the loss function of the cross-modal contrastive learning module. By adopting the contrastive learning loss function, the cross-modal contrastive learning module can shorten the distance between similar cross-modal information in the latent space and increase the distance between dissimilar cross-modal information in the latent space, so that the intelligent navigation model can effectively encode language information and visual information.

2. The visual language navigation method based on the follower model according to claim 1, characterized in that: The visual variational autoencoder module first encodes the input visual image features to obtain a fitting distribution function, and decodes visual image features similar to the input visual image features from the fitting distribution function to increase the diversity of visual information during training and improve the generalization performance of the intelligent agent navigation model in unseen visual environments.

3. A visual language navigation method based on a follower model according to claim 1 or 2, characterized in that: The visual encoder in the visual variational autoencoder module encodes the attention feature vector , output the current time step Encoded memory vector , then use Calculate the mean and logarithmic variance as shown in formula (4-1) and formula (4-2): ； ； in is the mean, is the logarithmic variance, and is the corresponding weight matrix, and is the corresponding bias term; then from the high-dimensional standard Gaussian distribution Randomly sample a noise vector from , calculate the sample memory vector to be input into the visual decoder , as shown in formula (4-3): ； Input into the visual decoder, and after decoding, a new visual feature representation vector is reconstructed ; Each time step The loss function of the visual variational autoencoder module is It consists of two parts, namely the attention feature vector The reconstruction loss and the KL divergence loss of the visual encoder fitting distribution function are shown in formula (4-4): ； ； in is the attention feature vector The number of dimensions, It is The mean of the dimension, It is The variance of the dimension, is all time steps The sum of The smaller the value of , the better the visual variational autoencoder module is trained.

4. The visual language navigation method based on the follower model according to claim 3, characterized in that: In the follower model, the reconstructed and the previous time step Visual image features for predicting navigation direction Perform splicing to obtain the splicing vector , the concatenated vector Then input it into the decoder of the follower model; Calculate each time step The cross entropy loss between the predicted action and the true action label ; In calculation For different time steps , multiplied by the time step The increasing weight of ; Weight The calculation method is the same as multiplying the weight After As shown in formula (4-6) and formula (4-7): ； ； ； in is the maximum number of time steps allowed by the agent navigation model when making action predictions, a hyperparameter , is the original cross entropy loss, is the weighted cross entropy loss, is all time steps The sum of The smaller the value of , the better the agent navigation model is trained.

5. The visual language navigation method based on the follower model according to claim 1, characterized in that: In the cross-modal contrastive learning module, visual information refers to each time step encoded by the visual encoder. Memory vector The visual feature matrix formed by splicing , language information refers to the language feature matrix obtained by encoding the encoder of the original follower model .

6. The visual language navigation method based on the follower model according to claim 5, characterized in that: In the process of training the intelligent agent navigation model, a small batch training method is adopted, and the number of data samples input to the intelligent agent navigation model each time is When a batch of data samples is completely processed in the agent navigation model, the language feature matrix can be obtained. and the visual feature matrix , as follows: Language feature matrix and the visual feature matrix Perform alignment processing, as shown in Equations (4-9) and (4-10): ； ； Then the aligned language feature matrix and the visual feature matrix Use the softmax function for each row Normalization is performed to obtain and , and Each line As shown in formula (4-11) and formula (4-12): ； ； Then the matrix With the matrix Perform the product of each row element and then sum it to get the vector ; The matrix With the matrix Perform the product of each row element and then sum it to get the vector ; and Each dimension of , indicating the Dimension, as shown in Equations (4-13) and (4-14): ； ； vector Equivalent to the original sample in the contrastive learning paradigm, called the original sample ,vector Equivalent to the positive sample in the contrastive learning paradigm, called the positive sample , and a queue randomly initialized from a standard normal distribution at the beginning of training to store negative samples equivalent to the contrastive learning paradigm , the length is , That is, the number of negative samples stored in the queue Q. Whenever a batch of data samples completes the calculation of all loss functions in the intelligent navigation model, the data samples of the batch are stored in the queue The head of the queue and delete the queue at the same time The same number of old data samples are stored at the end of the queue, so that the queue can be dynamically updated as the training process progresses. purpose; Calculate the original sample and positive examples Similarity , calculate the original sample and negative sample queues Similarity , and then and Put them together to get the similarity vector where index value 0 is the positive similarity and other index values ​​are the negative similarity. ; Convert the loss function of the contrastive learning model into a Classification cross entropy loss function for classification tasks with multiple categories , as shown in formula (4-15): ； in It's the real The probability value of the class, The smaller the value of , the better the cross-modal contrastive learning module is trained.

7. A visual language navigation device, characterized in that: include: The follower model unit is used to preprocess the visual image features through the encoder and predict the next time step action through the decoder; Cross-modal contrastive learning module, used to learn cross-modal information of language and vision; The visual variational autoencoder module includes a visual encoder and a visual decoder, which are used to learn the distribution of visual image features; The follower model unit, cross-modal contrastive learning module and visual variational autoencoder module together constitute the intelligent agent navigation model module; the loss function of the intelligent agent navigation model module is given by 、 and composition, is the loss function of the visual variational autoencoder module, including the visual feature reconstruction loss and the KL divergence that measures the distribution similarity; It is the loss function of the follower model unit, and an additional hyperparameter that increases over time is added to constrain the intelligent agent navigation model to prevent the navigation error of the intelligent agent navigation model from accumulating too much over time. is the loss function of the cross-modal contrastive learning module. Using the contrastive learning loss function, the cross-modal contrastive learning module can bring similar cross-modal information closer together in the latent space and distance dissimilar cross-modal information further apart in the latent space, enabling the agent navigation model to effectively encode both linguistic and visual information. The intelligent agent navigation model solving unit is used to solve the intelligent agent navigation model using the corresponding loss functions of the follower model unit, cross-modal contrastive learning module and visual variational autoencoder module.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the visual language navigation method based on the follower model according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the follower model-based visual language navigation method according to any one of claims 1 to 6 are implemented.

Images