Visual language navigation method and application based on cross-modal semantic alignment pre-training

Abstract

This invention discloses a visual language navigation method and application based on cross-modal semantic alignment pre-training. The method comprises: 1. pre-training an intelligent agent by masking and reconstructing a random segment of instruction information; 2. pre-training the intelligent agent by extracting discriminative phrases from the instruction information and aligning them with trajectory segments; 3. pre-training the intelligent agent using two proposed pre-training tasks and fine-tuning the overall network structure using imitation learning and reinforcement learning to obtain an optimal model. This method fully exploits the potential fine-grained semantic alignment relationships in trajectory-instruction pairs, enabling the intelligent agent to better complete navigation tasks.

Description

Technical Field

[0001] The present invention belongs to the field of multimodal signal processing, and in particular relates to image processing technology and natural language processing technology. It proposes a pre-training method based on cross-modal semantic alignment for visual language navigation tasks and its application. Background Art

[0002] Vision-language navigation is an emerging research field. Its goal is to build an intelligent agent that can communicate with humans using natural language and navigate in real 3D environments. This task can benefit many downstream applications, such as smart house cleaning, driver assistance systems, and intelligent patrols.

[0003] Current methods for visual language navigation tasks can be roughly divided into the following two categories: 1. Methods based on recurrent neural networks. This type of method combines an LSTM network with an attention mechanism. The LSTM network compresses and encodes the locations that the agent has visited, and the attention mechanism combines instruction information and compressed historical information to make decisions. This type of method encodes historical information into a fixed-length vector, resulting in the loss of a lot of important historical information, which is not conducive to the agent's judgment of its current state. 2. Transformer-structured navigation methods (HAMT). This method explicitly records every location that the agent has visited and uses the Transformer structure to encode historical information. This type of method uses a pre-training-fine-tuning training paradigm to train the network. By designing different proxy tasks to pre-train the network, the agent can better model the dependencies between different modal information.

[0004] Although these methods have achieved very advanced results, the potential fine-grained cross-modal semantic alignment relationship in trajectory-command pairs has not been systematically explored, which makes the intelligent agent unable to accurately judge its current state during navigation, making the navigation process not very interpretable. Summary of the Invention

[0005] In order to address the shortcomings of the above-mentioned existing technologies, the present invention proposes a visual language navigation method and application based on cross-modal semantic alignment pre-training, in order to explore the potential fine-grained cross-modal semantic alignment relationship in trajectory-command pairs, improve the accuracy of navigation, and enable the intelligent agent to better complete the navigation task.

[0006] In order to solve the above technical problems, the present invention adopts the following technical solutions:

[0007] The visual language navigation method based on cross-modal semantic alignment pre-training is characterized by being applied to the navigation task of an intelligent agent in a three-dimensional environment and is performed in the following steps:

[0008] Step 1: The agent obtains a complete navigation trajectory τ in the environment E and the instruction information X corresponding to the navigation trajectory τ. The instruction information X and the navigation trajectory τ form a positive trajectory-instruction pair (X, τ); where the instruction information X = (x1, x2, ..., x l ,…,x L ), x l Indicates the lth word in the instruction information X, L is the length of the instruction information; navigation trajectory is constructed from the panoramic view obtained by the agent at each moment; where T is the length of the complete trajectory; τ t is the trajectory segment at time t, is the real action at time t, O t is the set of observed features at time t, and O t ={O t,1 ,O t,2 ,…,O t,k ,…,O t,K ,O stop}, Q t,k represents the observation features of the kth viewpoint in the panorama obtained by the agent at time t, and O t,k =[v t,k ; r t,k ];v t,k represents the visual features of the kth viewpoint in the panorama obtained by the agent at time t; r t,k represents the spatial features of the location of the kth viewpoint in the panoramic image obtained by the agent at time t; O stop is the end mark;

[0009] Step 2: The agent obtains a complete navigation trajectory τ in the environment E' E' and the navigation trajectory τ E' Corresponding instruction information X E' ; Among them, the instruction information X E' It is independent of the navigation trajectory τ and is recorded as X neg ; By command information X neg and the navigation trajectory τ form a negative trajectory-command pair (X neg ,τ);

[0010] Step 3: Build a masked speech fragment reconstruction model consisting of an instruction encoder, a trajectory encoder, a cross-modal encoder, and a decoder to perform the pre-training task of trajectory-based masked speech fragment reconstruction;

[0011] Step 3.1: Select the continuous language segment from position u to position v in the instruction information X, and mask the continuous language segment with the flag [MASK] to obtain the masked continuous language segment x u:v And the masked instruction information X \u:v , where 0 <u<v<L;

[0012] Step 3.2: The masked instruction information X \u:v Input into the instruction encoder for encoding to obtain the encoded mask instruction information X' \u:v ; Input the navigation trajectory τ into the trajectory encoder for encoding to obtain the encoded trajectory information τ';

[0013] Step 3.3: Place X' \u:v and v' are input into the cross-modal encoder to obtain the mask instruction information after cross-modal encoding using τ' and using X' \u:v Trajectory information after cross-modal encoding

[0014] Step 3.4: and The input is processed into the decoder, and the output generates a continuous language segment x u:v probability;

[0015] Step 3.5: Use formula (1) to construct the loss function L of the masked language fragment reconstruction model TMFM :

[0016] L TMFM = -logP(x u: v|X \u:v ,τ) (1)

[0017] In formula (1), P(x u:v |X \u:v ,τ) represents the instruction information X after the given mask \u:v and navigation trajectory τ, the masked language segment reconstruction model generates a continuous language segment x u:v probability;

[0018] Step 4: Build a contrastive semantic alignment model consisting of a phrase extractor and a contrastive semantic aligner and perform pre-training;

[0019] Step 4.1: Let the command information in the positive trajectory-command pair (X, τ) be recorded as X pos ; and X pos Input into the instruction encoder to obtain the encoded instruction information X pos ' and word attention matrix A pos ∈RL×L ;

[0020] The phrase extractor converts the encoded instruction information X pos' Defined as the positive phrase embedding matrix And calculate the word attention matrix A pos The outer product Q pos =A pos (A pos ) T ;in, represents the lth phrase in the positive phrase embedding matrix; X pos The element in row i and column j of Represents the i-th phrase and the jth phrase similarity between

[0021] if and Then from the positive phrase embedding matrix P pos Delete the i-th phrase The deleted positive phrase embedding matrix is ​​recorded as Among them, M is the number of positive phrases in the positive phrase embedding matrix after deletion, and ∈ is a fixed threshold value; Represents the i-th phrase and the first phrase The similarity between Represents the jth phrase and the first phrase similarity between represents the mth positive phrase in the positive phrase embedding matrix after deletion;

[0022] Step 4.2: Follow the process in step 4.1 to check X neg Process and get the deleted negative phrase embedding matrix in, represents the zth negative phrase in the negative phrase embedding matrix after deletion, and Z represents the number of negative phrases in the negative phrase embedding matrix after deletion;

[0023] Step 4.3: The contrastive semantic aligner uses formula (2) to calculate and track segments Positive correlation scores between

[0024]

[0025] In formula (2), W s , H τ , b sare parameters to be learned, σ is the activation function;

[0026] Step 4.4: The contrastive semantic aligner uses formula (3) to calculate and the negative correlation score between the trajectory segment τ′

[0027]

[0028] Step 4.5: The contrastive semantic aligner uses the softmax function to compare the positive correlation coefficients and negative correlation coefficient Perform normalization operation to obtain the normalized positive correlation coefficient and negative correlation coefficient

[0029] Step 4.6: The contrastive semantic aligner uses formula (4) to construct the loss function L of the contrastive semantic alignment model CSMA :

[0030]

[0031] Step 5: According to the process of steps 2 and 3, a batch of B trajectory-instruction pairs are trained in the input mask language fragment reconstruction model and the contrastive semantic alignment model, and the loss functions of equations (1) and (4) are calculated to update the model parameters until the maximum number of iterations is reached or the loss function converges, thereby obtaining the pre-trained overall model, which is used to enable the intelligent agent to generate a complete navigation trajectory when performing the navigation task under the given environment E and instruction information X.

[0032] The electronic device of the present invention includes a memory and a processor, and is characterized in that the memory is used to store a program that supports the processor to execute the visual language navigation method, and the processor is configured to execute the program stored in the memory.

[0033] The present invention provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps of the visual language navigation method are executed.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. The cross-modal semantic alignment pre-training task proposed in this invention can effectively overcome the problem of important historical information loss in existing recurrent neural network-based methods, and can also overcome the problem of insufficient mining and utilization of cross-modal semantic alignment relationships in existing Transformer-based methods. The intelligent agent trained through the two proposed pre-training tasks can model the fine-grained alignment relationships in language and visual information, allowing the intelligent agent to understand its current state during navigation, thereby improving the interpretability of the intelligent agent's navigation process.

[0036] 2. This paper proposes a trajectory-based pre-training task for reconstructing masked speech segments. Specifically, a segment within a complete instruction is masked, and the agent performs cross-modal alignment of the masked instruction and visual information to identify relevant visual information that can be used to reconstruct the masked speech segment. This pre-training task effectively encourages the agent to discover alignment relationships between transmodal state information, thereby improving the agent's navigation success rate.

[0037] 3. This paper proposes a contrastive semantic alignment pre-training task. Specifically, this method extracts several discriminative phrases from command information and aligns these phrases with trajectory segments. Compared to existing methods, this pre-training task fully exploits fine-grained semantic alignment relationships within trajectory-command pairs, further improving the success rate of intelligent navigation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is a flow chart of the trajectory-based masked language fragment reconstruction method proposed in the present invention;

[0039] Figure 2 A schematic diagram of the cross-modal encoder structure used in the present invention;

[0040] Figure 3 This is a schematic diagram of the structure of the contrastive semantic alignment model proposed in the present invention;

[0041] Figure 4 The figure shows the performance comparison between the present invention and the existing method on the R2R dataset;

[0042] Figure 5 The figure shows the performance comparison results of the present invention and the existing method on the R4R dataset. DETAILED DESCRIPTION

[0043] In this embodiment, a visual language navigation method based on cross-modal semantic alignment pre-training is based on the newly proposed HAMT method (Transformer architecture). It combines the advantages of large-scale pre-training methods in visual language understanding tasks and proposes a pre-training task of trajectory-based masked language fragment reconstruction and a contrastive semantic alignment pre-training task, thereby fully utilizing the potential semantic alignment relationship in trajectory-command pairs to achieve better navigation accuracy. Specifically, it includes the following steps:

[0044] Step 1: The agent obtains a complete navigation trajectory τ in the environment E and the instruction information X corresponding to the navigation trajectory. The instruction information X and the navigation trajectory τ form a positive trajectory-instruction pair (X, τ); where the instruction information X = (x1, x2, ..., x l ,…,x L ), x l Indicates the lth word in the instruction information X, L is the length of the instruction information; navigation trajectory is constructed from the panoramic view obtained by the agent at each moment; where T is the length of the complete trajectory; τ t is the trajectory segment at time t, which is composed of the observation feature set O at time t t ={O t,1 ,O t,2 ,…,O t,k ,…,O t,K ,O stop} and the real action at time t Composition; among them, O t,k represents the observation features of the kth viewpoint in the panorama obtained by the agent at time t, and O t,k =[v t，k ; r t，k ];v t，k represents the visual features of the kth viewpoint in the panorama obtained by the agent at time t; r t，k represents the spatial features of the location of the kth viewpoint in the panoramic image obtained by the agent at time t; O stop is the end tag.

[0045] Step 2: The agent obtains a complete navigation trajectory τ in the environment E′ E′ and the instruction information X corresponding to the navigation trajectory E′ . Command information X E′ It is completely independent of the navigation trajectory τ, and is recorded as X neg By command information X neg and the navigation trajectory τ form a negative trajectory-command pair (X neg , τ).

[0046] Step 3: Build a masked speech fragment reconstruction model consisting of an instruction encoder, a trajectory encoder, a cross-modal encoder, and a decoder to perform the pre-training task of trajectory-based masked speech fragment reconstruction; Figure 1 An overall flowchart of the trajectory-based masked language fragment reconstruction method is provided.

[0047] Step 3.1: Select the continuous language segment from position u to position v in the instruction information X, mask the continuous language segment with the flag [MASK], and obtain the masked continuous language segment x u：v And the masked instruction information X \u：v , where 0<u<v<L;

[0048] Step 3.2: The masked instruction information X \u：v Input into the instruction encoder for encoding to obtain the encoded mask instruction information X′ \u：v ; Input the navigation trajectory τ into the trajectory encoder for encoding to obtain the encoded trajectory information τ′; wherein, the instruction encoder adopts the BERT model. The trajectory encoder first uses formula (1) to transform the observation feature set O t The element O in t，k Encode observation features:

[0049]

[0050] In formula (1), W v and W r are the parameters to be learned, Indicates the type of the k-th viewpoint at time t, Used to characterize the observed feature encoding, LayerNorm is the layer normalization operation; O′ t，k Represents the observation feature of the k-th viewpoint at time t after the observation feature encoding. t，k After being input into the two-layer Transformer model, the panoramic image feature O′ is obtained through average pooling operation. t .

[0051] The trajectory encoder uses formula (2) to calculate each element τ in the navigation trajectory τ t Encode trajectory features:

[0052]

[0053] In formula (2), W τ and W a are learnable parameters, represents the position code at time t, It is used to characterize trajectory coding, LayerNorm is the layer normalization operation; the trajectory information after trajectory coding is recorded as τ′={τ′1,...,τ′ t ,...,τ′ T}.

[0054] Step 3.3: X′ \u：v and τ′ are input into the cross-modal encoder to obtain the mask instruction information after cross-modal encoding using τ′ and using X′ \u：v Trajectory information after cross-modal encoding The cross-modal encoder uses Figure 2 LXMERT model shown.

[0055] Step 3.4: and The input is processed into the decoder and the output generates a continuous language segment x u：v The decoder adopts the Transformer structure.

[0056] Step 3.5: Use formula (3) to construct the loss function L of the masked language fragment reconstruction model TMFM :

[0057] L TMFM = -log P(x u：v |X \u：v , τ) (3)

[0058] In formula (3), P(x u：v |X \u：v ,τ) represents the instruction information X after the given mask \u：v and navigation trajectory τ, the masked language segment reconstruction model generates a continuous language segment x u：v probability;

[0059] Step 4: Construct a contrastive semantic alignment model consisting of a phrase extractor and a contrastive semantic aligner and perform pre-training; the specific flow chart is as follows Figure 3 shown.

[0060] Step 4.1: Let the command information in the positive trajectory-command pair (X, τ) be recorded as X pos ; X pos Input into the instruction encoder to obtain the encoded instruction information X pos′ And the word attention matrix A pos ∈R L×L ;

[0061] The phrase extractor converts the encoded instruction information X pos′ Defined as the positive phrase embedding matrix And calculate the word attention matrix A pos The outer product Q pos =A pos (A pos ) T ;in, represents the lth phrase in the positive phrase embedding matrix; Q pos The element in row i and column j of Represents the i-th phrase and the jth phrase similarity between

[0062] if and Then from the positive phrase embedding matrix P pos Delete the i-th phrase The deleted positive phrase embedding matrix is ​​recorded as Among them, M is the number of positive phrases in the positive phrase embedding matrix after deletion, and ∈ is a fixed threshold value; Represents the i-th phrase and the first phrase The similarity between Represents the jth phrase and the first phrase The similarity between them. represents the mth positive phrase in the positive phrase embedding matrix after deletion.

[0063] Step 4.2: Follow the process in step 4.1 to check X neg Process and get the deleted negative phrase embedding matrix represents the zth negative phrase in the negative phrase embedding matrix after deletion, and Z represents the number of negative phrases in the negative phrase embedding matrix after deletion.

[0064] Step 4.3: Comparative semantic aligner uses formula (4) to calculate and track segments Positive correlation scores between

[0065]

[0066] In formula (4), W s , H τ , b s are parameters to be learned, σ is the activation function;

[0067] Step 4.4: Comparative semantic aligner uses formula (5) to calculate and the negative correlation score between the trajectory segment τ′

[0068]

[0069] Step 4.5: Comparative semantic aligner uses softmax function to compare positive correlation coefficients and negative correlation coefficient Perform normalization operation to obtain the normalized positive correlation coefficient and negative correlation coefficient

[0070] Step 4.6: The contrastive semantic aligner uses formula (6) to construct the loss function L of the contrastive semantic alignment model CSAM :

[0071]

[0072] Step 5: According to the process of steps 2 and 3, a batch of B trajectory-instruction pairs are input into the mask language fragment reconstruction model and the contrastive semantic alignment model for training, and the loss functions of formulas (3) and (6) are calculated to update the model parameters. At the same time, the masked language reconstruction method MLM (Masked Language Modeling), masked image region reconstruction method MRM (Masked Region Modeling), single action prediction / regression method SAP / R (Single Action Prediction / Regression), spatial relationship prediction method SPREL (Spatial Relationship Prediction), and instruction trajectory matching method ITM (Instruction Trajectory Matching) in the HAMT model are used to train the instruction encoder, trajectory encoder, and cross-modal encoder in the model until the maximum number of iterations is reached or the loss function converges, thereby obtaining the pre-trained total model;

[0073] Step 6: During the fine-tuning process of the overall model, the agent cannot obtain the complete positive trajectory-command pair (X, τ). The agent obtains the command information X in the environment E, the observation feature set O1 at time t = 1, and the set of locations that can be reached at time t = 1, which is recorded as the action space Among them, a 1，n represents the nth location that the agent can go to at time t = 1, and N1 represents the number of locations that the agent can go to at time t = 1. The agent selects an action from the action space A1 and transfers to time t = 2, and obtains the observation feature set O2 at time t = 2 and the set of locations that can be visited at time t = 2, which is recorded as action space A2. At time t, the set of all locations that the agent can go to is recorded as action space Among them, N tDescribes the number of locations the agent can go to at time t.

[0074] Use formula (7) to construct the loss function L of the total model after pre-training FT , and then use the gradient descent method to fine-tune the pre-trained total model to update the model parameters, and finally obtain the optimal model for performing the navigation task:

[0075]

[0076] In formula (7), λ IL is the weighting coefficient, is the actual location of the agent at time t, D t is the advantage function in the A3C algorithm at time t. Indicates the agent's choice of action The probability of Indicates the real action selected by the agent at time t probability.

[0077] In order to quantitatively evaluate the effect of the present invention and verify its effectiveness, the method of the present invention is compared with the existing algorithms on the R2R dataset and the R4R dataset. Figure 4 As shown in the figure, the method of the present invention has a significant improvement in navigation success rate compared with the existing method based on recurrent neural network; compared with the navigation method based on Transformer structure The navigation success rate (SR) on the unseen validation set increased by 1.6%, and the path length weighted success rate (SPL) on the test set increased by 1.8%. Figure 5 As shown, the present invention is compared with the navigation method based on Transformer structure There was a 2.1% improvement in navigation success rate.

[0078] In this embodiment, an electronic device includes a memory and a processor. The memory is used to store a program that supports the processor to execute the navigation method. The processor is configured to execute the program stored in the memory.

[0079] In this embodiment, a computer-readable storage medium stores a computer program on the computer-readable storage medium. When the computer program is executed by a processor, the steps of the navigation method are executed.

Claims

1. A visual-language navigation method based on cross-modal semantic alignment pre-training is characterized by being applied to the navigation task of an intelligent agent in a three-dimensional environment and proceeding as follows: Step 1: The agent obtains the environment A complete navigation track and the navigation trajectory Corresponding instruction information , by the instruction information and navigation trajectory Constructing a positive trajectory-command pair ;in, Command information , Indicates command information The lth word in is the length of the instruction information; navigation trajectory It is constructed by the panoramic view obtained by the agent at each moment; is the length of the complete trajectory; is the trajectory segment at time t, is the real action at time t, is the set of observed features at time t, and , represents the observation features of the kth viewpoint in the panorama obtained by the agent at time t, and ; represents the visual features of the kth viewpoint in the panoramic image obtained by the agent at time t; Represents the spatial features of the location of the k-th viewpoint in the panoramic image obtained by the agent at time t; is the end mark; Step 2: The agent obtains the environment A complete navigation track and the navigation trajectory Corresponding instruction information ; Among them, the instruction information and navigation trajectory Not relevant, and recorded as ; By command information and navigation trajectory Constructing a negative trajectory-command pair ; Step 3: Build a masked speech fragment reconstruction model consisting of an instruction encoder, a trajectory encoder, a cross-modal encoder, and a decoder to perform the pre-training task of trajectory-based masked speech fragment reconstruction; Step 3.1: From the instruction information Select from the location To location Continuous language segments, and the continuous language segments are masked with the flag [MASK] to obtain the masked continuous language segments And the instruction information after masking ,in, ; Step 3.2: Mask the instruction information Input into the instruction encoder for encoding to obtain the encoded mask instruction information ; Navigation track Input into the trajectory encoder for encoding to obtain the encoded trajectory information ; Step 3.3: and Input into the cross-modal encoder and obtain the Mask instruction information after cross-modal encoding and use Trajectory information after cross-modal encoding ; Step 3.4: and The input is processed into the decoder and the output generates a continuous language segment probability; Step 3.5: Use formula (1) to construct the loss function of the masked language fragment reconstruction model : (1) In formula (1), Indicates the instruction information after the given mask and navigation trajectory In the case of , the masked language segment reconstruction model generates continuous language segments probability; Step 4: Build a contrastive semantic alignment model consisting of a phrase extractor and a contrastive semantic aligner and perform pre-training; Step 4.1: Set the positive trajectory-command pair The instruction information in is recorded as ; and Input into the instruction encoder to obtain the encoded instruction information and word attention matrix ; The phrase extractor converts the encoded instruction information Defined as the positive phrase embedding matrix And calculate the word attention matrix The outer product ;in, represents the first a phrase; The element in row i and column j of Represents the i-th phrase and the jth phrase similarity between if and , then from the positive phrase embedding matrix Delete the i-th phrase , so the deleted positive phrase embedding matrix is ​​recorded as ,in, is the number of positive phrases in the positive phrase embedding matrix after deletion, is a fixed threshold value; Represents the i-th phrase Hedi phrases The similarity between Represents the jth phrase Hedi phrases similarity between represents the mth positive phrase in the positive phrase embedding matrix after deletion; Step 4.2: Follow the process in step 4.1 to Process and get the deleted negative phrase embedding matrix ;in, represents the zth negative phrase in the negative phrase embedding matrix after deletion, and Z represents the number of negative phrases in the negative phrase embedding matrix after deletion; Step 4.3: The contrastive semantic aligner uses formula (2) to calculate and track segments Positive correlation scores between : (2) In formula (2), , are parameters to be learned, is the activation function; Step 4.4: The contrastive semantic aligner uses formula (3) to calculate and track segments Negative correlation scores between : (3) Step 4.5: The contrastive semantic aligner uses the softmax function to compare the positive correlation coefficients and negative correlation coefficient Perform normalization operation to obtain the normalized positive correlation coefficient and negative correlation coefficient ; Step 4.6: The contrastive semantic aligner uses formula (4) to construct the loss function of the contrastive semantic alignment model : (4) Step 5: According to the process of step 2 and step 3, a batch of B trajectory-instruction pairs are trained in the input mask language fragment reconstruction model and the contrast semantic alignment model, and the loss function of formula (1) and formula (4) is calculated to update the model parameters until the maximum number of iterations is reached or the loss function converges, thereby obtaining the pre-trained total model to realize the intelligent agent in a given environment. and instruction information Generate a complete navigation trajectory when performing navigation tasks in the case of 2. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store a program that supports the processor to execute the visual language navigation method according to claim 1, and the processor is configured to execute the program stored in the memory.

3. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the visual language navigation method according to claim 1 are executed.

Images