A visual language navigation method based on adaptive prompt information during testing

Abstract

The application discloses a visual language navigation method based on adaptive prompt information during testing, comprising the following steps: obtaining natural language-based navigation instructions and 360-degree panoramic visual observation information of an intelligent agent; extracting low-frequency visual features of sub-images and selecting valuable information from an image memory bank for prompting; extracting a text feature vector corresponding to the navigation instructions and an initial state vector of the current environment; selecting historical information from the visual feature memory bank using the state vector for prompting; inputting the state vector and the instruction features into a visual-language interaction module to generate navigation action selection; updating the normalization layer in the full connection layer according to the prediction score of the navigation action selection; and repeating the above steps until the intelligent agent stops selecting or reaches the maximum number of movement steps. The application can effectively enhance the navigation performance of the pre-trained intelligent agent model in unknown test environments.

Description

Technical Field

[0001] This invention belongs to the field of visual language navigation, and specifically relates to a visual language navigation method based on adaptive prompting information during testing. Background Technology

[0002] Efficiently understanding and executing human instructions remains a significant challenge in the field of embodied intelligence. In recent years, Visual and Language Navigation (VLN) has become an important platform for measuring an agent's ability to follow instructions. In practical applications, a trained VLN agent needs to execute user instructions in real-time and online across diverse times and environments. However, to date, most existing VLN tasks have not employed an online setup. Typically, they follow a separate training-test paradigm, where the model is trained on a training set and then evaluated on a test set, without adaptive model updates during testing.

[0003] Due to the diversity of environments during online testing (e.g., different room layouts, object types and quantities), fixed pre-trained models inevitably encounter data distribution differences, posing a challenge to agent performance. For existing methods, directly updating the model in real-time through supervised learning is infeasible due to the lack of sufficient labeled information in online testing environments. Furthermore, other learning paradigms, such as unsupervised domain adaptation or semi-supervised learning, are also limited in the current context due to efficiency and user privacy concerns.

[0004] Test-time adaptation (TTA), a promising online model update method in recent years, adaptively adjusts the model using unlabeled test samples to adapt to test data with different distributions, thereby improving the model's generalization ability and practicality. However, most existing TTA methods fail to adequately balance model adaptability and stability, making them difficult to directly apply to VLN tasks. Furthermore, VLN models involve complex modules such as feature extraction, multimodal interaction and fusion, and inference prediction. Updating all model parameters during TTA not only increases time and computational costs but may also lead to problems such as accumulated errors and catastrophic forgetting, thus impairing model performance. Summary of the Invention

[0005] To address the aforementioned technical problems, it is necessary to introduce effective prompts to help the agent continuously accumulate experience during instruction execution, dynamically enhancing its understanding and execution capabilities. Therefore, this invention provides a visual language navigation method based on test-time adaptive prompts. By introducing a test-time adaptive approach into the visual language navigation prompt module, and updating a prompt memory containing a small number of parameters through test environment information, the agent can acquire valuable prompts to assist navigation prediction, thereby improving its navigation performance and generalization ability.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: A visual language navigation method based on test-time adaptive prompts includes the following steps: Step S1: Obtain navigation instructions based on natural language and 360-degree panoramic visual observation information of the intelligent agent; Step S2: Extract low-frequency visual features from the sub-images and select valuable information from the image memory bank to provide hints; Step S3: Extract the text feature vectors corresponding to the natural language navigation instructions and the initial state vector of the current environment; Step S4: Use the state vector to select historical information from the visual feature memory bank to provide prompts; Step S5: Input the state vector and command features into the visual-language interaction module to generate navigation action selection; Step S6: Update the normalization layer in the fully connected layer according to the predicted score selected by the navigation action; Step S7: Repeat steps S2 to S6 until the agent chooses to stop or reaches the maximum number of steps.

[0007] The beneficial effects of this invention are as follows: (1) The present invention effectively enhances the adaptability and generalization of the model in diverse testing environments.

[0008] (2) By introducing an adaptive prompt information memory, the present invention can dynamically acquire and utilize valuable information in the test environment, effectively improving the navigation accuracy and execution efficiency of the intelligent agent.

[0009] (3) This invention improves the model performance during online testing by combining multi-level optimization strategies to adaptively update the three levels of low-frequency information, visual features and model parameters of the image.

[0010] In summary, this invention utilizes the Test-Time Adaptive (TTA) method to enable pre-trained models to efficiently leverage historical memory and accumulated knowledge in diverse testing environments, thereby adapting to and executing user commands in real time. This invention addresses the performance degradation problem of fixed models in visual language navigation under diverse testing environments by using a method that combines test-time adaptive prompts with joint updates of some parameters. Attached Figure Description

[0011] Figure 1 This is a framework diagram of the visual language navigation method based on adaptive prompting information during testing according to the present invention. Detailed Implementation

[0012] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0013] like Figure 1 As shown, the visual-language navigation method based on adaptive prompts during testing proposed in this invention first dynamically enhances the image representation of the input panoramic image using an image prompt information memory bank; secondly, it improves the accuracy of environmental representation by adaptively fusing visual features; finally, it selectively updates the key parameters of the model to balance the model's adaptability and stability, thereby improving the agent's navigation performance and generalization ability in changing testing environments. Figure 1 As shown, the method achieves adaptive optimization during testing at three levels: input image, visual features, and model parameters. Specifically, the method includes the following steps: Step S1: Receive natural language navigation instructions containing a description of the navigation destination and visual observation information of the agent. The natural language navigation instructions refer to: the agent can observe the surrounding environment in 360 degrees, acquire a panoramic RGB image, and discretize the 360-degree panoramic RGB image into several segments in the horizontal and vertical directions (the number of segments is set to...). ), thus obtaining the corresponding N sub-images. , where i represents the image number.

[0014] Step S2, according to the above The process involves extracting corresponding low-frequency visual features from the sub-image, querying the image information memory using these features, adaptively selecting and extracting valuable low-frequency visual information as current image cues to enhance the sub-image, and updating the image information memory. This includes the following steps: Step S21: Transform each sub-image from the time domain to the frequency domain using Fourier transform. For a given sub-image... Its Fourier transform is expressed as:

[0015] in, It is a sub-image The Fourier transform result in the frequency domain These represent the image in the horizontal direction ( Direction) and vertical direction ( The number of pixels in the direction, i.e., the width and height of the image. These represent the spatial coordinates of the image in the horizontal direction. , These are frequency domain coordinates, representing the corresponding horizontal and vertical frequency components in the frequency domain.

[0016] Subsequently, a low-pass filter was applied to extract the low-frequency components. : ; in, This represents a low-pass filter, which preserves low-frequency components and attenuates high-frequency components in the frequency domain.

[0017] The above low-frequency characteristics It describes the overall structure and color distribution in the current subview image, which helps the agent to grasp the global spatial layout and approximate path direction of the navigation environment, so as to better identify the main areas, the general location of objects, and the three-dimensional spatial structure in the environment.

[0018] Step S22: Calculate the similarity score between the currently extracted low-frequency visual features and the features already stored in the image information memory. : ; in, The spectral representation of the k-th feature already stored in the image information memory. This represents the index of features in the image information memory.

[0019] Based on similarity ranking, the most similar pairs are selected and weighted for fusion. One low-frequency cue message is used to obtain the fused low-frequency cue message. : ; in, The weights are calculated based on the similarity scores. This indicates the first image cue retrieved from the memory. A low-frequency cue information vector.

[0020] Next, the low-frequency components of the original image are enhanced using low-frequency cue information, and then transformed back to the time domain using inverse Fourier transform to generate a low-frequency enhanced image. This allows the agent to utilize historical visual information to optimize its perception of the current environment. ; ; in, This represents the enhanced spectral representation. The inverse Fourier transform of the enhanced spectrum converts the frequency domain information back to the time domain. This indicates an assignment operation, where the enhanced image obtained from the inverse transformation above replaces the original sub-image.

[0021] Step S23: Based on the current sub-image The closest matching image cue found in the image cue memory. Low-frequency alerts are updated frequently to ensure the effectiveness and accuracy of image alerts. ; in, This represents the weighting coefficient for impulse updates, balancing new information with historical memory.

[0022] Step S3: Extract the text feature vectors corresponding to the natural language navigation instructions and the initial state vectors corresponding to the current environment. Extract scene feature vectors based on visual observations of candidate views, including the following steps: Step S31: Combine the classification identifier [CLS] and natural language navigation instructions. The segmentation identifier [SEP] is input to a pre-trained language encoder model (such as BERT) to extract text feature vectors. and initial state vector : ; in, This represents a sequence of text feature vectors generated by a pre-trained language encoder (such as BERT).

[0023] Step S32: For the current time... The panoramic view observed by the intelligent agent Input a pre-trained visual encoder model (such as ResNet) and combine it with corresponding location information features. Generate the current scene features : ; ; in, This represents the current panoramic view observation generated by a pre-trained visual encoder (e.g., ResNet). The visual feature vector.

[0024] Step S4: Use the currently extracted visual features The system adaptively queries the visual feature memory and extracts valuable historical visual information as cues to enhance the representation of the current state. Simultaneously, the system dynamically updates the visual feature vectors in the memory using an impulse update strategy to improve the model's adaptability to uncertain environments, including the following steps: Step S41: Query the visual feature memory for the most similar feature based on the cosine similarity score. Each visual feature is fused to obtain scene feature cues. Dynamically update scene features: ; ; ; in, Indicates the features of the current scene With the visual feature memory bank Features The cosine similarity score, Representing vectors norm ( Representing vectors The (elements) The weights are calculated based on the cosine similarity scores, and the weights in the visual feature memory are the th element. visual features Scene feature prompt information The size of the contribution, Indicates the features of the current scene and scene feature hints retrieved from the memory bank Perform a merge update.

[0025] Step S42: To ensure the dynamic adaptability of the visual feature memory, the impulse update strategy is also used to update the visual feature memory: ; in, The weighting coefficient for impulse updates controls the ratio of fusion between old and new information, enabling the memory to gradually accumulate new environmental information and maintain high adaptability to navigation tasks in constantly changing test environments.

[0026] Step S5: Convert the state vector from the previous time step. Instruction feature vector and scene feature vectors The input is fed into a pre-defined visual-language interaction module (such as the multimodal fusion model LXMERT) to generate a fused visual feature representation. and language feature representation Update the real-time state vector : ; in, They represent the current time. The initial visual feature representation and language feature representation.

[0027] Subsequently, a weighted attention weight between the real-time state vector and the scene feature vector is calculated based on a cross-modal attention mechanism. and According to the policy function Calculate the probability score of the next action The navigation action with the highest probability is the one selected. : ; ; ; in, To normalize each attention weight to an activation function that sums to 1 for a vector Each element in The calculation is as follows: . This indicates the number of heads in a multi-head attention mechanism. Representing the query matrix Dimension size, This is the policy function, used to calculate the probability score of the current agent's actions. This indicates the last layer in the LXMERT module. The head attention outputs a query matrix of the state vector. These represent the key matrices output by the scene feature vector and the instruction text feature vector, respectively. This represents the encoded historical observation information. This represents the parameters of the current agent model.

[0028] Then, visual features are represented. and language feature representation Perform cross-modal matching and match it with the real-time state vector. Fusion projection, combined with current navigation actions Generate a new final state vector : ; in, The function is used to update the agent's state vector. By combining visual features, linguistic features, and the current navigation action, a new final state vector is generated.

[0029] Step S6: Calculate the entropy based on the predicted score of the navigation action selection. Adaptively update the normalization layer (LN layer) in the fully connected layer module of the intelligent agent. ): ; in, Indicates the current moment The predicted first The probability of a navigation action.

[0030] ; in, Let each represent the affine parameter in the c-th LN layer. This represents the learning rate. The other two parameters in the LN layer (mean) and standard deviation Updates are performed during forward propagation. Represents entropy Regarding the affine parameters of the LN layer The gradient.

[0031] Step S7: Update the state vector Freezing instruction feature vector Scene feature vectors obtained from the new location Send the input to the preset visual-language interaction module and repeat steps S2-S6 until the agent chooses to stop moving or reaches the maximum number of movement steps.

[0032] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A visual language navigation method based on test-time adaptive prompting information, characterized in that, Includes the following steps: Step S1: Obtain navigation instructions based on natural language and 360-degree panoramic visual observation information of the intelligent agent; Step S2: Extract low-frequency visual features from the sub-images and select valuable information from the image memory bank for cues, including the following steps: Step S21: Use Fourier transform to convert each sub-image from the time domain to the frequency domain; For a given sub-image Its Fourier transform is expressed as: in, It is a sub-image The Fourier transform result in the frequency domain These respectively represent the image in the horizontal direction, i.e. Direction and perpendicular direction The number of pixels in each direction, i.e., the width and height of the image. These represent the spatial coordinates of the image in the horizontal direction. , is the frequency domain coordinate, representing the corresponding horizontal and vertical frequency components in the frequency domain, where j is the imaginary unit; Subsequently, a low-pass filter was applied to extract the low-frequency components. : ； in, A low-pass filter preserves low-frequency components and attenuates high-frequency components in the frequency domain. Step S22: Calculate the similarity score between the currently extracted low-frequency visual features and the features already stored in the image information memory. : ； in, The spectral representation of the nth feature already stored in the image information memory; Based on similarity ranking, the most similar pairs are selected and weighted for fusion. One low-frequency cue message is used to obtain the fused low-frequency cue message. : ； in, The weights are calculated based on the similarity scores. This indicates the first image cue retrieved from the memory. A low-frequency cue information vector; The low-frequency components of the original image are enhanced using low-frequency cue information. This image is then transformed back to the time domain using inverse Fourier transform to generate a low-frequency enhanced image. This allows the agent to utilize historical visual information to optimize its perception of the current environment. ； ； in, This represents the enhanced spectral representation. The inverse Fourier transform of the enhanced spectrum converts the frequency domain information back to the time domain. This indicates an assignment operation, that is, replacing the original sub-image with the enhanced image obtained from the inverse Fourier transform described above; Step S23: Based on the current sub-image The closest matching image cue found in the image cue memory. Low-frequency alerts are updated frequently to ensure the effectiveness and accuracy of image alerts. ； in, This represents the weighting coefficient for impulse updates, balancing new information with historical memory; Step S3: Extract the text feature vectors corresponding to the natural language navigation instructions and the initial state vector of the current environment; Step S4: Use the state vector to select historical information from the visual feature memory bank to provide prompts; Step S5: Input the state vector and command features into the visual-language interaction module to generate navigation action selection; Step S6: Update the normalization layer in the fully connected layer according to the predicted score selected by the navigation action; Step S7: Repeat steps S2 to S6 until the agent chooses to stop or reaches the maximum number of steps.

2. The visual language navigation method based on test-time adaptive prompting information according to claim 1, characterized in that, Step S3 includes the following steps: Step S31: Combine the classification identifier [CLS] and natural language navigation instructions. The segmentation identifier (SEP) is input to the pre-trained language encoder model to extract text feature vectors. and initial state vector : ； in, This represents a sequence of text feature vectors generated by a pre-trained language encoder; Step S32: For the current time... The panoramic view observed by the intelligent agent Input a pre-trained visual encoder model and combine it with corresponding positional information features. Generate the current scene features : ； ； in, This represents the current panoramic view observation generated by the pre-trained visual encoder. The visual feature vector.

3. The visual language navigation method based on test-time adaptive prompting information according to claim 1, characterized in that, Step S4 includes the following steps: Step S41: Query the visual feature memory for the most similar feature based on the cosine similarity score. Each visual feature is fused to obtain scene feature cues. Dynamically update scene features: ； ； ； in, Indicates the features of the current scene With the visual feature memory bank Features The cosine similarity score, Representing vectors norm, The weights are calculated based on the cosine similarity scores, and the weights in the visual feature memory are the th element. visual features Scene feature prompt information The size of the contribution, Indicates the features of the current scene and scene feature hints retrieved from the memory bank Perform a merge update; Step S42: To ensure the dynamic adaptability of the visual feature memory, the impulse update strategy is also used to update the visual feature memory: ； in, The weighting coefficient for impulse updates controls the ratio of fusion between old and new information, enabling the memory to gradually accumulate new environmental information and maintain high adaptability to navigation tasks in constantly changing test environments.

4. The visual language navigation method based on test-time adaptive prompting information according to claim 1, characterized in that, Step S5 includes: converting the previous time-state vector... Instruction feature vector and scene feature vectors Input is fed into a preset visual-language interaction module to generate a fused visual feature representation. and language feature representation Update the real-time state vector : ； in, They represent the current time. Initial visual feature representation and linguistic feature representation; The weighted attention weight between the real-time state vector and the scene feature vector is calculated based on a cross-modal attention mechanism. and According to the policy function Calculate the probability score of the next action The navigation action with the highest probability is the one selected. : ； ； ； in, For activation function, This indicates the number of heads in a multi-head attention mechanism. Representing the query matrix Dimension size, This is the policy function, used to calculate the probability score of the current agent's actions; This indicates the last layer in the LXMERT module. The head attention outputs a query matrix of the state vector. These represent the key matrices output by the scene feature vector and the instruction text feature vector, respectively. This represents the encoded historical observation information. Represents the parameters of the current agent model; Representing visual features and language feature representation Perform cross-modal matching and match it with the real-time state vector. Fusion projection, combined with current navigation actions Generate a new final state vector : ； in, The function is used to update the agent's state vector. By combining visual features, linguistic features, and the current navigation action, a new final state vector is generated.

5. A visual language navigation method based on test-time adaptive prompting information according to claim 1, characterized in that, Step S6 includes: calculating entropy based on the prediction score of navigation action selection. Adaptively update the normalized LN layer in the fully connected layer module of the intelligent agent: ； in, Indicates the current moment The predicted first The probability of a navigation action; ； in, They represent the times at time 1 and 2 respectively. Affine parameters in the c-th LN layer Represents the learning rate; the mean in the LN layer. and standard deviation Updates are performed during forward propagation. Represents entropy Regarding the affine parameters of the LN layer The gradient.

6. The visual language navigation method based on test-time adaptive prompting information according to claim 1, characterized in that, Step S7 includes: updating the state vector Freezing instruction feature vector Scene feature vectors obtained from the new location Send the input to the preset visual-language interaction module and repeat steps S2-S6 until the agent chooses to stop moving or reaches the maximum number of movement steps.

Images