Visual language navigation method based on self-adaptive prompt information during testing

By introducing the method of combining the adaptive prompt information and partial parameters during testing in visual language navigation, the performance degradation of fixed models in diversified environments is solved, and efficient navigation and generalization of agents in diversified environments is achieved. Improve capability.

CN119048767AActive Publication Date: 2024-11-29INST OF AUTOMATION CHINESE ACAD OF SCI

Patent Information

Application Number
CN202411248639.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-06
Publication Date
2024-11-29
Estimated Expiration
2044-09-06

AI Technical Summary

Technical Problem

Existing visual language navigation technologies are difficult to maintain model performance in a diverse test environment, and fixed models are difficult to update in real time in the case of data distribution differences and lack of labeling information, resulting in performance degradation and cumulative errors.

Method used

Using the method based on adaptive prompt information during testing, the image and visual feature memory dynamically updates the image and visual feature memory through obtaining and utilizing natural language navigation instructions and 360-degree panoramic visual information, and adaptively updates the model parameters with multi-level optimization strategies to improve the model Adaptability and navigation performance in a diverse environment.

Benefits of technology

It effectively improves the navigation accuracy and execution efficiency of the agent in a diverse test environment, balances the adaptability and stability of the model, and improves the generalization ability and online testing performance of the model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048767A_ABST
    Figure CN119048767A_ABST
Patent Text Reader

Abstract

The invention discloses a visual language navigation method based on self-adaptive prompt information during testing. The visual language navigation method comprises the following steps: acquiring a navigation instruction based on a natural language and 360-degree panoramic visual observation information of an intelligent agent; low-frequency visual features of the sub-images are extracted, and valuable information is selected from the image memory bank for prompting; extracting a text feature vector corresponding to the navigation instruction and an initial state vector of the current environment; historical information is selected from the visual feature memory bank for prompting by using the state vector; inputting the state vector and the instruction feature into a vision-language interaction module to generate navigation action selection; updating a normalization layer in the full connection layer according to the prediction score selected by the navigation action; and repeating the above contents until the intelligent agent selects to stop or reaches the maximum movement step number limit. According to the method, the navigation performance of the pre-trained agent model in an unknown test environment can be effectively enhanced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention belongs to the field of visual language navigation, and in particular relates to a visual language navigation method based on adaptive prompt information during testing. Background Art

[0002] Efficiently understanding and executing human instructions remains a significant challenge in the field of embodied intelligence. In recent years, vision and language navigation (VLN) has become an important platform for measuring the ability of intelligent agents to follow instructions. In practical applications, trained VLN agents need to execute user instructions in real time and online in a variety of time and environments. However, to date, most existing VLN tasks do not adopt an online setting. Typically, they follow an independent training-testing paradigm, that is, the model is trained on the training set and then evaluated on the test set, without adaptive model updates during testing.

[0003] Due to the diversity of the environment during online testing (e.g., different room layouts, object types and numbers, etc.), fixed pre-trained models will inevitably encounter data distribution differences, which poses a challenge to the performance of the agent. For existing methods, it is not feasible to directly update the model in real time through supervised learning due to the lack of sufficient labeled information in the online testing environment. In addition, other learning paradigms such as unsupervised domain adaptation or semi-supervised learning are also limited in the current context due to execution efficiency and user privacy protection issues.

[0004] Test-time adaptation (TTA) is an online model updating method that has attracted much attention in recent years. It uses unlabeled test samples to adaptively adjust the model to adapt to test data of different distribution types, thereby improving the generalization ability and practicality of the model. However, most existing TTA methods fail to fully balance the adaptability and stability of the model and are difficult to directly apply to VLN tasks. In addition, the VLN model involves complex modules such as feature extraction, multimodal interaction and fusion, and inference and prediction. If all parameters of the model are updated during the TTA process, it will not only increase the time cost and computational cost, but may also lead to problems such as cumulative errors and catastrophic forgetting, thereby damaging the model performance. Summary of the invention

[0005] In order to solve the above technical problems, it is necessary to introduce effective prompt information to help the intelligent agent accumulate experience in the process of executing instructions and dynamically enhance its understanding and execution ability. Therefore, the present invention provides a visual language navigation method based on test-time adaptive prompt information, which introduces the test-time adaptive method into the prompt module of visual language navigation, and updates the prompt information memory containing a small number of parameters through the test environment information. The intelligent agent can obtain valuable prompt information to assist navigation prediction, thereby improving its navigation performance and generalization ability.

[0006] To achieve the above object, the present invention adopts the following technical solution: A visual language navigation method based on adaptive prompt information during testing, comprising the following steps: Step S1, obtaining natural language-based navigation instructions and 360-degree panoramic visual observation information of the agent; Step S2, extracting low-frequency visual features of the sub-image and selecting valuable information from the image memory library for prompting; Step S3, extracting a text feature vector corresponding to the natural language navigation instruction and an initial state vector of the current environment; Step S4, using the state vector to select historical information from the visual feature memory library for prompting; Step S5: input the state vector and instruction features into the vision-language interaction module to generate navigation action selection; Step S6: updating the normalization layer in the fully connected layer according to the prediction score of the navigation action selection; Step S7: Repeat steps S2 to S6 until the agent chooses to stop or reaches the maximum limit on the number of moving steps.

[0007] The beneficial effects of the present invention are: (1) This invention effectively enhances the adaptability and generalization of the model in diverse testing environments.

[0008] (2) By introducing an adaptive prompt information memory library, the present invention can dynamically acquire and utilize valuable information in the test environment, effectively improving the navigation accuracy and execution efficiency of the intelligent agent.

[0009] (3) The present invention combines a multi-level optimization strategy to adaptively update the three levels of image low-frequency information, visual features, and model parameters, further improving the model performance during online testing.

[0010] In summary, the present invention uses the test-time adaptation (TTA) method to enable the pre-trained model to efficiently use historical memory and accumulated knowledge in a diverse test environment to adapt to and execute user instructions in real time. The present invention uses a method of jointly updating test-time adaptive prompt information and partial parameters to solve the problem of performance degradation of fixed models in visual language navigation in a diverse test environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 It is a framework diagram of the visual language navigation method based on adaptive prompt information during testing of the present invention. DETAILED DESCRIPTION

[0012] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0013] like Figure 1 As shown in the figure, the visual language navigation method based on adaptive prompt information during testing proposed by the present invention first uses the image prompt information memory library to dynamically enhance the image representation of the input panorama; secondly, it improves the accuracy of the environment representation by adaptively fusing visual features; finally, it selectively updates the key parameters of the model to balance the adaptability and stability of the model, thereby improving the navigation performance and generalization ability of the intelligent agent in a changing test environment. Figure 1 As shown in FIG. 1 , the method realizes test-time adaptive optimization at three levels: input image, visual features, and model parameters. Specifically, the method comprises the following steps: Step S1: receiving a natural language navigation instruction containing a description of a navigation destination and visual observation information of an agent, wherein the natural language navigation instruction means that the agent can observe the surrounding environment 360 degrees, obtain a panoramic RGB image, and discretize the 360-degree panoramic RGB image into a number of segments in the horizontal and vertical directions (the number of segments is set to ), get the corresponding N sub-images , i represents the sequence number of the image.

[0014] Step S2: According to the above The method extracts the corresponding low-frequency visual features from the sub-image, and uses the current low-frequency visual features to query in the image information memory, adaptively selects and extracts valuable low-frequency visual information as the current image prompt to enhance the sub-image, and updates the image information memory, including the following steps: Step S21: Use Fourier transform to transform each sub-image from the time domain to the frequency domain. , its Fourier transform is expressed as:

[0015] in, It is a sub-image The Fourier transform result in the frequency domain is, Respectively represent the image in the horizontal direction ( direction) and vertical direction ( direction), i.e. the width and height of the image, Represents the spatial coordinates of the image in the horizontal direction , is the frequency domain coordinate, indicating the corresponding horizontal and vertical frequency domain components in the frequency domain.

[0016] Subsequently, a low-pass filter is applied to extract the low-frequency components : ; in, Represents a low-pass filter, which retains low-frequency components and attenuates high-frequency components in the frequency domain.

[0017] The above low frequency characteristics It describes the overall structure and color distribution in the current sub-view image, which helps the agent grasp the global spatial layout and general path direction of the navigation environment, so as to better identify the main areas in the environment, the approximate location of objects, and the three-dimensional spatial structure.

[0018] Step S22: Calculate the similarity score between the currently extracted low-frequency visual features and the features stored in the image information memory : ; in, represents the spectral representation of the kth feature stored in the image information memory, Represents the index of a feature in the image information memory.

[0019] According to the similarity ranking, select and weight the closest fusion low-frequency prompt information, and obtain the fused low-frequency prompt information : ; in, is the weight calculated based on the similarity score, Indicates the first A low-frequency cue information vector.

[0020] Next, the low-frequency components of the original image are enhanced using low-frequency cue information, which is converted back to the time domain via an inverse Fourier transform to generate a low-frequency enhanced image so that the agent can use historical visual information to optimize current environmental perception: ; ; in, represents the enhanced spectrum representation, represents the inverse Fourier transform of the enhanced spectrum, converting the frequency domain information back to the time domain, represents an assignment operation, i.e. replacing the original sub-image with the enhanced image obtained by the above inverse transformation.

[0021] Step S23: Based on the current sub-image , the closest query in the image prompt memory The low-frequency prompt information is updated with impulse to ensure the effectiveness and accuracy of the image prompt information: ; in, Represents the weight coefficient of the impulse update to balance new information and historical memory.

[0022] Step S3, extracting a text feature vector corresponding to the natural language navigation instruction and an initial state vector corresponding to the current environment, and extracting a scene feature vector based on visual observation of the candidate view, including the following steps: Step S31: Classification identifier [CLS], natural language navigation instruction and segmentation identifier [SEP] to extract text feature vectors from a pre-trained language encoder model (such as BERT) and the initial state vector : ; in, Represents a sequence of text feature vectors generated by a pre-trained language encoder (e.g. BERT).

[0023] Step S32: For the current moment , observe the panoramic view obtained by the agent Input the pre-trained visual encoder model (such as ResNet) and combine it with the corresponding position information features Generate the current scene features : ; ; in, Represents the current panoramic view observation generated by a pre-trained visual encoder (e.g., ResNet) The visual feature vector of .

[0024] Step S4: Use the currently extracted visual features Adaptively query the visual feature memory and extract valuable historical visual information as prompts to enhance the current state representation. At the same time, the system dynamically updates the visual feature vector in the memory through the impulse update strategy to improve the model's adaptability in uncertain environments, including the following steps: Step S41: Query the closest visual feature memory using the cosine similarity score visual features, and fuse them to obtain scene feature prompt information , dynamically update scene features: ; ; ; in, Indicates the current scene characteristics and the visual feature memory Features The cosine similarity score of Representation vector The norm of Representation vector No. elements), Represents the weight calculated based on the cosine similarity score, representing the first Visual features Prompt information in scene features The contribution size of Indicates the characteristics of the current scene and scene feature prompt information obtained from the memory bank Perform fusion update.

[0025] Step S42: To ensure the dynamic adaptability of the visual feature memory, the visual feature memory is updated using the same impulse update strategy: ; in, The weight coefficient representing the impulse update controls the ratio of the fusion of new and old information, so that the memory bank can gradually accumulate new environmental information and maintain high adaptability to navigation tasks in a constantly changing test environment.

[0026] Step S5: The state vector of the previous moment , instruction feature vector And the scene feature vector Input into the preset visual-language interaction module (such as the multimodal fusion model LXMERT) to generate the fused visual feature representation and language feature representation , update the real-time state vector : ; in, Respectively represent the current time Initial visual feature representation and language feature representation.

[0027] Then, the weighted attention weight between the real-time state vector and the scene feature vector is calculated based on the cross-modal attention mechanism. and , according to the policy function Calculate the probability score of the next action , the navigation action with the highest probability is selected : ; ; ; in, To normalize each attention weight into an activation function with a probability distribution whose sum is 1, for the vector Each element in , calculated as follows: . represents the number of heads in the multi-head attention mechanism, Represents the query matrix The dimension size of is the strategy function, which is used to calculate the action probability score of the current agent. Indicates the last layer in the LXMERT module The query matrix of the head attention output state vector, The key matrices representing the output of the scene feature vector and the instruction text feature vector respectively. represents the encoded historical observation information, Represents the parameters of the current agent model.

[0028] Then the visual features are represented and language feature representation Perform cross-modal matching and compare it with the real-time state vector Fusion projection, combined with current navigation actions , generating a new final state vector : ; in, Function is used to update the state vector of the agent , combining visual features, language features and current navigation actions to generate a new final state vector.

[0029] Step S6: Calculate entropy based on the predicted score of the navigation action selection , adaptively update the normalization layer (LN layer: ): ; in, Indicates at the current moment The predicted The probability of a navigation action.

[0030] ; in, denote the affine parameters in the cth LN layer, represents the learning rate. The other two parameters in the LN layer (mean and standard deviation ) is updated in the forward propagation. Representation Entropy About LN layer affine parameters gradient.

[0031] Step S7: Update the state vector , frozen instruction feature vector and the scene feature vector obtained at the new position Send it to the preset vision-language interaction module and repeat steps S2-S6 until the agent chooses to stop moving or reaches the maximum limit of moving steps.

[0032] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A visual language navigation method based on adaptive prompt information during testing, characterized in that: The following steps are involved: Step S1, obtaining natural language-based navigation instructions and 360-degree panoramic visual observation information of the agent; Step S2, extracting low-frequency visual features of the sub-image and selecting valuable information from the image memory library for prompting; Step S3, extracting a text feature vector corresponding to the natural language navigation instruction and an initial state vector of the current environment; Step S4, using the state vector to select historical information from the visual feature memory library for prompting; Step S5: input the state vector and instruction features into the vision-language interaction module to generate navigation action selection; Step S6: updating the normalization layer in the fully connected layer according to the prediction score of the navigation action selection; Step S7: Repeat steps S2 to S6 until the agent chooses to stop or reaches the maximum limit on the number of moving steps.

2. According to claim 1, a visual language navigation method based on adaptive prompt information during testing is characterized in that: The step S2 comprises the following steps: Step S21: convert each sub-image from the time domain to the frequency domain using Fourier transform; For a given sub-image , whose Fourier transform is expressed as: ; in, It is a sub-image The Fourier transform result in the frequency domain is, Respectively represent the image in the horizontal direction, that is direction and the vertical direction, i.e. The number of pixels in the direction, that is, the width and height of the image, Respectively represent the spatial coordinates of the image in the horizontal direction, , is the frequency domain coordinate, indicating the corresponding horizontal and vertical frequency domain components in the frequency domain, and j is an imaginary unit; Subsequently, a low-pass filter is applied to extract the low-frequency components : ; in, Represents a low-pass filter, which retains low-frequency components and attenuates high-frequency components in the frequency domain; Step S22: Calculate the similarity score between the currently extracted low-frequency visual features and the features stored in the image information memory : ; in, Represents the spectral representation of the kth feature stored in the image information memory; According to the similarity ranking, select and weight the closest fusion low-frequency prompt information, and obtain the fused low-frequency prompt information : ; in, is the weight calculated based on the similarity score, Indicates the first A low-frequency cue information vector; The low-frequency components of the original image are enhanced using low-frequency cue information, which is then converted back to the time domain via inverse Fourier transform to generate a low-frequency enhanced image so that the agent can use historical visual information to optimize current environmental perception: ; ; in, represents the enhanced spectrum representation, represents the inverse Fourier transform of the enhanced spectrum, converting the frequency domain information back to the time domain, represents the assignment operation, i.e. replacing the original sub-image with the enhanced image obtained by the above inverse transformation; Step S23: Based on the current sub-image , the closest query in the image prompt memory The low-frequency prompt information is updated with impulse to ensure the effectiveness and accuracy of the image prompt information: ; in, Represents the weight coefficient of the impulse update to balance new information and historical memory.

3. The visual language navigation method based on adaptive prompt information during testing according to claim 1, characterized in that: The step S3 comprises the following steps: Step S31: Classification identifier [CLS], natural language navigation instruction and segmentation identifier [SEP] input into the pre-trained language encoder model to extract text feature vectors and the initial state vector : ; in, represents a sequence of text feature vectors generated by a pre-trained language encoder; Step S32: For the current moment , observe the panoramic view obtained by the agent Input the pre-trained visual encoder model and combine it with the corresponding position information features Generate the current scene features : ; ; in, represents the current panoramic view observation generated by the pre-trained visual encoder The visual feature vector of .

4. The visual language navigation method based on adaptive prompt information during testing according to claim 1, characterized in that: The step S4 comprises the following steps: Step S41: Query the closest visual feature memory using the cosine similarity score visual features, and fuse them to obtain scene feature prompt information , dynamically update scene features: ; ; ; in, Indicates the current scene characteristics and the visual feature memory Features The cosine similarity score of Representation vector The norm of Represents the weight calculated based on the cosine similarity score, representing the first Visual features In the scene feature prompt information The contribution size of Indicates the characteristics of the current scene and scene feature prompt information obtained from the memory bank Perform integration updates; Step S42: To ensure the dynamic adaptability of the visual feature memory, the visual feature memory is updated using the same impulse update strategy: ; in, The weight coefficient representing the impulse update controls the ratio of the fusion of new and old information, so that the memory bank can gradually accumulate new environmental information and maintain high adaptability to navigation tasks in a constantly changing test environment.

5. The visual language navigation method based on adaptive prompt information during testing according to claim 1, characterized in that: The step S5 comprises: converting the state vector at the previous moment , instruction feature vector And the scene feature vector Input into the preset visual-language interaction module to generate the fused visual feature representation and language feature representation , update the real-time state vector : ; in, Respectively represent the current time Initial visual feature representation and language feature representation; Calculate the weighted attention weight between the real-time state vector and the scene feature vector based on the cross-modal attention mechanism and , according to the policy function Calculate the probability score of the next action , the navigation action with the highest probability is selected : ; ; ; in, is the activation function, represents the number of heads in the multi-head attention mechanism, Represents the query matrix The dimension size of is the strategy function, which is used to calculate the action probability score of the current agent; Indicates the last layer in the LXMERT module The query matrix of the head attention output state vector, The key matrices representing the output of the scene feature vector and the instruction text feature vector respectively; represents the encoded historical observation information, Represents the parameters of the current agent model; Representing visual features and language feature representation Perform cross-modal matching and compare it with the real-time state vector Fusion projection, combined with current navigation actions , generating a new final state vector : ; in, Function is used to update the state vector of the agent , combining visual features, language features and current navigation actions to generate a new final state vector.

6. The visual language navigation method based on adaptive prompt information during testing according to claim 1, characterized in that: The step S6 comprises: calculating entropy based on the prediction score of the navigation action selection , adaptively update the normalization layer LN layer in the fully connected layer module of the intelligent agent: ; in, Indicates at the current moment The predicted The probability of a navigation action; ; in, Respectively indicate at time The affine parameters in the cth LN layer, Represents the learning rate; the mean in the LN layer and standard deviation In the forward pass, the update is performed. Representation Entropy About LN layer affine parameters gradient.

7. The visual language navigation method based on adaptive prompt information during testing according to claim 1, characterized in that: The step S7 comprises: updating the updated state vector , frozen instruction feature vector and the scene feature vector obtained at the new position Send it to the preset vision-language interaction module and repeat steps S2-S6 until the agent chooses to stop moving or reaches the maximum limit of moving steps.

Citation Information

Patent Citations

  • Global positioning method of laser-navigated AGV (automatic guided vehicle)

    CN104729500A

  • Mobile robot vision-language navigation method based on multi-modal features

    CN115082915A

  • Visual language navigation method based on historical context information enhancement

    CN118010026A

  • Three-path Fourier-time domain modulation network framework for high-precision intra-operative navigation

    CN118570423A

  • System and method for edge-enhancement of digital images using wavelets

    US20100183222A1

Cited By

  • Visual language navigation method based on adaptive memory refinement and state action fine tuning

    CN122049631A

  • Navigation method and device

    WO2026175081A1