Variable environment visual language navigation method and system based on historical reflection

Abstract

The invention provides a variable environment visual language navigation method and system based on historical reflection, and relates to the technical field of computer vision, natural language processing and robot navigation. The navigation history information and the corresponding code embedding instruction used in the scene graph code embedding and next action are obtained; a gating network and a reflection network are utilized to analyze navigation historical information used in scene graph code embedding and next action and a corresponding code embedding instruction, and a correction instruction is obtained; the reflection network is a large language model used for generating a correction instruction; and based on the correction instruction, performing calculation by utilizing a graph perception self-attention mechanism, performing path navigation on the intelligent agent, obtaining a visual language navigation result, and completing visual language navigation in the variable environment. According to the method, the problems of poor adaptability and low fault-tolerant capability of visual language navigation are solved.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision, natural language processing and robot navigation technology, and in particular to a variable environment visual language navigation method and system based on historical reflection. Background Art

[0002] The Vision-and-Language Navigation (VLN) task aims to simulate human navigation behavior in a real environment. Its core goal is to enable an intelligent agent, that is, an entity that can autonomously perceive the environment and take actions to achieve a certain goal, to receive and understand natural language instructions, combine visual observations, and generate a reasonable sequence of action decisions to complete navigation tasks in an unknown environment. For example, the agent may receive an instruction: "Go downstairs, walk to the dining table, turn left to the kitchen, and stop in front of the refrigerator." Then, according to the content of the instruction, it needs to gradually complete the navigation path through visual perception and environmental interaction, and finally reach the target location. Due to the low cost, high controllability and scalability of the simulated environment to a variety of scenarios, many studies on vision-and-language navigation tasks rely on experiments based on simulated environments. However, migrating research results in simulated environments to real-world scenarios with higher complexity and uncertainty has always been one of the research focuses and technical difficulties in this field.

[0003] Existing visual language navigation technology frameworks are usually based on the "perfect instruction hypothesis", which assumes that natural language instructions can be perfectly aligned with the navigation path in the physical environment. This assumption assumes that the provider of the instructions has a full understanding of the navigation scenario and can generate clear and unambiguous instructions. However, in practical applications, this assumption is often not true because the instructions generated by the user are affected by a variety of subjective and objective factors and may not fully reflect the actual situation of the navigation environment. This deviation may cause the agent to face significant challenges when performing navigation tasks.

[0004] Existing research usually ignores the possible imperfect matching between natural language instructions and the physical environment, which leads to insufficient robustness of the agent in the real environment. How to design a more adaptive and fault-tolerant visual language navigation method in the case of imperfect instructions, combining the dynamic characteristics of the environment and the subjective bias of the instruction generator, is one of the core issues that need to be solved urgently. Summary of the invention

[0005] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method and system for variable environment visual language navigation based on historical reflection, which solves the problems of poor adaptability and low fault tolerance of visual language navigation.

[0006] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a variable environment visual language navigation method based on historical reflection, comprising: S1: Use the large model visual encoder and query transformer to process the variable environment visual observation data to obtain the scene graph encoding embedding, the navigation history information used in the next action, and the corresponding encoding embedding instructions; S2: Analyze the scene graph encoding embedding, the navigation history information used in the next action, and the corresponding encoding embedding instruction using a gating network and a reflection network to obtain a correction instruction; wherein the reflection network is a large language model for generating the correction instruction; S3: Based on the correction instructions, the graph-aware self-attention mechanism is used for calculation, path navigation is performed on the agent, visual language navigation results are obtained, and visual language navigation in a variable environment is completed.

[0007] Furthermore, the S1 includes: Using a frozen large model visual encoder, the variable environment visual observation data is processed to obtain image features; wherein the frozen large model visual encoder is a large model visual encoder with fixed weights in a lightweight manner; Using the query transformer, the learnable query embedding is fused with the instruction embedding through a self-attention mechanism to obtain the instruction information; Based on the cross-attention mechanism, the image features and the instruction information are fused to obtain the scene graph encoding embedding, the navigation history information used in the next action, and the corresponding encoding embedding instruction.

[0008] Furthermore, the expression of the scene graph encoding embedding is: ; in, represents the scene graph encoding embedding, represents the cross attention mechanism, Represents image features, Indicates command information.

[0009] Further, the S2 includes: splicing the scene graph code embedding, the navigation history information used in the next action and the corresponding code embedding instruction into sequence data, inputting the sequence data into a gated network, and obtaining a gated network output result; Input the output result of the gating network into the binary classification fully connected layer, and obtain the error label value through calculation; When the error label value is greater than the threshold, the reflection network is used to identify the specific error type, and the correction instructions are generated through the large language model autoregression.

[0010] Furthermore, the expression of the error label value is: ; ; in, Indicates the error label value, express The standard deviation of represents a binary classification fully connected layer, represents the output result of the gating network, represents the gating network, represents the scene graph encoding embedding, Indicates the encoding embedded instruction, Represents historical navigation information.

[0011] Furthermore, the expression of the error type is: ; ; in, Indicates the error type. represents the maximum value selection function, express function, The first hyperparameter representing the error type identification, represents the function value calculated based on the large language model, The second hyperparameter representing the error type identification, represents a large language model, represents the scene graph encoding embedding, Indicates the encoding embedded instruction, Represents historical navigation information; The expression of the modified instruction is: ; in, Indicates a correction instruction. represents a large language model, represents the function value calculated based on the large language model, Indicates the error type.

[0012] Further, the S3 includes: Based on the action encoding sequence, the path points and visual features are unified to obtain node embedding; Embedding the correction instruction and the node to perform cross-modal cross attention calculation to obtain a cross-modal attention result; Use variable environment visual observation data to determine the connectivity between nodes and build a dynamic topology map; Combining the cross-modal attention result with the dynamic topological graph, performing a graph-aware self-attention mechanism, and obtaining a graph-aware result; The graph perception result is analyzed by using a feedforward neural network, and the distance and difficulty of reaching the node are comprehensively considered to obtain an action score; wherein the action score is used to reflect the efficiency of the intelligent agent in reaching the node; Based on the action score, the node with the highest score is selected, and the optimal path is navigated to the node to obtain a visual language navigation result, thereby completing the visual language navigation in a variable environment.

[0013] Furthermore, the expression of the highest score node is: ; ; ; ; ; ; ; ; in, represents the highest scoring node, represents the maximum value selection function, represents the action score, represents a feed-forward neural network, Represents the graph perception result, represents the graph-aware self-attention mechanism, Represents the topology parameters, represents the learnable transformation matrix of the query, represents the learnable transformation matrix of the key, represents a constant feature dimension, Represents additional graph structure information, A learnable transformation matrix representing the value, represents the pairwise distance matrix between nodes obtained from the edge information of the graph, represents the learnable weight matrix, represents the learnable bias vector, represents the cross-modal attention result, Represents a dynamic topology graph, represents the cross attention mechanism, represents node embedding, Indicates the coded correction instruction, Represents the navigable points after redundancy removal, represents the edges that can be traversed, represents average pooling, represents the action encoding sequence, represents the direction embedding, Represents step embedding.

[0014] A changeable environment visual language navigation system based on historical reflection, including: The visual language understanding module is used to process the variable environment visual observation data using the large model visual encoder and query transformer to obtain the scene graph encoding embedding; A visual language reflection module, for determining whether to call the reflection network as part of the modular network using the gated network, and when the reflection network needs to be called, the reflection network analyzes the historical navigation information and the scene graph encoding embedding to obtain a correction instruction; The action prediction module is used to perform calculations based on the correction instructions using the graph-aware self-attention mechanism to navigate the path of the intelligent agent, obtain visual language navigation results, and complete visual language navigation in a variable environment.

[0015] The beneficial effects of the present invention are as follows: by introducing the historical reflection mechanism, the intelligent agent can use the historical information accumulated during the navigation process to deeply understand the environmental status and user intentions, and dynamically adjust the parsing of natural language instructions to bridge the gap between instructions and actual scenes, thereby improving the robustness and adaptability of the navigation system in real scenes. (1) The gated network and the reflection network in the visual language reflection module dynamically detect and correct environmental changes and deviations in human instructions, such as the addition of obstacles, the removal of landmarks, the modification of textures or colors, and the replacement of descriptors. Combined with the historical reflection mechanism, the intelligent agent can dynamically adjust the path planning and instruction parsing during the navigation process to bridge the gap between instructions and the environment, thereby greatly improving the accuracy and adaptability of the navigation system in complex scenes. (2) Using pre-trained large models (such as BLIP, BLIP2, InstructBLIP, LLaVA, etc.) and query transformers to perform deep alignment of vision and language, the intelligent agent can generate high-quality navigation instructions in a dynamically changing environment. The historical reflection module conducts in-depth analysis of historical information and combines it with the dynamic update of the graph network to ensure that instruction parsing will not degrade due to environmental changes or increased path length, thereby enhancing the stability and robustness of instruction parsing. (3) By generating correction suggestions and combining multimodal features, new navigation strategies are dynamically generated, enabling the intelligent agent to flexibly respond to diverse navigation environments. By training on large-scale data collected in different environments and using efficient fine-tuning methods such as LoRA, the navigation model of the present invention exhibits strong generalization capabilities in different scenarios. At the same time, combined with the dynamic maintenance mechanism of the graph network of the action prediction module, the intelligent agent can plan the path more efficiently and avoid redundant actions and repeated exploration, greatly improving navigation efficiency. (4) The graph-aware self-attention mechanism is combined with the transmembrane state attention results and the dynamic topological map to analyze the score of each node, making the selection of the optimal path more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein: Figure 1 It is a schematic diagram of an application scenario of a variable environment visual language navigation system based on historical reflection according to some embodiments of this specification; Figure 2 It is an exemplary flow chart of a variable environment visual language navigation method based on historical reflection according to some embodiments of this specification. DETAILED DESCRIPTION

[0017] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0018] Embodiment 1 Figure 1 It is a module diagram of a variable environment visual language navigation system based on historical reflection according to some embodiments of this specification.

[0019] In some embodiments, the variable environment visual language navigation system based on historical reflection may include a visual language understanding module, a visual language reflection module and an action prediction module.

[0020] The visual language understanding module is used to process the variable environment visual observation data using the large model visual encoder and query transformer to obtain the scene graph encoding embedding.

[0021] The visual language understanding module mainly consists of a query transformer and a large model visual encoder. The query transformer encodes each view into a fixed-length visual tag. For the candidate image obtained by the agent, the visual encoder is used to extract features. These visual features are then processed by a cross-attention mechanism with a learnable query embedding and self-attention learning with the text embedding of the instruction to obtain instruction-based image queries. After linear projection, these queries are input into the large model as image tags for processing. To cope with imperfect instructions, the language input in this method includes not only instructions but also new path planning information provided by the reflection module.

[0022] The visual language reflection module is used to use the gated network to determine whether to call the reflection network as part of the modular network. When it is necessary to call, the reflection network analyzes the historical navigation information and the scene graph encoding embedding to obtain correction instructions.

[0023] The visual-linguistic reflection module consists of a gating network and a history reflection network.

[0024] The gating network is specifically designed to determine whether the current historical information contains errors, such as: obstacles added, landmarks removed, texture and color modified, descriptor replaced, etc. If any of these errors are detected, the agent will dynamically activate the history reflection module to generate navigation strategies for these error conditions. In order to ensure efficient navigation, instead of directly using the visual language model to identify potential visual-language mismatches, the gating network classifies visual and language information based on the Transformer architecture, and the episodes marked as mismatched by the gating network are passed to the visual-language reflection module. In this module, the model identifies the error type, generates potential solutions, and formulates instructions through a structured process.

[0025] The reflection network is used to generate navigation strategies after the gating network confirms an error. In this case, the reflection network analyzes the current error based on historical information and proposes a correction suggestion and corresponding verification criteria. This suggestion is then used to generate new navigation instructions, replacing the original instructions in the multimodal alignment stage. At the same time, in the decision stage, the new instructions are added to the prompt so that the decision model can perform multi-step reasoning. After the decision model is labeled, if the network's new suggestion is verified and it is determined that the comment associated with the suggestion is inconsistent with the actual navigation history, a new suggestion-comment pair is generated and the above process is repeated. If the two are consistent, the navigation process ends.

[0026] The action prediction module is used to perform calculations based on the correction instructions using the graph-aware self-attention mechanism to navigate the path of the intelligent agent, obtain visual language navigation results, and complete visual language navigation in a variable environment.

[0027] A graph-based representation method is designed in the action prediction module to leverage the understanding and reasoning capabilities of the large language model, ensure the effectiveness of the semantic mapping from visual language input to action, and enhance the model's understanding and representation capabilities of natural language. The topology graph is dynamically maintained and used as a memory mechanism to track the navigation path. The next step is selected from the entire constructed topology graph. When imperfections in the instructions are confirmed in the historical path, the visual markers obtained through visual language understanding, as well as the new instructions and corresponding strategies derived from the visual language reflection module are input into the system prompts, which can effectively plan and reverse track unvisited nodes.

[0028] In some embodiments, the action prediction module can use supervised learning or reinforcement learning, and can be completed in steps: first train the action prediction to obtain path planning capabilities, and then introduce the gating and reflection network to achieve error correction and re-planning. It is also possible to combine multi-module losses with back propagation to achieve end-to-end optimization.

[0029] The graph network consists of visited nodes and adjacent unexplored nodes in the path. All candidate visual observations of each visited node are represented by average pooling, while each incompletely explored node is represented by partial pooling of the corresponding visual observations of all its adjacent visited nodes. Each view is represented by the sum of the following three parts: 1. Visual features, representing the visual information of the node; 2. Direction embedding, representing the location of the node; 3. Step embedding, representing the traversal order of the agent in the current task; the step embedding of the unexplored node is set to 0, indicating that the "stop node" of the stop action will be added to the graph network.

[0030] At each time step, the navigation graph consists of a set of nodes and a set of edges. The node embeddings are passed into a multi-layer cross-modal Transformer to model the relationship between instructions and nodes. Specifically, the node embeddings are first cross-modally aligned with the instructions and then further processed by a graph-aware self-attention module. The graph-aware self-attention module combines the distance and visual similarity between nodes to enhance contextual understanding.

[0031] In some embodiments, a variable environment visual language navigation system based on historical reflection can be used to execute a variable environment visual language navigation method based on historical reflection, including: S1: using a large model visual encoder and a query transformer to process the variable environment visual observation data to obtain a scene graph coding embedding; S2: using a gated network to analyze historical navigation information, coding embedding instructions and the scene graph coding embedding to obtain correction instructions; S3: based on the correction instructions, performing path navigation on the intelligent agent to obtain visual language navigation results, and completing the visual language navigation of the variable environment.

[0032] In some embodiments of the present specification, the processor uses a variable environment visual language navigation system based on historical reflection to perform a variable environment visual language navigation method based on historical reflection. In this way, the difference between instructions and actual scenes can be bridged, and the robustness and adaptability of the navigation system in real scenes can be improved. (1) The gated network and the reflection network in the visual language reflection module dynamically detect and correct environmental changes and deviations in human instructions, such as the addition of obstacles, the removal of landmarks, the modification of textures or colors, and the replacement of descriptors. Combined with the historical reflection mechanism, the intelligent agent can dynamically adjust the path planning and instruction parsing during the navigation process, bridge the gap between instructions and the environment, and thus greatly improve the accuracy and adaptability of the navigation system in complex scenes. (2) Using pre-trained large models (such as BLIP, BLIP2, InstructBLIP, LLaVA, etc.) and query transformers to deeply align vision and language, the intelligent agent can generate high-quality navigation instructions in a dynamically changing environment. The historical reflection module ensures that instruction parsing will not degrade due to environmental changes or increased path length through in-depth analysis of historical information and dynamic updates of the graph network, thereby enhancing the stability and robustness of instruction parsing. (3) By generating correction suggestions and combining multimodal features, new navigation strategies are dynamically generated, enabling the intelligent agent to flexibly respond to diverse navigation environments. By training on large-scale data collected in different environments and using efficient fine-tuning methods such as LoRA, the navigation model of the present invention exhibits strong generalization capabilities in different scenarios. At the same time, combined with the dynamic maintenance mechanism of the graph network of the action prediction module, the intelligent agent can plan the path more efficiently and avoid redundant actions and repeated exploration, greatly improving navigation efficiency.

[0033] Embodiment 2 Figure 2 is an exemplary flow chart of a variable environment visual language navigation method based on historical reflection according to some embodiments of this specification. Figure 2 As shown, the process includes the following steps. In some embodiments, the process can be executed by a processor.

[0034] S1: Use the large model visual encoder and query transformer to process the variable environment visual observation data to obtain the scene graph encoding embedding, the navigation history information used in the next action, and the corresponding encoding embedding instructions.

[0035] The large model visual encoder is an encoder for extracting image features. For example, the large model visual encoder may include BLIP, BLIP2, InstructBLIP, LLaVA, etc.

[0036] Scene graph encoding embeddings are embedding vectors used to align visual features with language features.

[0037] In some embodiments, the processor can implement S1 based on the following steps: using a frozen large model visual encoder to process the variable environment visual observation data to obtain image features; using a query transformer to fuse the learnable query embedding with the instruction embedding through a self-attention mechanism to obtain instruction information; based on a cross-attention mechanism, the image features and the instruction information are fused to obtain a scene graph encoding embedding, the navigation history information used in the next action, and the corresponding encoding embedded instructions.

[0038] A frozen large model visual encoder is a large model visual encoder with fixed weights.

[0039] In some embodiments, the processor can use lightweight methods such as full freezing, partial freezing or LoRA to adapt the large model visual encoder to the navigation and reflection tasks to obtain a frozen large model visual encoder.

[0040] The variable environment visual observation data is the image data acquired by the agent's vision.

[0041] Image features are features that reflect the variable environment visual observation data and are related to the environment.

[0042] In some embodiments, the processor may use the frozen core weights of the large model visual encoder to train and fine-tune the query transformer to obtain a trained query transformer.

[0043] The query transformer can be a Q-Former (Querying Transformer) model.

[0044] In some embodiments, the processor may input variable environment visual observation data into a frozen large model visual encoder for processing to obtain image features.

[0045] Learnable query embedding and instruction embedding are descriptions of path points and information about action sequences.

[0046] Instruction features are features that comprehensively describe the movement of the agent.

[0047] In some embodiments, the processor may utilize a query transformer to fuse 32 fixed-length learnable query embeddings with instruction embeddings using a self-attention mechanism to obtain instruction features.

[0048] In some embodiments, the expression embedded in the scene graph encoding may be: ; in, represents the scene graph encoding embedding, represents the cross attention mechanism, Represents image features, Indicates command information.

[0049] S2: Analyze the scene graph code embedding, the navigation history information used in the next action, and the corresponding code embedding instruction by using the gated network and the reflection network to obtain a correction instruction.

[0050] The gating network is a lightweight neural network used to correct errors in the encoding embedded instructions. For example, the gating network can include a lightweight Transformer.

[0051] Historical navigation information includes visual observations and instructions of the past several steps, the sequence of visited nodes, etc.

[0052] Encoded embedded instructions are instructions in the history that do not match the visual observations and navigation map.

[0053] Correction commands are command parameters that match visual observations and navigation maps.

[0054] In some embodiments, the processor can implement S2 based on the following steps: historical navigation information, encoding embedding instructions and the scene graph encoding embedding are spliced ​​into sequence data, input into the gated network, and obtain the gated network output result; the gated network output result is input into the binary classification fully connected layer, and the error label value is obtained through calculation; when the error label value is greater than the threshold, the reflection network is used to identify the specific error type, and the correction instruction is generated through large language model autoregression.

[0055] The sequence data is the concatenation of historical navigation information, coded embedded instructions and scene graph coded embedded data.

[0056] The error label value is a numerical value that reflects whether there is a mismatch problem in the output result of the gating network.

[0057] In some embodiments, the expression of the error label value may be: ; ; in, Indicates the error label value, express The standard deviation of represents a binary classification fully connected layer, represents the output result of the gating network, represents the gating network, represents the scene graph encoding embedding, Indicates the encoding embedded instruction, Represents historical navigation information.

[0058] In some embodiments, when the error label value is greater than a threshold, it is determined that there is a mismatch problem, and the reflection network needs to be called to identify the specific error type and generate correction instructions, otherwise the original instructions continue to be executed.

[0059] The reflective network can be a deep neural network.

[0060] In some embodiments, the gating network can be trained with a binary or multi-class cross entropy loss to determine whether there is a mismatch.

[0061] In some embodiments, the reflection network can generate correction instructions based on a large language model, which can be trained by autoregressive language modeling or reinforcement learning (such as RLHF).

[0062] In some embodiments, the expression of the error type may be: ; ; in, Indicates the error type. represents the maximum value selection function, express function, The first hyperparameter representing the error type identification, represents the function value calculated based on the large language model, The second hyperparameter representing the error type identification, represents a large language model, represents the scene graph encoding embedding, Indicates the encoding embedded instruction, Represents historical navigation information.

[0063] The first hyperparameter for error type identification and the second hyperparameter for error type identification are learnable function parameters.

[0064] In some embodiments, the expression of the correction instruction may be: ; in, Indicates a correction instruction. represents a large language model, represents the function value calculated based on the large language model, Indicates the error type.

[0065] In some embodiments, the preset prompt word form can be: "Your task is to cut and extract the declarative text into action descriptors and landmarks. Please note that some phrases are combined with words indicating directions to better display spatial location information, and each phrase must contain at least one action. Do not rephrase. The representation of this text is "%s". Please return only your answer separated by "|". Each sentence can be disassembled more than once. " In the reflection part, after the large language model reflection function receives the call signal of the gating network, it combines the instruction information and image features , make targeted modifications. For example: "Given that the current path is blocked, we propose the following diversion plan: Backtrack: Backtrack from the corridor or any possible side passage in the room. Avoid the sofa: Move slightly to the left or right to avoid the sofa that blocks the way to the carpet. Revised steps: Backtrack from the corridor: Re-enter the bedroom and keep along the left route taken initially. Bypass the sofa: Lateral movement: Turn right at the corner of the living room and move laterally to bypass the sofa. Alternative route: If lateral movement is not possible, cross the dining area diagonally and approach the left side of the terrace. Criticism: The agent should expect to backtrack from the corridor first, then reach the living room by laterally bypassing the sofa or diagonally through the dining area, and finally stop near the carpet in front." S3: Based on the correction instructions, the graph-aware self-attention mechanism is used for calculation, path navigation is performed on the agent, visual language navigation results are obtained, and visual language navigation in a variable environment is completed.

[0066] The visual language navigation result is the result that reflects the final navigation path of the intelligent agent.

[0067] In some embodiments, the processor can implement S3 based on the following steps: based on the action coding sequence, unify the path points and visual features to obtain node embedding; perform cross-modal cross-attention calculations on the correction instructions and the node embedding to obtain cross-modal attention results; use variable environment visual observation data to determine the connectivity between nodes and construct a dynamic topological map; combine the cross-modal attention results with the dynamic topological map to perform a graph perception self-attention mechanism to obtain a graph perception result; use a feedforward neural network to analyze the graph perception results, comprehensively consider the distance and difficulty to reach the node, and obtain an action score; based on the action score, select the highest scoring node, navigate to the node along the optimal path, obtain a visual language navigation result, and complete the visual language navigation of the variable environment.

[0068] The action coding sequence is a sequence that reflects the actions of the entire navigation process of the intelligent agent.

[0069] In some embodiments, the processor may obtain an action code sequence based on the action instruction by removing redundancy and performing encoding processing.

[0070] Node embeddings are the embedded representations of unvisited nodes.

[0071] In some embodiments, the processor may combine average pooling, direction embedding, and step embedding for unvisited nodes to construct node embedding.

[0072] The cross-modal attention result is the attention result of the comprehensive encoded modified instructions and node embeddings.

[0073] In some embodiments, the processor may encode the correction instruction to obtain an encoded correction instruction.

[0074] A dynamic topology map is a topology map used to determine the connectivity between nodes.

[0075] In some embodiments, the processor may utilize variable environment visual observation data, analyze the relationships between various nodes, and construct a dynamic topology map.

[0076] The graph perception result is the result reflecting the distance and visual similarity between nodes.

[0077] The action score is data that scores each node in the graph perception result based on the distance and difficulty of the agent reaching the node; the higher the action score, the more efficient the agent is in reaching the node.

[0078] The highest scoring node is the node with the highest current score.

[0079] In some embodiments, the processor can select the node with the highest score, navigate to the node along the shortest or optimal path, select environmental nodes in combination with the graph network structure and perform navigation actions, and compare and verify the historical path with the correction suggestions. If there is still a mismatch, the reflection process is repeated until the intelligent agent successfully reaches the target node or confirms that there is no further feasible path, thereby obtaining a visual language navigation result.

[0080] In some embodiments, the processor uses a two-layer feedforward network to process the output node representation to generate action scores. The agent selects the node with the highest score as the target and navigates to the selected node along the shortest path in the graph network. At the same time, the method masks the scores of visited nodes to encourage the agent to explore other nodes.

[0081] In some embodiments, the expression of the highest scoring node may be: ; ; ; ; ; ; ; ; in, represents the highest scoring node, represents the maximum value selection function, represents the action score, represents a feed-forward neural network, Represents the graph perception result, represents the graph-aware self-attention mechanism, Represents the topology parameters, represents the learnable transformation matrix of the query, represents the learnable transformation matrix of the key, represents a constant feature dimension, Represents additional graph structure information, A learnable transformation matrix representing the value, represents the pairwise distance matrix between nodes obtained from the edge information of the graph, represents the learnable weight matrix, represents the learnable bias vector, represents the cross-modal attention result, Represents a dynamic topology graph, represents the cross attention mechanism, represents node embedding, Indicates the coded correction instruction, Represents the navigable points after redundancy removal, represents the edges that can be traversed, represents average pooling, represents the action encoding sequence, represents the direction embedding, Represents step embedding.

[0082] In some embodiments of this specification, by introducing a historical reflection mechanism, the intelligent agent can use the historical information accumulated during the navigation process to deeply understand the environmental state and user intentions, and dynamically adjust the parsing of natural language instructions to bridge the gap between instructions and actual scenes, thereby improving the robustness and adaptability of the navigation system in real scenes. (1) Dynamically detect and correct environmental changes and deviations in human instructions through the gated network and reflection network in the visual language reflection module, such as the addition of obstacles, removal of landmarks, modification of textures or colors, and replacement of descriptors. Combined with the historical reflection mechanism, the intelligent agent can dynamically adjust path planning and instruction parsing during navigation, bridge the gap between instructions and the environment, thereby greatly improving the accuracy and adaptability of the navigation system in complex scenes. (2) Use pre-trained large models (such as BLIP, BLIP2, InstructBLIP, LLaVA, etc.) and query transformers to perform deep alignment of vision and language, so that the intelligent agent can generate high-quality navigation instructions in a dynamically changing environment. The historical reflection module conducts in-depth analysis of historical information and combines it with the dynamic update of the graph network to ensure that instruction parsing will not degrade due to environmental changes or increased path length, thereby enhancing the stability and robustness of instruction parsing. (3) By generating correction suggestions and combining multimodal features, new navigation strategies are dynamically generated, enabling the intelligent agent to flexibly respond to diverse navigation environments. By training on large-scale data collected in different environments and using efficient fine-tuning methods such as LoRA, the navigation model of the present invention exhibits strong generalization capabilities in different scenarios. At the same time, combined with the dynamic maintenance mechanism of the graph network of the action prediction module, the intelligent agent can plan the path more efficiently and avoid redundant actions and repeated exploration, greatly improving navigation efficiency. (4) The graph-aware self-attention mechanism is combined with the transmembrane state attention results and the dynamic topological map to analyze the score of each node, making the selection of the optimal path more accurate.

Claims

1. A variable environment visual language navigation method based on historical reflection, characterized in that: include: S1: Use the large model visual encoder and query transformer to process the variable environment visual observation data to obtain the scene graph encoding embedding, the navigation history information used in the next action, and the corresponding encoding embedding instructions; S2: using a gating network and a reflection network, historically reflecting on the scene graph encoding embedding, the navigation history information used in the next action, and the corresponding encoding embedding instruction to obtain a correction instruction; wherein the reflection network is a large language model for generating the correction instruction; S3: Based on the correction instructions, the graph-aware self-attention mechanism is used for calculation, path navigation is performed on the agent, visual language navigation results are obtained, and visual language navigation in a variable environment is completed.

2. The variable environment visual language navigation method based on historical reflection according to claim 1 is characterized in that: The S1 includes: Using a frozen large model visual encoder, the variable environment visual observation data is processed to obtain image features; wherein the frozen large model visual encoder is a large model visual encoder with fixed weights in a lightweight manner; Using the query transformer, the learnable query embedding is fused with the instruction embedding through a self-attention mechanism to obtain the instruction information; Based on the cross-attention mechanism, the image features and the instruction information are fused to obtain the scene graph encoding embedding, the navigation history information used in the next action, and the corresponding encoding embedding instruction.

3. The variable environment visual language navigation method based on historical reflection according to claim 2 is characterized in that: The expression for embedding the scene graph encoding is: ； in, represents the scene graph encoding embedding, represents the cross attention mechanism, Represents image features, Indicates command information.

4. The variable environment visual language navigation method based on historical reflection according to claim 1 is characterized in that: The S2 includes: splicing the scene graph code embedding, the navigation history information used in the next action and the corresponding code embedding instruction into sequence data, inputting the sequence data into a gated network, and obtaining a gated network output result; Input the output result of the gating network into the binary classification fully connected layer, and obtain the error label value through calculation; When the error label value is greater than the threshold, the reflection network is used to identify the specific error type, and the correction instructions are generated through the large language model autoregression.

5. The variable environment visual language navigation method based on historical reflection according to claim 4 is characterized in that: The expression of the error label value is: ； ； in, Indicates the error label value, express The standard deviation of represents a binary classification fully connected layer, represents the output result of the gating network, represents the gating network, represents the scene graph encoding embedding, Indicates the encoding embedded instruction, Represents historical navigation information.

6. The variable environment visual language navigation method based on historical reflection according to claim 4 is characterized in that: The expression of the error type is: ； ； in, Indicates the error type. represents the maximum value selection function, express function, The first hyperparameter representing the error type identification, represents the function value calculated based on the large language model, The second hyperparameter representing the error type identification, represents a large language model, represents the scene graph encoding embedding, Indicates the encoding embedded instruction, Represents historical navigation information; The expression of the modified instruction is: ； in, Indicates a correction instruction. represents a large language model, represents the function value calculated based on the large language model, Indicates the error type.

7. The variable environment visual language navigation method based on historical reflection according to claim 1 is characterized in that: The S3 includes: Based on the action encoding sequence, the path points and visual features are unified to obtain node embedding; Embedding the correction instruction and the node to perform cross-modal cross attention calculation to obtain a cross-modal attention result; Use variable environment visual observation data to determine the connectivity between nodes and build a dynamic topology map; Combining the cross-modal attention result with the dynamic topological graph, and calculating using a graph-aware self-attention mechanism to obtain a graph-aware result; The graph perception result is analyzed by using a feedforward neural network, and the distance and difficulty of reaching the node are comprehensively considered to obtain an action score; wherein the action score is used to reflect the efficiency of the intelligent agent in reaching the node; Based on the action score, the node with the highest score is selected, and the optimal path is navigated to the node to obtain a visual language navigation result, thereby completing the visual language navigation in a variable environment.

8. The variable environment visual language navigation method based on historical reflection according to claim 7 is characterized in that: The expression of the highest score node is: ； ； ； ； ； ； ； ； in, represents the highest scoring node, represents the maximum value selection function, represents the action score, represents a feed-forward neural network, Represents the graph perception result, represents the graph-aware self-attention mechanism, Represents the topology parameters, represents the learnable transformation matrix of the query, represents the learnable transformation matrix of the key, represents a constant feature dimension, Represents additional graph structure information, A learnable transformation matrix representing the value, represents the pairwise distance matrix between nodes obtained from the edge information of the graph, represents the learnable weight matrix, represents the learnable bias vector, represents the cross-modal attention result, Represents a dynamic topology graph, represents the cross attention mechanism, represents node embedding, Indicates the coded correction instruction, Represents the navigable points after redundancy removal, represents the edges that can be traversed, represents average pooling, represents the action encoding sequence, represents the direction embedding, Represents step embedding.

9. A variable environment visual language navigation system based on historical reflection, used to execute the variable environment visual language navigation method based on historical reflection as claimed in any one of claims 1 to 8, characterized in that: include: The visual language understanding module is used to process the variable environment visual observation data using the large model visual encoder and query transformer to obtain the scene graph encoding embedding; A visual language reflection module, for determining whether to call the reflection network as part of the modular network using the gated network, and when the reflection network needs to be called, the reflection network analyzes the historical navigation information and the scene graph encoding embedding to obtain a correction instruction; The action prediction module is used to perform calculations based on the correction instructions using the graph-aware self-attention mechanism to navigate the path of the intelligent agent, obtain visual language navigation results, and complete visual language navigation in a variable environment.

Images