Continuous environment visual language navigation method based on global-local mamba causal world model

Abstract

The application discloses a continuous environment visual language navigation method based on a global-local Mamba causal world model, and belongs to the cross field of computer vision, robot autonomous navigation and artificial intelligence. Firstly, a global-local Mamba perceiver extracts local detail information and global trend information on different time scales. Secondly, the strict shunt of prior features and posterior features decouples the causal reasoning path based on historical information from the training calibration path based on current observation. Finally, in the structured latent space, a multi-step forward deduction is performed on the future action sequence, and the expected cumulative discounted return is taken as an optimization target to select the current optimal action output. The technical route enables the system to not only have the ability to respond after seeing the current scene, but also have the ability to imagine the future evolution according to the historical state and pre-avoid risks.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer vision, robot autonomous navigation and artificial intelligence, and specifically relates to a continuous environment visual language navigation method based on the global-local Mamba causal world model. Background Technology

[0002] In Visual Language Navigation-CE (VLN-CE) tasks in continuous environments, agents must perform low-level actions in a continuous physical space, facing imperfect actuations and complex visual dynamics. Current mainstream technologies mainly rely on passive visual and linguistic modality matching, typically using Transformer models to learn a direct response mapping from instruction-observation pairs to actions.

[0003] These existing technologies have the following drawbacks: A short-sighted operational perspective exists: existing navigation methods often simplify tasks into rigid, one-step conditioned reflex mappings of "seeing something - doing something." This approach ignores the continuity and causal relationships of actions in the physical world. This causes the agent to act as if it lacks predictive ability, unable to anticipate the potential physical consequences of its movements (such as whether it will collide or get stuck in a dead end). It can only passively correct its trajectory after making mistakes such as taking the wrong path or causing a collision, severely lacking the planning ability to proactively avoid potential risks.

[0004] Long-range dependency modeling is inefficient: When robots perform long-distance navigation in continuous environments, the resulting action sequences and visual images are extremely long, requiring them to effectively remember and understand past spatiotemporal information. However, existing latent world models primarily rely on traditional RNNs or Transformers as their infrastructure, which struggles to handle such long-range memories. RNNs inherently suffer from forgetfulness, losing crucial early memories as the route lengthens; while Transformers experience an extremely high quadratic increase in computational burden as the memory span increases, severely consuming system computing power. This makes it difficult for current technologies to efficiently capture and utilize long-distance route dependencies. Summary of the Invention

[0005] This invention provides a continuous environment visual language navigation method based on the global-local Mamba causal world model, which solves the problems of existing technologies ignoring the coherence and causal relationships of actions in the physical world, and the difficulty of existing technologies in efficiently capturing and utilizing long-distance route dependencies.

[0006] This invention is achieved through the following technical solution: A continuous environment visual-language navigation method based on a global-local Mamba causal world model, the method comprising the following steps: Step 1: Obtain task input and initialize runtime parameters; Step 2: Perform multimodal coding and composite feature construction based on the task in Step 1; Step 3: Construct a global-local Mamba perceptron and extract multi-scale spatiotemporal features from the composite features constructed in Step 2; Step 4: Construct a causal state inference network and perform structured latent state updates based on the multi-scale spatiotemporal features extracted in Step 3; Step 5: Based on the deterministic state in the state update in Step 4, perform auxiliary supervision and loss optimization during the offline training phase; Step 6: Based on the offline training in Step 5, perform online prospective planning and action output; Step 7: Based on the online forward planning in Step 6, perform closed-loop updates and terminate the task.

[0007] Furthermore, step 1 specifically involves the system receiving natural language instructions at the start of each navigation task. and initial observation ; at any time step The system receives the current RGB-D observations. Actions at the previous moment The random state at the previous time step and the deterministic state at the previous moment ; The current observation The system uses a multimodal input consisting of first-person view images and depth information, and the observation resolution can be set according to computing power conditions. The system adopts a waypoint-based high-level action space, where a policy network selects high-level navigation actions from a set of candidate directions and candidate distances, and then the low-level controller converts them into continuous control actions for execution. The task termination condition is: the output stops moving, or the maximum time step is reached. .

[0008] Furthermore, step 2 specifically involves, for the current observation Visual encoding is performed to obtain visual features. : (1) In the formula, Indicates a visual encoder; Indicates time step Visual feature representation; Then, the current visual features Actions at the previous moment and the random state at the previous moment Mapping to a unified feature space to construct a composite input embedding : (2) In the formula, For action embedding layer; For random state embedding layer; For feature fusion coding function; This is for composite input embedding.

[0009] Furthermore, step 3 specifically involves maintaining a short-term window sequence and a long-term difference sequence based on the composite input embedding obtained in step 2. The short-term window sequence is defined as: (3) Long-term difference sequences are defined as: (4) In the formula, Indicates the length of the local time window; Indicates the length of the global difference window; Local branching utilizes local Mamba operators to extract high-fidelity local dynamic features: (5) In the formula, Represents a locally selective state-space operator; Representation layer normalization; Represents posterior local features; The global branch uses the global Mamba operator to process long-term difference sequences and extracts macroscopic trend features through a nonlinear projection head: (6) In the formula, Represents a globally selective state-space operator; The projection head preferably includes a linear layer, a SiLU activation layer, and a layer normalization layer. Represents posterior global features; Local and global features are asymmetrically fused using a gated fusion module to obtain posterior features. : (7) In the formula, Indicates the gated fusion function; This represents a multilayer perceptron; Represents a linear mapping matrix; Represents the posterior features used for training calibration; Constructing prior features The prior features employ a network structure isomorphic to the posterior features, but their inputs must be time-shifted, using only the up-to-date data. Historical buffer sequence of moments and Perform the calculation: (8) In the formula, and These represent the global prior features and local prior features generated from the history window, respectively. It does not contain real observation information at the current moment, but only relies on historical information for subsequent state transitions and forward planning.

[0010] Furthermore, step 4 specifically involves using a cyclic state-space model to maintain the deterministic state. and random state The structured state of potential beliefs; The deterministic state update formula is: (9) In the formula, Indicates a gated loop unit; Indicates time step A deterministic state, used as long-term memory; During the training phase, posterior features are utilized. The random state is calibrated to construct the posterior distribution: (10) And the posterior random state is obtained using reparameterized sampling: (11) In the formula, and Let represent the mean and standard deviation of the posterior distribution, respectively; This represents element-wise multiplication; Indicates standard normal noise; In the prospective reasoning phase, based on the deterministic state... The random state at the previous time step Actions at the previous moment and prior features Predicting the prior distribution of a random state; First, construct the intermediate features: (12) The residual offset parameters are then output by the dynamic prediction network: (13) Finally, the prior values ​​of the random state are obtained: (14) In the formula, and These represent the mean and standard deviation of the residual shift relative to the random state at the previous time step, respectively.

[0011] Furthermore, step 5 specifically involves, during the offline training phase, based on the posterior state... and deterministic state We construct an observation reconstruction head, a reward prediction head, and a continuation probability prediction head, which are used to constrain latent state learning, respectively. The observation reconstruction head is defined as: (15) The reward prediction head is defined as: (16) The continuation probability prediction head is defined as: (17) In the formula, , and These represent the prediction heads for observation, reward, and continuation probability, respectively. , , These represent the predicted observation features, reward value, and task continuation probability, respectively. The total loss function is defined as: (18) In the formula, Indicates the loss in observation reconstruction; Indicates the predicted loss from the reward; This represents the loss from predicting the probability of continuation. This represents the KL divergence loss between the prior and posterior distributions. Represents the variance regularization term of the global difference branch; This represents the sequence-level alignment loss.

[0012] Furthermore, step 6 specifically involves, during the online reasoning phase, using the posterior-calibrated state at the current moment as the initial imagined state: (19) In forward-looking step Within, a potential space rolling deduction is performed on the candidate action sequence; In every imaginative step First, candidate actions are sampled by the policy prior network: (20) Then, the potential state at the next moment is predicted using the causal dynamic path: (twenty one) (twenty two) In the formula, For prior features generated recursively based on imagined history, the incorporation of future real observations is not allowed; Each imagined trajectory is scored using the reward head and continuation probability head obtained during training. (twenty three) (twenty four) Construct the objective function for the expected cumulative discounted return of the candidate action sequence: (25) In the formula, Discount factor; multiplication term Used to weight rewards based on the probability of continuing the task; The cross-entropy method is used to perform a population search optimization of the above objective function; after optimization, the first action of the optimal candidate action sequence is selected as the final navigation command at the current time. And send it to the underlying controller for execution.

[0013] Furthermore, step 7 specifically involves the underlying controller performing an action. Afterwards, the environment returned to new observations. The system updates the historical action sequence, composite embedding sequence, prior features and posterior features accordingly, and repeats steps 2 to 6 until the output stops the action or the maximum time step limit is reached, thereby completing the visual language navigation task in a continuous environment.

[0014] A continuous environment visual language navigation system based on a global-local Mamba causal world model, the system using the aforementioned continuous environment visual language navigation method based on a global-local Mamba causal world model, the system comprising the following steps: Initialization module: Task input acquisition and runtime parameter initialization; Feature construction module: Performs multimodal coding and composite feature construction based on the tasks of the initialization module; Feature extraction module: Constructs a global-local Mamba perceptron and performs multi-scale spatiotemporal feature extraction on the composite features constructed by the feature construction module; State update module: Constructs a causal state inference network and performs structured latent state updates on the composite features constructed by the feature construction module; Offline training module: Based on the deterministic state in the state update module, it performs auxiliary supervision and loss optimization during the offline training phase; Online planning module: online forward planning and action output; Update module: Based on the online planning module, perform closed-loop updates and task termination through online forward planning.

[0015] A continuous environment visual language navigation method using the global-local Mamba causal world model as described above is applied to enable an agent to complete forward-looking autonomous navigation in a continuous three-dimensional environment based on natural language instructions.

[0016] The beneficial effects of this invention are: This invention can be directly used to command physical robots to perform precise and robust continuous path forward planning and autonomous navigation based on natural language instructions in unknown and complex three-dimensional environments (such as intelligent exploration and rescue in high-risk scenarios like post-disaster nuclear accident sites).

[0017] This invention introduces the Mamba state-space modeling mechanism into the RSSM world model framework for continuous environment visual language navigation tasks, thereby improving the modeling capability of long temporal dependencies.

[0018] This invention proposes a global-local dual-branch perceptron that simultaneously processes short-term local windows and long-term difference sequences, thereby taking into account both fine-grained local dynamic information and global motion trend information.

[0019] The present invention employs an asymmetric causal decoupling mechanism that calculates posterior and prior features, clearly distinguishing between the training calibration path and the inference planning path, thereby preventing current observations from leaking into the future state prediction process.

[0020] This invention proposes a multi-step forward planning method based on a structured potential space, which achieves action decisions oriented towards benefit and safety through multi-step imagination and cross-entropy search without relying on future real observations.

[0021] This invention can more effectively model environmental dynamics and path dependencies over long time scales, reducing the problems of easy forgetting in traditional RNNs and high computational overhead of long sequences in Transformers.

[0022] This invention strictly decouples prior and posterior information, ensuring that forward planning relies solely on historical information for reasoning, thereby improving causal consistency and reducing the risk of reasoning distortion caused by information leakage.

[0023] This invention enables multi-step forward-looking simulations within a potential space, allowing for early assessment of the cumulative benefits and potential risks of candidate actions, thereby improving navigation success rate, path rationality, and robustness under complex physical disturbances.

[0024] This invention uses a latent spatial world model instead of pixel-level future image reconstruction for planning, which can reduce computational burden and reduce visual redundancy information interference that is irrelevant to decision-making. Attached Figure Description

[0025] Figure 1 This is an overall overview diagram of the present invention.

[0026] Figure 2 This is a structural diagram of the present invention. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of this application with unnecessary detail.

[0028] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0029] It should also be understood that the terminology used in this application specification is for the purpose of describing particular embodiments only and is not intended to limit the application. As used in this application specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0030] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0031] Many specific details are set forth in the following description in order to provide a full understanding of this application. However, this application may also be implemented in other ways different from those described herein. Those skilled in the art can make similar extensions without departing from the spirit of this application. Therefore, this application is not limited to the specific embodiments disclosed below.

[0032] Implementation Method 1 This implementation provides a continuous environment visual-language navigation method based on the global-local Mamba causal world model, such as... Figure 1-2 As shown, Figure 1This diagram illustrates the overall technical approach of the present invention for visual language navigation tasks in continuous environments. It shows how language commands, historical RGB-D information, and current RGB-D information are used as inputs, processed through a feature space, and then fed into a global-local Mamba perceptron. Finally, it combines with a world model to complete feature prediction and action decision output. The diagram also shows verification scenarios of the invention on simulators, datasets, and benchmark test results, demonstrating the overall application and technical effectiveness of the invention.

[0033] Figure 2 This diagram illustrates the specific module connections and internal working mechanism of the present invention. It shows how the current RGB-D, language commands, historical actions, and historical states jointly participate in the construction of composite features, which are then processed through global Mamba branches, local Mamba branches, and adaptation modules to form a unified representation before being input into the world model. Finally, the policy module outputs navigation actions. The diagram also shows the structure of the selective state space module and the rolling updates of potential states over time, illustrating the specific implementation idea of ​​the present invention for achieving forward-looking planning and action generation based on causal state modeling.

[0034] The method includes the following steps: Step 1: Obtain task input and initialize runtime parameters; Step 2: Perform multimodal coding and composite feature construction based on the task in Step 1; Step 3: Construct a global-local Mamba perceptron and extract multi-scale spatiotemporal features from the composite features constructed in Step 2; Step 4: Construct a causal state inference network and perform structured latent state updates based on the multi-scale spatiotemporal features (including posterior and prior features) extracted in Step 3. Step 5: Based on the deterministic state in the state update in Step 4, perform auxiliary supervision and loss optimization during the offline training phase; Step 6: Based on the offline training in Step 5, perform online prospective planning and action output; Step 7: Based on the online forward planning in Step 6, perform closed-loop updates and terminate the task.

[0035] Furthermore, step 1 specifically involves the system receiving natural language instructions at the start of each navigation task. and initial observation ; at any time step The system receives the current RGB-D observations. Actions at the previous moment The random state at the previous time step and the deterministic state at the previous moment ; The current observation The system uses a multimodal input consisting of first-person view images and depth information, and the observation resolution can be set according to computing power conditions. The system adopts a waypoint-based high-level action space, where a policy network selects high-level navigation actions from a set of candidate directions and candidate distances, and then the low-level controller converts them into continuous control actions for execution. The task termination condition is: the output stops moving, or the maximum time step is reached. The maximum time step The preferred range for the license is 100 to 300.

[0036] Furthermore, step 2 specifically involves, for the current observation Visual encoding is performed to obtain visual features. : (1) In the formula, Indicates a visual encoder; Indicates time step Visual feature representation; Then, the current visual features Actions at the previous moment and the random state at the previous moment Mapping to a unified feature space to construct a composite input embedding : (2) In the formula, For action embedding layer; For random state embedding layer; For feature fusion coding function; This is for composite input embedding.

[0037] The observation embedding dimension, action embedding dimension, and state embedding dimension can be set to 256 to 1024, preferably 512.

[0038] Furthermore, step 3 specifically involves maintaining a short-term window sequence and a long-term difference sequence based on the composite input embedding obtained in step 2. The short-term window sequence is defined as: (3) Long-term difference sequences are defined as: (4) In the formula, The local time window length is indicated, with a permissible range of 4 to 16, and more preferably 8. This represents the global difference window length, which has a permissible range of 16 to 64, and is more preferably 32. Local branching utilizes local Mamba operators to extract high-fidelity local dynamic features: (5) In the formula, Represents a locally selective state-space operator; Representation layer normalization; Represents posterior local features; The global branch uses the global Mamba operator to process long-term difference sequences and extracts macroscopic trend features through a nonlinear projection head: (6) In the formula, Represents a globally selective state-space operator; The projection head preferably includes a linear layer, a SiLU activation layer, and a layer normalization layer. Represents posterior global features; Local and global features are asymmetrically fused using a gated fusion module to obtain posterior features. : (7) In the formula, Indicates the gated fusion function; This represents a multilayer perceptron; Represents a linear mapping matrix; Represents the posterior features used for training calibration; To ensure strict causal consistency in the reasoning process, this invention further constructs prior features. The prior features employ a network structure isomorphic to the posterior features, but their inputs must be time-shifted, using only the up-to-date data. Historical buffer sequence of moments and Perform the calculation: (8) In the formula, and These represent the global prior features and local prior features generated from the history window, respectively. It does not contain real observation information at the current moment, but only relies on historical information for subsequent state transitions and forward planning.

[0039] The permitted range of the fusion feature dimension is preferably 512 to 1024, and more preferably 768.

[0040] Furthermore, step 4 specifically involves the present invention employing a cyclic state-space model to maintain deterministic states. and random state The structured state of potential beliefs; The deterministic state update formula is: (9) In the formula, Indicates a gated loop unit; Indicates time step The deterministic state is used as long-term memory; its dimensionality is preferably 256–1024, more preferably 512; the update relies only on prior features. This avoids current observations being leaked into forward-looking prediction paths; During the training phase, posterior features are utilized. The random state is calibrated to construct the posterior distribution: (10) And the posterior random state is obtained using reparameterized sampling: (11) In the formula, and Let represent the mean and standard deviation of the posterior distribution, respectively; This represents element-wise multiplication; Indicates standard normal noise; In the prospective reasoning phase, based on the deterministic state... The random state at the previous time step Actions at the previous moment and prior features Predicting the prior distribution of a random state; First, construct the intermediate features: (12) The residual offset parameters are then output by the dynamic prediction network: (13) Finally, the prior values ​​of the random state are obtained: (14) In the formula, and These represent the mean and standard deviation of the residual offset relative to the random state at the previous time step, respectively; the permissible range of the random state dimension is preferably 256 to 1024, and more preferably 512.

[0041] Furthermore, step 5 specifically involves, during the offline training phase, based on the posterior state... and deterministic state We construct an observation reconstruction head, a reward prediction head, and a continuation probability prediction head, which are used to constrain latent state learning, respectively. The observation reconstruction head is defined as: (15) The reward prediction head is defined as: (16) The continuation probability prediction head is defined as: (17) In the formula, , and These represent the prediction heads for observation, reward, and continuation probability, respectively. , , These represent the predicted observation features, reward value, and task continuation probability, respectively. The total loss function is defined as: (18) In the formula, Indicates the loss in observation reconstruction; Indicates the predicted loss from the reward; This represents the loss from predicting the probability of continuation. This represents the KL divergence loss between the prior and posterior distributions. Represents the variance regularization term of the global difference branch; This represents the sequence-level alignment loss; , , , , , All are weighting coefficients greater than 0, and their permitted range is preferably 0.01 to 10.

[0042] Furthermore, step 6 specifically involves, during the online reasoning phase, using the posterior-calibrated state at the current moment as the initial imagined state: (19) In forward-looking step Within this framework, a potential space rolling deduction is performed on the candidate action sequence; the look-ahead step size... The permitted range is preferably 3 to 20, more preferably 5 to 10; In every imaginative step First, candidate actions are sampled by the policy prior network: (20) Then, the potential state at the next moment is predicted using the causal dynamic path: (twenty one) (twenty two) In the formula, For prior features generated recursively based on imagined history, the incorporation of future real observations is not allowed; Each imagined trajectory is scored using the reward head and continuation probability head obtained during training. (twenty three) (twenty four) Construct the objective function for the expected cumulative discounted return of the candidate action sequence: (25) In the formula, The discount factor is preferably within the range of 0.90 to 0.999, more preferably 0.95 to 0.99; multiplication term It is used to weight rewards based on the probability of mission continuation, and to suppress dangerous trajectories that may lead to collisions, falls, jamming or premature mission termination. The cross-entropy method is preferably used to perform a population-based search optimization of the above objective function; population size The preferred license range is 64–512, more preferably 128–256; elite sample ratio The preferred range of the license is 5% to 20%, more preferably 10%; number of iterations The preferred range for the allowed action sequence is 2 to 8, and more preferably 3 to 5. After optimization, the first action of the optimal candidate action sequence is selected as the final navigation command for the current moment. And send it to the underlying controller for execution.

[0043] Furthermore, step 7 specifically involves the underlying controller performing an action. Afterwards, the environment returned to new observations. The system updates the historical action sequence, composite embedding sequence, prior features and posterior features accordingly, and repeats steps 2 to 6 until the output stops the action or the maximum time step limit is reached, thereby completing the visual language navigation task in a continuous environment.

[0044] This includes what effects can be achieved when certain parameters in the algorithm are taken to their optimal range or optimal value.

[0045] In a preferred embodiment, the observation input uses first-view RGB-D data; the visual encoder output dimension is set to 512; the composite input embedding dimension is set to 512; and the deterministic state dimension... Set to 512; Random state dimension Set to 512; Local window length Set to 8; Global difference window length Set to 32; global-local fusion feature dimension set to 768. In the online planning phase, the lookahead step size... It can be set to 8, discount factor. It can be set to 0.99, which is the population size for the cross-entropy method. It can be set to 256, the proportion of elite samples. It can be set to 10% to optimize the number of iterations. It can be set to 5.

[0046] In this embodiment, the system first constructs a composite input embedding based on natural language instructions and current RGB-D observations, and then generates posterior and prior features using a global-local Mamba perceptron. Subsequently, the posterior features are used for state calibration and auxiliary supervision during the training phase, while the prior features are used for deterministic state updates and stochastic state prior predictions during the online inference phase. Finally, the planner performs rolling deductions on a sequence of actions for several future steps in the latent space, calculates the expected cumulative discount reward, selects the optimal action from the candidate action sequence, and outputs it to the underlying controller.

[0047] When the above implementation method is adopted, the system can realize look-ahead planning based on historical causal information in continuous environments. Compared with reactive methods that only respond based on current observations, it is more conducive to improving generalization performance, path decision stability, and robustness to physical disturbances in unseen environments. To verify the effectiveness of the proposed continuous environment visual language navigation method based on a global-local Mamba perception and causal world model, the method of this invention was compared with several existing representative methods on an unseen environment validation set. The comparison results are shown in Table 1. The table selects two representative continuous environment visual language navigation test tasks: R2R Val-Unseen and RxR Val-Unseen, to evaluate the target arrival capability, path execution quality, and overall generalization performance of each method in unfamiliar scenarios. Through horizontal comparison under a unified evaluation index system, the navigation performance of the proposed method in complex unseen environments can be intuitively reflected.

[0048] In Table 1, S.RGB represents single-view RGB visual input, Pano. represents panoramic visual input, Depth represents depth information input, and Odo. represents odometry information input. TL and NE are cost-based evaluation indicators, where TL represents trajectory length and NE represents navigation error. The smaller the values ​​of these two indicators, the lower the navigation cost and the higher the target arrival accuracy. OSR, SR, SPL, and nDTW are performance-based evaluation indicators, where OSR represents Oracle success rate, SR represents success rate, SPL represents success rate normalized to path length, and nDTW represents normalized dynamic time warping index. The larger the values ​​of these indicators, the better the navigation success, path rationality, and trajectory consistency.

[0049] Table 1 shows that, using only single-view RGB and depth information, the method of the present invention achieved TL of 12.20, NE of 4.20, OSR of 68.1, SR of 59.3, and SPL of 51.3 in the R2RVal-Unseen test task; and NE of 5.82, SR of 55.3, SPL of 47.3, and nDTW of 64.3 in the RxRVal-Unseen test task. These results demonstrate that the method of the present invention exhibits high target-to-achievement power, low endpoint deviation, and good path execution quality in unseen environments, effectively enabling visual-language navigation tasks in continuous environments.

[0050] Further analysis reveals that, compared to other representative methods using single-view visual input in the table, the method of this invention exhibits significant advantages in several key indicators. Taking R2RVal-Unseen as an example, the NE index of the method of this invention is reduced to 4.20, indicating a smaller endpoint error when reaching the target area; the OSR index reaches 68.1, the SR index reaches 59.3, and the SPL index reaches 51.3, demonstrating that the method of this invention not only completes navigation tasks more stably but also maintains a high level of path execution efficiency. These results demonstrate that the global-local Mamba perception mechanism constructed in this invention can more fully integrate language commands, current observations, and historical temporal information, enhancing the robot's joint understanding of environmental structure and task semantics, thereby improving the accuracy of navigation decisions.

[0051] In the RxRVal-Unseen test task, the method of this invention also achieved superior results. Specifically, NE was 5.82, SR was 55.3, SPL was 47.3, and nDTW was 64.3, indicating that the method of this invention still possesses good path-following ability and trajectory consistency in more challenging unseen environment tasks. In particular, the high nDTW index indicates that the navigation trajectory generated by the method of this invention has high consistency with the reference trajectory, and can complete navigation well along a path that conforms to the task semantics, without easily encountering problems such as deviation, invalid detours, or premature failure.

[0052] Furthermore, as shown in Table 1, the method of this invention does not rely on panoramic visual input and odometry information. Despite relatively limited input modalities, it still achieves a high navigation success rate and good path efficiency. This demonstrates that the proposed technical solution possesses strong capabilities in historical state modeling, local dynamic change modeling, and long-range dependency modeling, enabling more stable and reliable navigation decisions under conditions of limited perceptual information. In other words, this invention does not simply rely on additional sensor information to improve performance; rather, it enhances the system's generalization ability and decision robustness by improving environmental representation methods, state inference mechanisms, and look-ahead planning methods.

[0053] The experimental results in Table 1 demonstrate that the continuous environment visual-language navigation method based on the global-local Mamba perception and causal world model proposed in this invention exhibits good overall performance in unseen environment test tasks. This method effectively reduces navigation endpoint error, improves task success rate and path execution quality, and maintains strong environmental adaptability and generalization ability under limited input conditions, thus verifying the effectiveness and practical value of the technical solution of this invention.

[0054] Table Notes: TL represents trajectory length; NE represents navigation error; OSR represents Oracle success rate; SR represents success rate; SPL represents success rate normalized to path length; nDTW represents normalized dynamic time warping index. For TL and NE, smaller values ​​are better; for OSR, SR, SPL, and nDTW, larger values ​​are better.

[0055] Table 1. Performance comparison of the method of the present invention and existing visual language navigation methods on an unseen environment validation set.

[0056] Implementation Method 2 This embodiment provides a continuous environment visual language navigation system based on a global-local Mamba causal world model. The system uses the continuous environment visual language navigation method based on a global-local Mamba causal world model as described in Embodiment 1. The system includes the following steps: Initialization module: Task input acquisition and runtime parameter initialization; Feature construction module: Performs multimodal coding and composite feature construction based on the tasks of the initialization module; Feature extraction module: Constructs a global-local Mamba perceptron and performs multi-scale spatiotemporal feature extraction on the composite features constructed by the feature construction module; State update module: Constructs a causal state inference network and performs structured latent state updates on the composite features constructed by the feature construction module; Offline training module: Based on the deterministic state in the state update module, it performs auxiliary supervision and loss optimization during the offline training phase; Online planning module: online forward planning and action output; Update module: Based on the online planning module, perform closed-loop updates and task termination through online forward planning.

[0057] Implementation Method 3 This embodiment provides a continuous environment visual language navigation method using a global-local Mamba causal world model as described in Embodiment 1, which enables an agent to complete forward-looking autonomous navigation in a continuous three-dimensional environment based on natural language instructions.

Claims

1. A continuous environment visual-language navigation method based on a global-local Mamba causal world model, characterized in that, The method includes the following steps: Step 1: Obtain task input and initialize runtime parameters; Step 2: Perform multimodal coding and composite feature construction based on the task in Step 1; Step 3: Construct a global-local Mamba perceptron and extract multi-scale spatiotemporal features from the composite features constructed in Step 2; Step 4: Construct a causal state inference network and perform structured latent state updates based on the multi-scale spatiotemporal features extracted in Step 3; Step 5: Based on the deterministic state in the state update in Step 4, perform auxiliary supervision and loss optimization during the offline training phase; Step 6: Based on the offline training in Step 5, perform online prospective planning and action output; Step 7: Based on the online forward planning in Step 6, perform closed-loop updates and terminate the task.

2. The method according to claim 1, characterized in that, Specifically, step 1 involves the system receiving natural language instructions at the start of each navigation task. and initial observation ; at any time step The system receives the current RGB-D observations. Actions at the previous moment The random state at the previous time step and the deterministic state at the previous moment ; The current observation The system uses a multimodal input consisting of first-person view images and depth information, and the observation resolution can be set according to computing power conditions. The system adopts a waypoint-based high-level action space, where a policy network selects high-level navigation actions from a set of candidate directions and candidate distances, and then the low-level controller converts them into continuous control actions for execution. The task termination condition is: the output stops moving, or the maximum time step is reached. .

3. The method according to claim 2, characterized in that, Step 2 specifically involves analyzing the current observations. Visual encoding is performed to obtain visual features. : （1） In the formula, Indicates a visual encoder; Indicates time step Visual feature representation; Then, the current visual features Actions at the previous moment and the random state at the previous moment Mapping to a unified feature space to construct a composite input embedding : （2） In the formula, For action embedding layer; For random state embedding layer; For feature fusion coding function; This is for composite input embedding.

4. The method according to claim 3, characterized in that, Specifically, step 3 involves maintaining a short-term window sequence and a long-term difference sequence based on the composite input embedding obtained in step 2. The short-term window sequence is defined as: （3） Long-term difference sequences are defined as: （4） In the formula, Indicates the length of the local time window; Indicates the length of the global difference window; Local branching utilizes local Mamba operators to extract high-fidelity local dynamic features: （5） In the formula, Represents a locally selective state-space operator; Representation layer normalization; Represents posterior local features; The global branch uses the global Mamba operator to process long-term difference sequences and extracts macroscopic trend features through a nonlinear projection head: （6） In the formula, Represents a globally selective state-space operator; The projection head preferably includes a linear layer, a SiLU activation layer, and a layer normalization layer. Represents posterior global features; Local and global features are asymmetrically fused using a gated fusion module to obtain posterior features. : （7） In the formula, Indicates the gated fusion function; This represents a multilayer perceptron; Represents a linear mapping matrix; Represents the posterior features used for training calibration; Constructing prior features The prior features employ a network structure isomorphic to the posterior features, but their inputs must be time-shifted, using only the up-to-date data. Historical buffer sequence of moments and Perform the calculation: （8） In the formula, and These represent the global prior features and local prior features generated from the history window, respectively. It does not contain real observation information at the current moment, but only relies on historical information for subsequent state transitions and forward planning.

5. The method according to claim 2, characterized in that, Step 4 specifically involves using a cyclic state-space model to maintain the deterministic state. and random state The structured state of potential beliefs; The deterministic state update formula is: （9） In the formula, Indicates a gated loop unit; Indicates time step A deterministic state, used as long-term memory; During the training phase, posterior features are utilized. The random state is calibrated to construct the posterior distribution: （10） And the posterior random state is obtained using reparameterized sampling: （11） In the formula, and Let represent the mean and standard deviation of the posterior distribution, respectively; This represents element-wise multiplication; Indicates standard normal noise; In the prospective reasoning phase, based on the deterministic state... The random state at the previous time step Actions at the previous moment and prior features Predicting the prior distribution of a random state; First, construct the intermediate features: （12） The residual offset parameters are then output by the dynamic prediction network: （13） Finally, the prior values ​​of the random state are obtained: （14） In the formula, and These represent the mean and standard deviation of the residual shift relative to the random state at the previous time step, respectively.

6. The method according to claim 5, characterized in that, Specifically, step 5 involves, during the offline training phase, based on the posterior state... and deterministic state We construct an observation reconstruction head, a reward prediction head, and a continuation probability prediction head, which are used to constrain latent state learning, respectively. The observation reconstruction head is defined as: （15） The reward prediction head is defined as: （16） The continuation probability prediction head is defined as: （17） In the formula, , and These represent the prediction heads for observation, reward, and continuation probability, respectively. , , These represent the predicted observation features, reward value, and task continuation probability, respectively. The total loss function is defined as: （18） In the formula, Indicates the loss in observation reconstruction; Indicates the predicted loss from the reward; This represents the loss from predicting the probability of continuation. This represents the KL divergence loss between the prior and posterior distributions. Represents the variance regularization term of the global difference branch; This represents the sequence-level alignment loss.

7. The method according to claim 2, characterized in that, Specifically, step 6 involves the online reasoning phase, where the posterior-calibrated state at the current moment is used as the initial imagined state: （19） In forward-looking step Within, a potential space rolling deduction is performed on the candidate action sequence; In every imaginative step First, candidate actions are sampled by the policy prior network: （20） Then, the potential state at the next moment is predicted using the causal dynamic path: （21） （22） In the formula, For prior features generated recursively based on imagined history, the incorporation of future real observations is not allowed; Each imagined trajectory is scored using the reward head and continuation probability head obtained during training. （23） （24） Construct the objective function for the expected cumulative discounted return of the candidate action sequence: （25） In the formula, Discount factor; multiplication term Used to weight rewards based on the probability of continuing the task; The cross-entropy method is used to perform a population search optimization of the above objective function; after optimization, the first action of the optimal candidate action sequence is selected as the final navigation command at the current time. And send it to the underlying controller for execution.

8. The method according to claim 7, characterized in that, Specifically, step 7 involves the underlying controller performing an action. Afterwards, the environment returned to new observations. The system updates the historical action sequence, composite embedding sequence, prior features and posterior features accordingly, and repeats steps 2 to 6 until the output stops the action or the maximum time step limit is reached, thereby completing the visual language navigation task in a continuous environment.

9. A continuous environment visual-language navigation system based on a global-local Mamba causal world model, characterized in that, The system uses the continuous environment visual-language navigation method based on the global-local Mamba causal world model as described in any one of claims 1-8, and the system includes the following steps: Initialization module: Task input acquisition and runtime parameter initialization; Feature construction module: Performs multimodal coding and composite feature construction based on the tasks of the initialization module; Feature extraction module: Constructs a global-local Mamba perceptron and performs multi-scale spatiotemporal feature extraction on the composite features constructed by the feature construction module; State update module: Constructs a causal state inference network and performs structured latent state updates on the composite features constructed by the feature construction module; Offline training module: Based on the deterministic state in the state update module, it performs auxiliary supervision and loss optimization during the offline training phase; Online planning module: online forward planning and action output; Update module: Based on the online planning module, perform closed-loop updates and task termination through online forward planning.

10. A continuous environment visual language navigation method using a global-local Mamba causal world model as described in any one of claims 1-8, which is applied to enable an agent to complete forward-looking autonomous navigation in a continuous three-dimensional environment based on natural language instructions.

Images