A structured evidence perception control method and system based on continuous environmental visual language navigation

Abstract

The application discloses a structured evidence perception control method and system based on continuous environment visual language navigation, which decomposes natural language instructions into an ordered subtask sequence. For different subtasks, a completion judgment function is defined through multi-dimensional progress state and reliable observation evidence to accurately determine whether the subtask is truly completed, instead of relying on single-frame observation. Reliable observation evidence is obtained through open vocabulary perception, candidate target screening table and double-channel visual language verification, and for geometric relationship termination type subtasks, an anchor point priority strategy is additionally used for relationship authentication. The action decision of the mobile robot adopts a fast-slow dual system cooperative mechanism: the fast system is responsible for frame-by-frame obstacle avoidance and local recovery control, and the slow system only intervenes when semantic conflict occurs or task handover is needed, and performs semantic supervision and arbitration. The application significantly improves the navigation performance, path quality and navigation efficiency of the mobile robot.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, robotics and autonomous navigation technology, and specifically relates to a structured evidence perception and control method and system based on continuous environment visual language navigation. Background Technology

[0002] Visual language navigation is an important research direction in the field of embodied intelligence. Its goal is to enable mobile robots to autonomously perceive, explore paths, and reach targets in unknown 3D environments based on natural language commands. Early research on visual language navigation, such as room-to-room baseline tasks, was mostly based on discrete topological maps or panoramic node environments. In such settings, mobile robots typically select jump actions among predefined candidate viewpoints, with the core focus on natural language command encoding, cross-modal visual semantic matching, and path node selection. This discretized paradigm simplifies physical motion and geometric constraints but fails to reflect the motion characteristics of robots or mobile robots in continuous physical space in the real world.

[0003] With the emergence of continuous environment navigation benchmarks such as R2R-CE, visual-language navigation tasks have been extended to continuous three-dimensional environments. In this paradigm, mobile robots can no longer rely on instantaneous jumps between discrete nodes, but must move gradually and continuously in physical space through low-level actions such as "forward," "left turn," "right turn," and "stop." Each action changes the mobile robot's position, orientation, and subsequent field of view. Local collisions, narrow passage blockages, viewpoint occlusion, and depth sensor noise accumulate over time, ultimately affecting the task execution state. Therefore, continuous environment visual-language navigation is not only a cross-modal semantic understanding problem, but also a hybrid control problem constrained by local geometry, action execution errors, observation noise, and long-term state transitions.

[0004] In continuous environment visual-language navigation, mobile robots typically rely solely on first-person RGB images, depth images, natural language commands, and their own historical action sequences for decision-making. Compared to discrete environment navigation, the system not only needs to determine whether a target object appears in the current view, but more importantly, it needs to determine whether the linguistic intent corresponding to the current subtask's natural language command has actually been completed in physical space. The first entry of a target into the robot's field of vision only indicates the presence of visual cues related to the task objective; it does not directly prove that the physical event required by the natural language command has been completed. For example, observing a target area does not equate to entering that area, approaching a landmark does not equate to having passed through or turned, and recognizing an anchor point does not equate to the corresponding spatial relationship being firmly established. If the task state is advanced solely based on the visibility of a single-frame target, target distance, or image geometry, it can easily lead to premature task switching or unexpected termination.

[0005] From the perspective of control systems, current zero-shot visual language navigation methods mainly face the following three interrelated core technical challenges:

[0006] (1) Unreliability of open-vocabulary perceptual observations (false detection problem). To achieve understanding of the open world, existing systems generally use visual language basic models such as CLIP and Grounding DINO for open-vocabulary target detection. However, the output of these models is essentially a highly uncertain semantic observation based on the statistical correlation between images and text. In real-world environments, wall decoration patterns, mirror reflections, television screen content, or distant blurred objects may all be misclassified by the model as high-confidence target candidates, forming false target observations. If these unfiltered and unverified false targets are directly input into the navigation decision closed loop, it will seriously interfere with the system's state estimation, action selection, and final stopping decision, causing the mobile robot to stop prematurely, wander in the wrong position, or deviate significantly from the path.

[0007] (2) Heterogeneity of Subtask Completion Criteria in Long Instructions. A complete natural language navigation instruction typically contains multiple subtasks with sequential dependencies, such as "enter the kitchen," "pass by the sink," "turn right at the door," "walk to the sofa," and "stop at the bedside." These subtasks differ fundamentally in their action intentions (entering, passing through, turning, approaching, and relationship termination) and target semantics, resulting in drastically different completion conditions in the physical world. "Entering" a region requires a combination of region semantics and effective displacement judgment; "passing through" requires a combination of approaching, traversing, and subsequent cues judgment; and "relationship termination" requires first confirming the anchor point entity before judging the spatial relationship. If a uniform target visibility threshold or simple distance threshold is used as the completion standard for all subtasks, it will inevitably lead to a large number of state misjudgments, such as misjudging "seeing the target" as "entry completed" or "approaching the anchor point" as "relationship satisfied," thereby causing incorrect subtask switching and navigation failure.

[0008] (3) The efficiency and stability contradiction between high-level semantic reasoning and low-level continuous control of large language models. Large language models, represented by the GPT series, have shown strong capabilities in natural language understanding, task decomposition, and common sense reasoning, providing new possibilities for zero-shot navigation. However, directly placing large language models at the core of the control closed loop, allowing them to receive perceptual information frame by frame and output low-level action commands, has significant drawbacks.

[0009] First, large models suffer from slow inference speed and high computational overhead. Using them for frame-by-frame control results in extremely low control frequency, failing to meet the demands of real-time obstacle avoidance and smooth motion in continuous environments. Second, the output of large models has a certain degree of randomness, and the low-level action sequences they directly generate may lack physical smoothness and consistency, easily leading to control oscillations. More importantly, tasks such as local obstacle avoidance and recovery from dead ends in continuous environments are essentially high-frequency, reactive control problems, better suited for handling by dedicated, efficient local controllers rather than by large models that excel at slow thinking and semantic planning. The inference cycle of large language models differs significantly from that of low-level motion control; a serial perception-inference-execution approach tends to reduce control frequency. While asynchronous architectures or high-low layer divisions can mitigate inference latency, high-level semantic modules may still repeatedly intervene due to temporary pauses, changes in perspective, or insufficient short-term evidence when the robot experiences local blockages, enters locally impassable areas, or performs recovery actions, affecting the continuity of the local recovery process.

[0010] Therefore, effectively decoupling and coordinating low-frequency, high-intelligence large language model semantic supervision with high-frequency, robust underlying motion control is the key to improving the overall efficiency and stability of the system.

[0011] Existing zero-shot visual-language navigation methods, such as the multi-expert discussion framework used in Discuss-Nav and the open-source large-model planning strategy used in Open-Nav, have made valuable attempts to utilize large models for task understanding. However, these methods typically treat the large model as a relatively independent planning module, lacking a structured, tightly coupled closed-loop feedback mechanism between its output and the mobile robot's current precise geometric state, fine-grained task execution progress, and historical action sequences. Furthermore, these methods lack a systematic process for cleaning, validating, and assigning weights to the high-noise observation data generated by the open vocabulary perception module. In addition, they fail to adequately consider the local geometric constraint anomalies that mobile robots may encounter when performing actions in continuous environments and the corresponding autonomous recovery mechanisms. These shortcomings limit the navigation success rate and path efficiency of existing methods in complex continuous environments.

[0012] In summary, to address the three major challenges in visual-language navigation in continuous environments—unreliable perceptual evidence, ambiguous sub-task state determination, and difficulty in coordinating high-level semantics and low-level control—a novel control method is urgently needed. This method requires establishing a refined state management mechanism for heterogeneous sub-tasks, effectively defining the rights to multimodal perceptual evidence, and designing an efficient system architecture that coordinates semantic reasoning and physical control, thereby achieving more reliable and efficient autonomous navigation under zero-shot conditions. Summary of the Invention

[0013] To address the problems existing in the prior art, this invention proposes a structured evidence-aware control (SEAL-Nav) method and system based on continuous environment visual language navigation. The proposed SEAL-Nav method models continuous environment navigation as a hybrid state control problem composed of discrete task states and continuous local motion states: natural language instructions are decomposed into a finite sequence of subtasks, each subtask corresponding to specific observation requirements, completion criteria, and state transition conditions; a multimodal evidence-aware module serves as a heterogeneous observation source oriented towards task states, filtering and verifying semantic and geometric observations; a fast and slow dual system respectively undertakes the high-frequency local execution loop and the low-frequency semantic supervision loop, achieving asynchronous collaboration through an event-triggered mechanism.

[0014] From a control structure perspective, the SEAL-Nav system proposed in this invention can be understood as a hierarchical decision-making system composed of a low-frequency semantic supervision loop and a high-frequency local execution loop. The semantic large model provides task constraints and uncertainty arbitration, while the local control module maintains the stability of closed-loop execution. Unlike existing end-to-end large model navigation, SEAL-Nav does not allow the large model to directly output low-level actions step by step. Instead, it restricts it to key nodes such as instruction decomposition, semantic arbitration, region reflection, and handover verification, thus decoupling high-level semantic reasoning from low-level physical control. In addition, the system also has a feedback buffer to cache the memory information of historical actions and observations (including historical observations, action sequences, and abnormal event records). At each time step, the feedback buffer outputs the memory information of historical actions and observations, which is transmitted to the multimodal evidence fusion module and the fast local controller, respectively, to provide historical context for the current decision and assist in target confirmation and obstacle avoidance recovery.

[0015] The method and system proposed in this invention aim to address the following issues in existing technologies: how to design differentiated and precise physical completion judgment conditions for sub-tasks of different semantic types to avoid misjudgments during state transitions and solve the problem that "target visibility does not equal physical event completion"; how to effectively clean and assign weights to high-noise observations generated by open-vocabulary perception to improve the reliability of navigation decision-making; and how to design a control architecture to coordinate low-frequency, highly intelligent semantic reasoning with high-frequency, robust motion control to achieve efficient and stable continuous navigation while ensuring intelligence, so as to realize reliable and efficient autonomous navigation of mobile robots in unknown continuous three-dimensional indoor environments.

[0016] To achieve the above-mentioned technical objectives, the present invention provides the following technical solution: A structured evidence perception and control method based on continuous environmental visual language navigation specifically includes the following steps: S1. Receive natural language instructions for instructing the mobile robot to perform navigation tasks, and preprocess the natural language instructions; S2. Decompose the preprocessed natural language instructions into an ordered sequence of subtasks. The task description of each subtask includes: target, action intention, and local language description. S3. Determine the task type of each subtask through the task routing function. The task types include: area entry, traversal, landmark turning, target approach, and geometric relationship termination. S4. At each time step t during the navigation task, acquire images from the first-person perspective of the mobile robot, including RGB images and depth images; S5. Define and update the internal state of the mobile robot at each time step t. ,in: For discrete task status marking; A local geometric map maintained online from depth images; It is a multidimensional progress state vector, which is updated by information from three levels: semantic features, geometric features and temporal features, to quantify the execution progress of the current subtask. For the memory information of historical actions and observations; S6. At each time step, based on the RGB image and depth image, perform multimodal perception and target weighting on the target of the currently executing sub-task to obtain multimodal observation evidence. For geometric relationship termination subtasks, after obtaining multimodal observation evidence, an additional anchor-first strategy is used to determine relationship rights. S7. At each time step, based on the current subtask type... Call the corresponding completion determination function Determine if the current subtask is complete; if If the condition is met, the task is considered complete; otherwise, it is considered incomplete. The completion determination function takes the internal state and multimodal observation evidence as input. The completion determination functions of different subtasks adopt different combinations of semantic features, geometric features, temporal features and multimodal observation evidence in the internal state. S8. Update the discrete task status flags based on the judgment result: ; If the discrete task state is marked at time step t+1 and If K is the total number of subtasks, then proceed to step S9; otherwise, proceed to step S10. S9, Output Stop Action The navigation task has ended; S10. A hierarchical decision-making architecture based on fast and slow dual-system collaboration: the fast system executes local action control frame by frame, while the slow system only intervenes to perform semantic arbitration when a semantic event is triggered, outputting navigation actions according to a preset priority. At the same time, a timer interlocking mechanism is used in the decision-making process; after each navigation action is output, the system returns to step S4 to continue execution.

[0017] Furthermore, the multidimensional progress state vector is specifically as follows: ; in, For target visibility, For the degree of proximity to the target, For the degree of regional matching, The degree of matching between the opening or the direction of passage. For the progress of the journey, For the degree of geometric relationship matching, The degree of stagnation.

[0018] Furthermore, step S6 specifically includes: The RGB image obtained at the current time step t is processed using an open vocabulary foundation model. Process the data to generate a set of candidate targets related to the current subtask objective. where n is the total number of candidate targets; For each candidate target Combined with the depth map obtained at the current time step t Estimate the approximate distance between it and the mobile robot. and confidence level ; The candidate target set is filtered by distance gating, category matching, spatial blacklisting, and geometric constraints to obtain the effective candidate set. ; Then, select the candidate target with the highest confidence from the effective candidate set as the optimal candidate target. ; Dual-channel visual language verification: Two heterogeneous visual language models are used to independently determine the weight of the optimal candidate target. The visual language weight determination function is as follows: ; in, This is the distance-gated threshold. and The independent weight determination results of the two heterogeneous visual language models are given if and only if the approximate distance does not exceed the distance gating threshold and both weight determination results are true. The evidence was confirmed as reliable multimodal observational evidence; the reliable multimodal observational evidence included the optimal candidate target. Visual features, approximate distance and semantic matching degree; if the optimal candidate target fails the dual-channel visual language verification, then multimodal observation evidence It is an empty set.

[0019] More specifically, for the geometric relation termination subtask, after obtaining multimodal observation evidence, the additional anchor-first strategy for relational weighting is as follows: From local language description Extract anchor point objects Set up a relationship ownership determination function for: ; in, This indicates the conditions for confirming the anchor point target. Indicates spatial relationship matching conditions. , For indicator functions, An anchor object identification function, when the anchor object If its existence can be detected, output a true value of 1; otherwise, output a false value of 0. This is a spatial relation matching function. This is the threshold for matching spatial relationships.

[0020] Further, in step S7, the completion determination function is specifically: For region entry subtasks, remember... The completion decision function is: ; in, For indicator functions, These are the thresholds for region matching, effective displacement, and opening matching, respectively. This represents the cumulative effective displacement since entering the candidate state; For the sub-task of passing through, remember... The completion decision function is: ; in, These are the thresholds for proximity and traversal progress, respectively. Indicates whether there are subsequent targets or clues to new areas; For landmark turning sub-tasks, remember... The completion decision function is: ; in, This represents the target landmark object corresponding to the current subtask; The relative angle between the robot's current orientation and the target landmark; The allowed steering angle threshold; Number of steps to confirm steering stability; For subtasks similar to the target class, let _ The completion decision function is: ; in, To stop the distance threshold, To get closer to the evidence Required number of stability confirmation steps; For geometric relation termination subtasks, let The completion decision function is: ; in, The number of stable confirmation steps required to confirm the relationship.

[0021] Further, in step S10, the step of outputting navigation actions according to a preset priority specifically includes: Based on a hierarchical decision-making architecture that coordinates fast and slow systems, navigation actions are output according to a preset priority strategy. , ; in, To restore the control policy, the control restoration trigger condition must be met. Execute at any time; For task handover strategy; For a slow system, a semantic supervision strategy is adopted if and only if there are semantic events. Executes when triggered; This is a regular rule-based action strategy based on the current task type.

[0022] Furthermore, the specific conditions for triggering the recovery control are as follows: ; in, For indicator functions, For forward traversable space estimation, For the most recent The effective displacement of the mobile robot within a time step For the most recent The number of times the mobile robot turns within a time step. These are the forward passable space threshold, the effective displacement threshold, and the number of turns threshold, respectively.

[0023] Furthermore, the slow system only intervenes to perform semantic arbitration when a semantic event is triggered, specifically as follows: Semantic events are triggered under abnormal circumstances such as conflicting multimodal observation evidence, insufficient multimodal observation evidence, uncertain region matching, prolonged robot stagnation, or failed task handover. Including: Conflicts in the confirmation of ownership of targets Long-term stagnation Uncertainty in regional matching Arbitration for termination of relationship And task handover failure When at least one semantic event is triggered, the slow system semantic supervision strategy is executed: the slow system performs semantic arbitration based on the exception type corresponding to the different semantic events triggered, and outputs correction instructions.

[0024] Furthermore, the timer interlock mechanism is specifically as follows: When the hierarchical decision-making architecture is in the recovery control triggered state, i.e. At this time, the trigger timer of the slow system is frozen: ; in, , This indicates the trigger timer of the slow system at time steps t+1 and t; after the control recovery ends, the trigger timer of the slow system resumes normal updates.

[0025] Furthermore, this application also discloses a structured evidence perception and control system based on continuous environment visual language navigation, which is used to perform the method described above, specifically including: The instruction parsing and routing module is used to decompose the input natural language instructions into an ordered sequence of subtasks and determine the type of each subtask; The multimodal perception and rights confirmation module is used to acquire first-view RGB images and depth images of the mobile robot, and obtain multimodal observation evidence through open vocabulary base model, candidate target screening and dual-channel visual language verification; and additionally performs relation rights confirmation for geometric relation termination subtasks. The status management and progress tracking module is used to build and update the progress status of the mobile robot task, maintain the local geometric map, and call the corresponding completion judgment function according to the current subtask type to determine whether the subtask is completed and update the task status; it is also equipped with a feedback buffer to store the memory information of historical actions and observations, providing historical context for decision-making. The fast and slow dual-system collaborative decision-making module is used to execute local action control frame by frame through the fast system, while the slow system only intervenes to perform semantic arbitration when a semantic event is triggered, and outputs navigation actions according to the preset priority. The action execution interface is used to convert the navigation actions into low-level control commands for the mobile robot and output them.

[0026] Based on the above technical solution, the present invention has at least the following beneficial effects: This invention introduces a structured state mechanism to design differentiated physical completion judgment functions based on multidimensional progress states for different types of subtasks, fundamentally solving the misjudgment problem that "visibility of the target does not equal completion of the physical event", and significantly improving the accuracy of subtask switching and the overall success rate of navigation.

[0027] This invention constructs a multimodal evidence confirmation mechanism, which consists of candidate target generation and screening, distance gating, dual-channel visual language verification, and anchor point priority strategy to form an evidence admission mechanism. Only reliable multimodal observation evidence is allowed to participate in the sub-task completion judgment. This effectively cleans up false targets and noisy observations generated by open vocabulary perception, provides highly reliable perception evidence for navigation decision-making, and greatly reduces erroneous stops and path deviations caused by perception illusions.

[0028] This invention designs a fast-slow dual-system collaborative mechanism to decouple high-frequency, robust low-level motion control (fast system) from low-frequency, highly intelligent semantic reasoning supervision (slow system). Through priority scheduling and timer freeze interlocking, it ensures that the stability of local recovery control is not interrupted by high-level reasoning, reduces the frequency of unnecessary high-level semantic intervention and the resulting additional path length, and ensures the real-time performance, efficiency and overall stability of control while utilizing the intelligence of large models.

[0029] This invention significantly improves the navigation performance, navigation path quality, and navigation system efficiency of mobile robots; at the same time, it constructs a complete hybrid state control system with clear internal states and well-defined decision-making logic, exhibiting good debuggability and interpretability. Attached Figure Description

[0030] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a diagram illustrating the overall framework of SEAL-Nav, a structured evidence perception and control method based on continuous environmental visual language navigation proposed in this invention. Figure 2 This is a schematic diagram of the R2R-CE task scenario and subtask decomposition in the embodiment; Figure 3 The diagram shows a comparison of SR, SPL, TL, and NE in the ablation experiment of the example. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0032] Although the steps in this invention are arranged by reference numerals, this is not intended to limit the order of the steps. Unless the order of the steps is explicitly stated or the execution of a step requires other steps as a basis, the relative order of the steps can be adjusted. It is understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items.

[0033] This embodiment uses the R2R-CE dataset run in Habitat Simulator as an example to demonstrate the specific implementation process of SEAL-Nav, a structured evidence-aware control method based on continuous environment visual language navigation proposed in this invention. Figure 1 As shown, the SEAL-Nav framework mainly consists of a slow semantic supervision module (slow system) responsible for upper-level reasoning, a fast local controller (fast system) responsible for lower-level execution, and a multimodal evidence fusion layer for implementing the evidence-driven fast and slow systems. Based on the SEAL-Nav framework, the method proposed in this invention specifically includes the following steps:

[0034] S1. Receive natural language instructions to direct the mobile robot to perform navigation tasks. ,in For instruction morphemes, The instruction length is specified, and the natural language instructions are preprocessed.

[0035] In a preferred embodiment, the preprocessing specifically involves instruction normalization, which eliminates redundant expressions from the original instructions. For example, colloquial instructions such as "Walk straight toward the bar with the chairs / stools, turn left..." are converted into more structured expressions.

[0036] S2. Decompose the preprocessed natural language instructions into an ordered sequence of subtasks. ,in This represents the total number of subtasks; the task description for each subtask includes: objective. Intention of the action and local language description ;

[0037] As a preferred embodiment, in this embodiment, such as Figure 2As shown, step S2 specifically involves:

[0038] The normalized natural language instructions are input into a large language model (such as GPT-4) to extract action transition points and target landmarks from the instructions, generating a sequence of logical subtasks; taking a typical instruction from the R2R-CE dataset as an example (e.g.) Figure 2 As shown, it can be decomposed into an ordered sequence containing multiple subtasks.

[0039] Each subtask Formalized as a triple, i.e. For example, for the subtask "enter the kitchen area", there are... = "kitchen" ="Enter", =“Enter the kitchen area”; the generated subtask sequence This will serve as the basis for explicit progress tracking during subsequent navigation. It should be noted that during actual task execution, the mobile robot receives natural language instructions in English.

[0040] S3, via task routing function Determine each subtask Task types The task type Includes: area entry, passage, landmark turning, target approach, and geometric termination;

[0041] In this embodiment, based on the action intent of the subtask and target The semantics are categorized, for example: the subtask type corresponding to the action intention "walk into" is area entry, the subtask type corresponding to "pass by" is traversal, and the subtask type corresponding to "stop at the corner of" is geometric termination. Task routing results. This determines the perception link, the combination of progress status dimensions, and the completion judgment function that will be used in the subsequent subtask.

[0042] S4. At each time step t during the navigation task, acquire the first-person view image of the mobile robot. Including RGB images and depth images ;

[0043] S5. Define and update the internal state of the mobile robot at each time step t. ,in:

[0044] Use it as a status marker for discrete tasks, recording the number of the currently executing subtask;

[0045] This is a local geometric map maintained online from depth images, used to describe traversable spaces and obstacles; in this embodiment, it is based on depth images. The local map is incrementally updated using an online grid-based algorithm. This allows for the estimation of the forward passable space. And the direction of the "opening" in the detection environment;

[0046] The multi-dimensional progress state vector is obtained by updating information from three levels: semantic features, geometric features, and temporal features, quantifying the execution progress of the current subtask. In a preferred embodiment, the multi-dimensional progress state vector is specifically: ,in, For target visibility, it is used to indicate whether there is visual evidence in the current observation that is related to the target of the current subtask; The degree of target proximity is used to characterize whether the target is visible from a distance or has already approached; The degree of region matching is used to determine whether the robot is in a region consistent with the subtask objective; The degree of matching between the opening or passage direction is used to determine whether the local geometry of the current environment supports continued movement in the commanded direction; The progress of the journey is used to determine whether the robot has actually passed through or crossed the target. This represents the degree of geometric relationship matching, used to determine whether a relational objective holds true. The degree of stagnation is used to determine whether the robot is stuck in a state of contact with a wall, collision, spinning in place, or ineffective forward movement; the multidimensional progress state vector is used to quantify the progress state of the current subtask.

[0047] It is the memory information of historical actions and observations.

[0048] It should be noted that the multidimensional progress state vector This method is an internal progress representation custom-designed for step-by-step language navigation tasks in mobile robots, used to address the problem that "target visibility does not equal subtask completion." Progress status in each dimension can be updated jointly by open-vocabulary visual labels, target grounding results, dual-channel visual language verification, a local geometric map constructed from a depth map, and memory information from historical actions and observations. For example, for the "region entry" subtask, it is updated through a scene classification model. Calculated through local maps and accumulate effective displacement For the "traversing" task, dynamic calculation is used. To quantify the progress of the traverse;

[0049] Subsequently, this application designs sub-task completion conditions by combining the progress status of the above different dimensions according to the current sub-task type; to avoid the navigation system switching tasks or stopping prematurely just because it sees the target, and thus continuously updating and comprehensively judging multimodal evidence based on the completion conditions of the current sub-task.

[0050] S6. At each time step, based on the RGB image and depth image, perform multimodal perception and target weighting on the target of the currently executing sub-task to obtain multimodal observation evidence. For geometric relationship termination subtasks, after obtaining multimodal observation evidence, an additional anchor-first strategy is used to determine relationship rights.

[0051] In a preferred embodiment, step S6 specifically includes:

[0052] The RGB image acquired at the current time step t is processed using an open vocabulary foundation model (i.e., the Grounding DINO model). Process the data to generate a set of candidate targets related to the current subtask objective. where n is the total number of candidate targets;

[0053] For each candidate target Combined with the depth map obtained at the current time step t Estimate the approximate distance between it and the mobile robot. and confidence level Confidence level represents the degree of credibility that the candidate target is the correct sub-task target given the current approximate distance and spatial relationship.

[0054] The candidate target set is filtered by distance gating, category matching, spatial blacklisting, and geometric constraints to obtain the effective candidate set. ;

[0055] In this embodiment, distance gating (distance gating threshold) is first applied. The system performs an initial screening of the candidate target set; subsequently, it sequentially performs three layers of filtering: category matching, spatial blacklisting, and basic geometric constraints.

[0056] Category matching: Match the semantic labels of candidate regions with the target description of the current subtask, and eliminate semantically irrelevant candidates;

[0057] Space blacklist: Removed from historical status Regions that are consistently recorded as high-confidence false positives (such as areas with specular reflections or repetitive textures).

[0058] Basic geometric constraints: Eliminate depth value anomalies ( or Candidates whose bounding box aspect ratio exceeds the reasonable range for indoor observation;

[0059] After the above screening, a valid candidate set is obtained. ; and then from Select the candidate target with the highest confidence level as the optimal candidate target. .

[0060] like Figure 1 As shown, this application first uses GSAM2 (a fusion model of the Grounding DINO model and the SAM2 model) for candidate generation and RAM++ for memory-enhanced retrieval; after obtaining the optimal candidate target, this application performs Dual VLM dual-channel visual language verification: two heterogeneous visual language models are used to independently determine the weight of the optimal candidate target, and the visual language weight determination function is:

[0061] ;

[0062] in, This is the distance-gated threshold. and The independent weight determination results of the two heterogeneous visual language models are given if and only if the approximate distance does not exceed the distance gating threshold and both weight determination results are true. Confirmed as reliable multimodal observational evidence, subsequent mobile robots will... Subtasks are executed to target the objective, and the completion status of the subtasks is determined based on a subsequently designed completion judgment function; the reliable multimodal observation evidence includes the optimal candidate objective. Visual features (obtained from RGB images), approximate distance and semantic matching degree (referring to the semantic similarity between the optimal candidate target and the subtask target); if the optimal candidate target fails the dual-channel visual language verification, then multimodal observation evidence... If the set is empty, a semantic event will be triggered, and the slow system will intervene to perform semantic arbitration and replanning.

[0063] Furthermore, for geometric relationship termination subtasks, after obtaining multimodal observational evidence, an additional anchor-first strategy is employed for relationship determination:

[0064] Geometric relationship termination subtasks, such as "stopping in the corner of the bed," "stopping in the corner of the bar," "stopping next to the table," and "stopping near the sofa," do not involve a single, directly detectable entity as the target. Instead, the target is composed of both an "anchor target" and a "spatial relationship." The completion criteria for such tasks cannot be determined solely by whether the target word appears in the image. For example, in "corner of the bed," the truly reliably detectable entity is "bed," while "corner" represents a spatial relationship relative to "bed." If only "corner" is considered as a candidate target for detection, unstable or uninterpretable results are likely to occur.

[0065] Therefore, this method additionally designs an anchor-based relationship judgment mechanism for geometric relationship termination subtasks: after the entire "corner of the bed" is verified by dual-channel visual language as the subtask objective and reliable evidence is obtained, it is then described from the local language. Extract anchor point objects Then, it checks whether the anchor object exists and whether the spatial relationship is satisfied, i.e., it sets the relationship confirmation determination function. for:

[0066] ;

[0067] in, This indicates the conditions for confirming the anchor point target. Indicates spatial relationship matching conditions. , For indicator functions, An anchor object identification function, when the anchor object If its existence can be detected, output a true value of 1; otherwise, output a false value of 0. This is a spatial relation matching function. This is the threshold for matching spatial relationships.

[0068] For single-anchor relationships, only one anchor object needs to be extracted; for multi-anchor relationships, multiple anchor objects are extracted, and it is further determined whether the mobile robot is located in the passage area or relationship area between multiple anchor objects. When the relationship confirmation function outputs a true value of 1, the geometric relationship termination subtask executes the subtask with the anchor object (the optimal candidate target includes the anchor object) as the target. Through the anchor priority strategy and relationship confirmation, the navigation system avoids separating "entity target confirmation" and "spatial relationship matching," and avoids directly using abstract relational terms as detection targets, thereby improving the stability and interpretability of judgments in relational stopping tasks.

[0069] In this embodiment, the anchor object identification function As a preferred implementation method, it can be further defined by integrating multiple types of evidence as follows: ,in:

[0070] ;

[0071] in, This represents the anchor point target score after integrating multiple types of evidence. This represents the semantic matching score of the open vocabulary labeling model for the anchor target; This represents the confidence level of the open vocabulary detection or segmentation model in detecting the anchor target; This represents the visual language model's confirmation score regarding whether the anchor target truly exists. This indicates whether the anchor point target has remained consistently observed throughout historical observations; These are the weighting coefficients; Confirm the threshold for the anchor point.

[0072] After the anchor point target is confirmed (i.e.) (Taking truth value 1), the navigation system uses a spatial relationship matching function to determine whether the current position of the mobile robot and the anchor point object satisfy the spatial relationship described in the local language. Spatial relationship matching function A larger value indicates that the current state better matches the spatial relationships in the local description, and for different relational terms, The focus of judgment differs, for example:

[0073] corner of The robot is required to approach the corner area of ​​the anchor point target, rather than simply facing the front of the anchor point target or seeing the anchor point target from a distance.

[0074] side of The robot must be located in the lateral region of the anchor point target.

[0075] Near The robot must be close enough to the anchor point target.

[0076] Between and The robot must be located in a passable space or visual relationship area between two anchor point targets.

[0077] In a preferred embodiment, the spatial relationship matching function can be obtained by combining visual language relationship judgment, local geometric cues, target proximity, and continuous confirmation results, as expressed by the formula:

[0078] ;

[0079] in, This represents the score of the visual language model's judgment on whether the current viewpoint satisfies the relational description. It represents geometric relationship clues obtained from depth maps or local geometry maps; This indicates the degree of proximity between the robot and the anchor point target; This indicates whether the relationship is stable across multiple consecutive time steps; These are the weighting coefficients.

[0080] S7. At each time step, based on the current subtask type... Call the corresponding completion determination function Determine if the current subtask is complete; if If the condition is met, the task is considered complete; otherwise, it is considered incomplete. The completion determination function takes the internal state and multimodal observation evidence as input. The completion determination functions of different subtasks adopt different combinations of semantic features, geometric features, temporal features and multimodal observation evidence in the internal state.

[0081] In a preferred embodiment, the completion determination function is specifically as follows:

[0082] For region entry subtasks, remember... The completion decision function is:

[0083] ;

[0084] in, For indicator functions, These are the thresholds for region matching, effective displacement, and opening matching, respectively. The cumulative effective displacement since entering the candidate state is calculated; the decision function ensures that "entry" must simultaneously satisfy semantic, displacement, and geometric passage conditions; it should be noted that in this application, effective displacement refers to the net translation distance of the robot on the horizontal plane, that is, the change in the robot's position coordinates in the world coordinate system. This application excludes the odometer accumulation when only rotation in place occurs, and only calculates the actual distance of the robot's center of mass moving forward, backward, or laterally, so as to accurately reflect whether the robot is in a "stagnant" or "micro-motion" state, thereby triggering recovery control.

[0085] For the sub-task of passing through, remember... The completion decision function is:

[0086] ;

[0087] in, These are the thresholds for proximity and traversal progress, respectively. Indicates whether a subsequent target or new area clue has appeared; the completion decision function forces the mobile robot to "pass through" the target and observe the changes in the environment before and after.

[0088] For landmark turning sub-tasks, remember... The completion decision function is:

[0089] ;

[0090] in, This represents the target landmark object corresponding to the current subtask; The relative angle between the robot's current orientation and the target landmark; The allowed steering angle threshold; To confirm the number of steps to a stable transition, i.e., at least It can be observed at every consecutive time step. and This is to ensure that the steering maneuver is completed and that drifting ceases; A counter representing memory information, containing memory information about historical actions and observations. Each record and The count increases by 1.

[0091] For subtasks similar to the target class, let _ The completion decision function is:

[0092] ;

[0093] in, To stop the distance threshold, To get closer to the evidence The required number of stability confirmation steps, i.e., at least It can be observed at every consecutive time step. and The decision function passes the multi-step stability requirement, avoiding "jitter stop" caused by jitter in single-frame observation.

[0094] For geometric relation termination subtasks, let The completion decision function is:

[0095] ;

[0096] in, The threshold for the number of stable steps required to confirm a relationship requires at least... It can be observed at every consecutive time step. and ( In actual calculations, the following can be taken: The value); the completion decision function embodies the two-step right determination idea of ​​"anchor point first, then relationship", if and only if the anchor point target confirmation condition. Matching conditions for spatial relationships Only when the condition is met simultaneously and continues for multiple frames will the navigation system consider the geometric relationship termination subtask to have the stopping condition, in order to reduce the risk of erroneous stopping due to false detection of open vocabulary, misjudgment of visual language model in a single frame, or occlusion.

[0097] Due to the misjudgment issue that "target visibility does not equate to subtask completion," this application establishes a reliable evidence admission mechanism for the mobile robot's real-time observation results through step S6 above. Only after reliable multimodal observation evidence is formed does the system select the corresponding semantic, geometric, and temporal progress states for completion determination based on the current subtask type. The discrete task state is only advanced when the completion conditions of corresponding physical events such as area entry, traversal, landmark turning, target approach, or geometric relationship termination are met; if the completion conditions are not met, the current subtask continues execution. Through this processing, "target visibility" is distinguished from "the physical event required by the language instruction has been completed," reducing the risk of unreliable observations contaminating the task state and premature termination before physical behavior is completed.

[0098] S8. Update the discrete task status flags based on the judgment result:

[0099] ;

[0100] If the discrete task state is marked at time step t+1 and If K is the total number of subtasks, then proceed to step S9; otherwise, proceed to step S10.

[0101] S9, Output Stop Action The navigation task has ended;

[0102] S10. A hierarchical decision-making architecture based on fast and slow dual-system collaboration: the fast system executes local action control frame by frame, while the slow system only intervenes to perform semantic arbitration when a semantic event is triggered, outputting navigation actions according to a preset priority. At the same time, a timer interlocking mechanism is used in the decision-making process; after each navigation action is output, the system returns to step S4 to continue execution.

[0103] In a preferred embodiment, step S10 specifically comprises:

[0104] Based on a hierarchical decision-making architecture that coordinates fast and slow systems, navigation actions are output according to a preset priority strategy. ,

[0105] ;

[0106] This application first determines the execution strategy based on a preset priority. Then, under the current strategy, the navigation system can adaptively output the navigation action instructions to be executed based on the task type, internal state, multimodal observation evidence, etc.

[0107] In the above formula, To restore control strategy, To restore the trigger condition, the formula is expressed as:

[0108] ;

[0109] in, For indicator functions, For forward traversable space estimation, For the most recent Effective displacement at each time step This refers to the number of recent turns. These are the forward passable space threshold, effective displacement threshold, and turning number threshold, respectively; if and only if the recovery control trigger condition is met. Execute recovery control strategy at time (e.g., scanning, backing away).

[0110] This refers to a task handover strategy; typically an update. This may be accompanied by confirmatory actions; the confirmatory actions refer to updating the system's internal state queue. After the slow system intervenes and handles the anomaly (such as target conflict or jamming), the slow system will issue a corrected instruction. Before the fast system executes the specific task handover action, it must first confirm and synchronize this new state to ensure that the decisions of the upper and lower layers are consistent.

[0111] For a slow system, a semantic supervision strategy is adopted if and only if there are semantic events. Execute when activated The slow system intervenes and performs semantic arbitration; after the semantic arbitration is completed, the slow system outputs the corrected subtask instructions or updates the hidden state, and triggers the task handover strategy. Update discrete task status flags and progress status This completes the state synchronization between the fast and slow systems; the slow system does not directly output low-level action commands.

[0112] In this embodiment, the semantic event Including: Conflicts in the confirmation of ownership of targets Long-term stagnation Uncertainty in regional matching Arbitration for termination of relationship And task handover failure Semantic events are triggered under abnormal circumstances such as conflicting multimodal observation evidence, insufficient evidence, uncertain region matching, prolonged robot stagnation, or task handover failure. Specifically:

[0113] When the candidate target output by the open vocabulary foundation model semantically contradicts the target described by the current subtask instruction, a target ownership conflict is triggered. ;

[0114] When a mobile robot, within multiple consecutive time steps, experiences a very small effective translational distance despite having passable space ahead (i.e., it is in a state of "circling in place" or "not moving forward at all"), it triggers a prolonged stagnation. ;

[0115] When a mobile robot observes regional features, but the confidence level of matching these visual features with the pre-established semantic map region remains at a low, fluctuating, or uncertain level for an extended period, causing the system to be unable to confirm whether it has truly entered the target region, regional matching becomes uncertain. ;

[0116] When the navigation system needs to confirm the validity of complex spatial relationships through a slower system (such as when relationship rights confirmation is required), relationship termination arbitration is triggered. ;

[0117] When a subtask is determined to be completed, and the navigation system attempts to switch to the next subtask, a task handover failure is triggered if the new observation environment or target cannot be effectively confirmed. ;

[0118] When at least one semantic event is triggered, the slow system's semantic supervision strategy is executed: the slow system performs semantic arbitration based on the anomaly type corresponding to the triggered semantic event and outputs specific correction instructions; for example, when a "target rights conflict" is detected, the slow system outputs a target correction suggestion or a subtask replanning result; when a "long-term stagnation" is detected, the slow system outputs a recovery intention or anomaly type indication. After semantic arbitration is completed, the system updates the task status, progress status, or recovery intention, while the specific underlying navigation actions are still generated and executed by the local controller or recovery controller in the fast system.

[0119] In addition, this application adds a timer interlock mechanism: when the hierarchical decision architecture is in the recovery control triggered state, i.e. At this time, the trigger timer of the slow system is frozen:

[0120] ;

[0121] in, , This indicates the trigger timing of the slow system at time step t+1, t. After the recovery control ends, the trigger timer of the slow system resumes normal updates. This mechanism ensures the continuity and stability of high-priority physical control processes such as local escape from difficulties, without being interrupted by time-consuming semantic reasoning involving the slow system. For the slow system, it also avoids the mobile robot being frequently and accidentally awakened due to instantaneous environmental changes when dealing with physical jitter or fine-tuning. Only when the recovery action is completely finished and the timer returns to normal will the slow system re-enter the standby state, thereby greatly reducing the probability of the slow system being accidentally activated and ensuring the stability of the overall system.

[0122] This is a regular rule-based action strategy based on the current task type; that is, in all other cases, the fast system determines the action based on the current task type. Perform routine actions (e.g., move toward the target, explore along the wall).

[0123] This concludes the description of the entire implementation process of the method proposed in this invention. Furthermore, this application also discloses a structured evidence perception and control system based on continuous environment visual language navigation. This system is used to execute the method described above, and specifically includes:

[0124] The instruction parsing and routing module is used to decompose the input natural language instructions into an ordered sequence of subtasks and determine the type of each subtask;

[0125] The multimodal perception and rights confirmation module is used to acquire first-view RGB images and depth images of the mobile robot, and obtain multimodal observation evidence through open vocabulary base model, candidate target screening and dual-channel visual language verification; and additionally performs relation rights confirmation for geometric relation termination subtasks.

[0126] The status management and progress tracking module is used to build and update the progress status of the mobile robot task, maintain the local geometric map, and call the corresponding completion judgment function according to the current subtask type to determine whether the subtask is completed and update the task status; it is also equipped with a feedback buffer to store the memory information of historical actions and observations, providing historical context for decision-making.

[0127] The fast and slow dual-system collaborative decision-making module is used to execute local action control frame by frame through the fast system, while the slow system only intervenes to perform semantic arbitration when a semantic event is triggered, and outputs navigation actions according to the preset priority.

[0128] The action execution interface is used to convert the navigation actions into low-level control commands for the mobile robot and output them.

[0129] In this embodiment, to systematically and comprehensively evaluate the effectiveness of the proposed SEAL-Nav, performance verification was also conducted. Closed-loop evaluation and comparative analysis were performed on the val-unseen set of the authoritative continuous environment visual language navigation benchmark R2R-CE. This partition includes multiple indoor scenes that have never appeared during the training phase, which can objectively test the agent's zero-shot generalization navigation capability in unknown environments.

[0130] The experimental platform used Habitat Simulator v2.0, and a continuous 3D environment was constructed using the Matterport3D indoor scene dataset. At each time step, the agent could only receive first-person view RGB images, depth images, and natural language commands. The action space was discrete: {move forward 0.25 meters, turn left 10 degrees, turn right 10 degrees, stop}. The success criterion for the navigation task was that the agent actively executed the "stop" action within 3 meters of the target location. A maximum of 1000 steps was set to prevent infinite exploration.

[0131] To accurately measure the navigation performance of SEAL-Nav, we compared it with several existing visual-language navigation methods. The complete results of the comparative experiments are recorded in Table 1. The table below lists the key zero-shot methods and the results of our invention:

[0132] Table 1 Comparison of navigation performance on val-unseen for R2R-CE

[0133] As shown in the table above, among zero-shot methods, the proposed SEAL-Nav achieves the best overall performance. Its task success rate (SR) reaches 33.0%, and its path length-weighted success rate (SPL) reaches 27.39%, both the highest among the compared methods. Compared to Open-Nav, SR and SPL are significantly improved by 14.0 and 11.29 percentage points, respectively. Compared to SmartWay, SEAL-Nav leads by 4.0 percentage points in SR and 4.93 percentage points in SPL, and has a shorter average trajectory length (TL = 10.44 meters), indicating that this method improves both success rate and path exploration efficiency. Furthermore, SEAL-Nav's normalized dynamic time warping score (nDTW) reaches 56.09, also outperforming previously reported zero-shot results, demonstrating a higher spatiotemporal consistency between the actual motion trajectory generated by the agent and the human demonstration or ideal path. In addition, NE (Navigation Error) is used to measure the shortest geographical distance between the agent's termination position and the target position. The lower the value, the more accurate the endpoint positioning. OSR (Oracle Success Rate) represents the proportion of tasks that enter the target success radius at any time. The higher the value, the stronger the ability of the trajectory to cover the target area. As can be seen from the table, the method proposed in this invention also has better performance in NE and OSR compared to most existing technologies.

[0134] As shown in Table 1, there is still a significant gap between the performance of SEAL-Nav and the best supervised learning method (such as BEVBert, SR=60%). This clearly demonstrates that, under zero-shot conditions where no task-specific data is used for training, navigation systems still face severe challenges in understanding complex long commands, long-range planning, and accurate geometric relationship judgment. It also illustrates the value of this invention—significantly narrowing the performance gap with supervised learning methods under zero-shot conditions.

[0135] Furthermore, to quantitatively verify the contributions of the three core mechanisms of this invention (structured state, multimodal evidence confirmation, and fast-slow dual-system coordination), we designed rigorous ablation experiments. Three ablation configurations were set up: w / o State (removing the structured state mechanism), w / o Evidence (removing multimodal objective confirmation), and w / o Scheduling (removing fast-slow system scheduling). The results of the ablation experiments are presented in bar chart format. Figure 3 The specific data and analysis are shown in Tables 2 and 3 below:

[0136] Table 2 Ablation Configuration Instructions

[0137] Table 3 Comparison of ablation test results (R2R-CE val-unseen)

[0138] The core role of the structured state mechanism: When this mechanism is removed (without State) and only a uniform target visibility is used to determine the completion of all subtasks, the performance degradation is most drastic: SR plummets from 33.0% to 22.0%, and SPL drops from 27.39% to 18.35%. This strongly demonstrates that designing differentiated completion determination functions for subtasks with different physical meanings is fundamental to improving navigation success rates. Without this mechanism, the system cannot distinguish between "entering" and "seeing," leading to numerous premature or erroneous subtask switching.

[0139] The robustness contribution of the multimodal evidence ownership mechanism: After removing the evidence ownership module (without Evidence), SR decreased to 25.0% and SPL decreased to 20.62%. This verifies the necessity of cleaning and ownership confirmation of observational evidence in real-world scenarios with high open-vocabulary perception noise. Without this mechanism, the system is susceptible to false target interference, leading to erroneous stops or approaching incorrect targets, thus increasing navigation errors.

[0140] The efficiency optimization value of the fast and slow dual-system scheduling mechanism: After removing the system scheduling mechanism (without Scheduling), SR decreased to 27.0%, but SPL significantly decreased to 19.84%, while the average trajectory length (TL) surged from 10.44 meters to 12.16 meters. This indicates that the module primarily optimizes the system's execution efficiency and trajectory quality. Without priority scheduling and interlocking mechanisms, high-level semantic reasoning may inappropriately interrupt low-level recovery control. Furthermore, high-frequency, unnecessary high-level semantic intervention leads to inefficient agent processing when locally blocked, resulting in longer, more circuitous additional paths.

[0141] In addition to performance verification and ablation experiments, this embodiment also includes operational efficiency and qualitative case studies to support its findings.

[0142] The operational efficiency statistics are shown in Table 4 below:

[0143] Table 4. Performance and Model Call Statistics on R2R-CE val-unseen

[0144] Data shows that in a navigation task with an average length of 362.54 steps, the high-level brain (slow system) is invoked only 151.17 times on average, accounting for 41.7% of the total steps. This directly confirms the success of the "fast and slow dual-system" architecture: decoupling the high-frequency, low-level perception, obstacle avoidance, and recovery control (fast system) from the low-frequency, high-level semantic understanding and arbitration (slow system). The large model (slow system) is only triggered at key decision nodes such as task decomposition, semantic conflict, and state transition, rather than being invoked frame by frame, thus achieving efficient control while ensuring intelligence.

[0145] In summary, the structured evidence perception control method based on continuous environment visual language navigation proposed in this invention fundamentally solves the misjudgment problem of "seeing is completing" by designing differentiated physical completion judgment functions based on multidimensional progress states for different types of sub-tasks through a structured state mechanism, and achieves precise switching of task states. The multimodal evidence confirmation mechanism uses distance gating, dual-channel visual language verification, and anchor point priority strategy to perform hierarchical filtering and cleaning of high-noise observations generated by open vocabulary perception, providing highly reliable perception evidence for navigation decisions and significantly reducing erroneous stops and path deviations caused by perceptual illusions. The fast and slow dual-system collaborative mechanism decouples the high-frequency, robust low-level motion control (fast system) from the low-frequency, highly intelligent semantic reasoning supervision (slow system) and adopts priority scheduling and timer freeze interlocking strategies to ensure that the stability of local recovery control is not interrupted by high-level reasoning, reduces the frequency of unnecessary high-level semantic intervention and the additional path length caused by it, and ensures the real-time performance, efficiency, and overall stability of control while utilizing the intelligence of large language models.

[0146] The organic synergy of the three design points in this invention constitutes a complete control closed loop for achieving highly reliable and efficient zero-sample navigation in continuous and unknown environments; it has significant effectiveness and technical advantages in improving the success rate, path efficiency, trajectory quality and system robustness of visual language navigation in continuous environments.

[0147] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0148] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0149] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A structured evidence perception and control method based on continuous environmental visual language navigation, characterized in that, Specifically, the following steps are included: S1. Receive natural language instructions for instructing the mobile robot to perform navigation tasks, and preprocess the natural language instructions; S2. Decompose the preprocessed natural language instructions into an ordered sequence of subtasks. The task description of each subtask includes: target, action intention, and local language description. S3. Determine the task type of each subtask through the task routing function. The task types include: area entry, traversal, landmark turning, target approach, and geometric relationship termination. S4. At each time step t during the navigation task, acquire images from the first-person perspective of the mobile robot, including RGB images and depth images; S5. Define and update the internal state of the mobile robot at each time step t. ,in: For discrete task status marking; A local geometric map maintained online from depth images; It is a multidimensional progress state vector, which is updated by information from three levels: semantic features, geometric features and temporal features, to quantify the execution progress of the current subtask; For the memory information of historical actions and observations; S6. At each time step, based on the RGB image and depth image, perform multimodal perception and target weighting on the target of the currently executing sub-task to obtain multimodal observation evidence. For geometric relationship termination subtasks, after obtaining multimodal observation evidence, an additional anchor-first strategy is used to determine relationship rights. S7. At each time step, based on the current subtask type... Call the corresponding completion determination function Determine whether the current subtask has been completed; if If the condition is met, the task is considered complete; otherwise, it is considered incomplete. The completion determination function takes the internal state and multimodal observation evidence as input. The completion determination functions of different subtasks adopt different combinations of semantic features, geometric features, temporal features and multimodal observation evidence in the internal state. S8. Update the discrete task status flags based on the judgment result: ； If the discrete task state is marked at time step t+1 and If K is the total number of subtasks, then proceed to step S9; otherwise, proceed to step S10. S9, Output Stop Action The navigation task has ended; S10. A hierarchical decision-making architecture based on fast and slow dual-system collaboration: the fast system executes local action control frame by frame, while the slow system only intervenes to perform semantic arbitration when a semantic event is triggered, outputting navigation actions according to a preset priority. At the same time, a timer interlocking mechanism is used in the decision-making process; after each navigation action is output, the system returns to step S4 to continue execution.

2. The structured evidence perception and control method based on continuous environment visual language navigation according to claim 1, characterized in that, The multidimensional progress state vector is specifically: ； in, For target visibility, For the degree of proximity to the target, For the degree of regional matching, The degree of matching between the opening or the direction of passage. For the progress of the journey, For the degree of geometric relationship matching, The degree of stagnation.

3. The structured evidence perception and control method based on continuous environment visual language navigation according to claim 1, characterized in that, Step S6 specifically includes: The RGB image obtained at the current time step t is processed using an open vocabulary foundation model. Process the data to generate a set of candidate targets related to the current subtask objective. where n is the total number of candidate targets; For each candidate target Combined with the depth map obtained at the current time step t Estimate the approximate distance between it and the mobile robot. and confidence level ; The candidate target set is filtered by distance gating, category matching, spatial blacklisting, and geometric constraints to obtain the effective candidate set. ; Then, select the candidate target with the highest confidence from the effective candidate set as the optimal candidate target. ; Dual-channel visual language verification: Two heterogeneous visual language models are used to independently determine the weight of the optimal candidate target. The visual language weight determination function is as follows: ； in, This is the distance-gated threshold. and The independent weight determination results of the two heterogeneous visual language models are given if and only if the approximate distance does not exceed the distance gating threshold and both weight determination results are true. The evidence was confirmed as reliable multimodal observational evidence; the reliable multimodal observational evidence included the optimal candidate target. Visual features, approximate distance and linguistic semantic matching degree; if the optimal candidate target fails the dual-channel visual language verification, then multimodal observation evidence... It is an empty set.

4. The structured evidence perception and control method based on continuous environment visual language navigation according to claim 1, characterized in that, For the geometric relation termination subtask, after obtaining multimodal observation evidence, the additional anchor-first strategy for relation determination is as follows: From local language description Extract anchor objects Set up a relationship ownership determination function for: ； in, This indicates the conditions for confirming the anchor point target. Indicates spatial relationship matching conditions. , For indicator functions, An anchor object identification function, when the anchor object If its existence can be detected, output a true value of 1; otherwise, output a false value of 0. This is a spatial relation matching function. This is the threshold for matching spatial relationships.

5. The structured evidence perception and control method based on continuous environment visual language navigation according to claim 2, characterized in that, In step S7, the completion determination function is specifically as follows: For region entry subtasks, remember... The completion decision function is: ； in, For indicator functions, These are the thresholds for region matching, effective displacement, and opening matching, respectively. This represents the cumulative effective displacement since entering the candidate state; For the sub-task of passing through, remember... The completion decision function is: ； in, These are the thresholds for proximity and traversal progress, respectively. Indicates whether there are subsequent targets or clues to new areas; For landmark turning sub-tasks, remember... The completion decision function is: ； in, This represents the target landmark object corresponding to the current subtask; The relative angle between the robot's current orientation and the target landmark; The allowed steering angle threshold; Number of steps to confirm steering stability; For subtasks similar to the target class, let's denote... The completion decision function is: ； in, To stop the distance threshold, To get closer to the evidence Required number of stability confirmation steps; For geometric relation termination subtasks, let The completion decision function is: ； in, The number of stable confirmation steps required to confirm the relationship.

6. The structured evidence perception and control method based on continuous environment visual language navigation according to claim 1, characterized in that, In step S10, the step of outputting navigation actions according to a preset priority specifically includes: Based on a hierarchical decision-making architecture that coordinates fast and slow systems, navigation actions are output according to a preset priority strategy. , ； in, To restore the control policy, the control restoration trigger condition must be met. Execute at any time; For task handover strategy; For a slow system, a semantic supervision strategy is adopted if and only if there are semantic events. Executes when triggered; This is a regular rule-based action strategy based on the current task type.

7. The structured evidence perception and control method based on continuous environment visual language navigation according to claim 6, characterized in that, The specific conditions for triggering the recovery control are as follows: ； in, For indicator functions, For forward traversable space estimation, For the most recent The effective displacement of the mobile robot within a time step For the most recent The number of times the mobile robot turns within a time step. These are the forward passable space threshold, the effective displacement threshold, and the number of turns threshold, respectively.

8. The structured evidence perception and control method based on continuous environment visual language navigation according to claim 1, characterized in that, The slow system only intervenes to perform semantic arbitration when a semantic event is triggered, specifically as follows: Semantic events are triggered under abnormal circumstances such as conflicting multimodal observation evidence, insufficient multimodal observation evidence, uncertain region matching, prolonged robot stagnation, or failed task handover. Including: Conflicts in the confirmation of ownership of targets Long-term stagnation Uncertainty in regional matching Arbitration for termination of relationship And task handover failure When at least one semantic event is triggered, the slow system semantic supervision strategy is executed: the slow system performs semantic arbitration based on the exception type corresponding to the different semantic events triggered, and outputs correction instructions.

9. A structured evidence perception and control method based on continuous environment visual language navigation according to claim 6, characterized in that, The timer interlock mechanism is specifically as follows: When the hierarchical decision-making architecture is in the recovery control triggered state, i.e. At this time, the trigger timer of the slow system is frozen: ； in, , This indicates the trigger timer of the slow system at time steps t+1 and t; after the control recovery ends, the trigger timer of the slow system resumes normal updates.

10. A structured evidence perception and control system based on continuous environmental visual language navigation, the system being used to execute the method as described in any one of claims 1-9, characterized in that, Specifically, it includes: The instruction parsing and routing module is used to decompose the input natural language instructions into an ordered sequence of subtasks and determine the type of each subtask; The multimodal perception and rights confirmation module is used to acquire first-view RGB images and depth images of the mobile robot, and obtain multimodal observation evidence through open vocabulary base model, candidate target screening and dual-channel visual language verification; and additionally performs relation rights confirmation for geometric relation termination subtasks. The status management and progress tracking module is used to build and update the progress status of the mobile robot task, maintain the local geometric map, and call the corresponding completion judgment function according to the current subtask type to determine whether the subtask is completed and update the task status; it is also equipped with a feedback buffer to store the memory information of historical actions and observations, providing historical context for decision-making. The fast and slow dual-system collaborative decision-making module is used to execute local action control frame by frame through the fast system, while the slow system only intervenes to perform semantic arbitration when a semantic event is triggered, and outputs navigation actions according to the preset priority. The action execution interface is used to convert the navigation actions into low-level control commands for the mobile robot and output them.

Images