Visual language navigation method and system based on cross-task incremental semantic memory graph

Abstract

The invention belongs to the field of artificial intelligence and robot navigation, and discloses a visual language navigation method and system based on a cross-task incremental semantic memory graph, and the method comprises the steps that an intelligent agent executes a zero-sample visual language navigation task in a continuous environment; performing cross-modal alignment on the natural language instruction and environment observation based on a multi-modal large language model, selecting candidate waypoints and updating task progress; a semantic memory graph is constructed and dynamically updated, wherein the semantic memory graph is used for structured storage and cross-task multiplexing of scene semantic information and a spatial topological relation sensed by an intelligent agent in historical tasks; and performing global path planning and local dynamic fine tuning based on the semantic memory graph. According to the method, the problem of task-by-task forgetting in a traditional method is solved, environment understanding and task reasoning capabilities are improved by constructing structured long-term memory, and navigation precision and robustness are optimized through a global-local collaborative strategy.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and robot navigation, in particular to a vision-and-language navigation method and system based on a cross-task incremental semantic memory graph. BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art.

[0003] Vision-and-Language Navigation (VLN) is a core task of embodied intelligence, which requires an agent to perceive, reason and perform actions in a complex environment according to natural language instructions. This task integrates language understanding, visual perception and action decision-making, and has broad prospects in service robots, indoor navigation systems, human-computer interaction and other scenarios.

[0004] However, existing vision-and-language navigation methods still have significant limitations: 1. Task-by-task forgetting problem: Existing methods usually only maintain short-term memory within a single task (episode), and the memory is cleared after the task is completed. This results in the agent being unable to accumulate and reuse historical experience even when performing the same task repeatedly in the same environment, severely restricting its adaptability and efficiency in real-world scenarios.

[0005] 2. Unstructured memory and weak reasoning ability: Some studies attempt to retain historical information, but are mostly limited to unstructured trajectory caching or rely on external Oracle information. Such memory lacks structured abstraction of the environment and semantics, making it difficult to support efficient semantic retrieval and complex relationship reasoning.

[0006] 3. Imbalance between global and local navigation: Long instruction navigation requires the agent to balance global task consistency and local target positioning. Existing methods often focus on one or the other: greedy strategies that rely on local perception tend to deviate from the global goal, while relying solely on global planning lacks flexibility in final positioning, making it difficult to achieve high-precision navigation.

[0007] With the development of large language models and multi-modal large models, zero-shot vision-and-language navigation has become possible, which avoids expensive supervised training and enhances generalization ability. However, how to introduce continuous learning and memory mechanisms into these zero-shot methods and effectively coordinate global and local navigation strategies remains a technical challenge. SUMMARY

[0008] To address the aforementioned issues, this invention proposes a visual language navigation method and system based on cross-task incremental semantic memory graphs. This method solves the "task-by-task forgetting" problem inherent in traditional methods, enhances environmental understanding and task reasoning capabilities by constructing structured long-term memory, and optimizes navigation accuracy and robustness through a global-local collaborative strategy.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a visual language navigation method based on cross-task incremental semantic memory graphs, comprising the following steps: The S1 agent performs zero-shot visual-language navigation tasks in a continuous environment. It performs cross-modal alignment of natural language commands and environmental observations based on a multimodal large language model, selects candidate waypoints, and updates the task progress. During navigation, S2 constructs and dynamically updates a cross-task incremental semantic memory graph. The semantic memory graph is used to structure and reuse the scene semantic information and spatial topological relationships perceived by the agent in historical tasks across tasks. S3 After the semantic memory map accumulates environmental knowledge, a global and local collaborative navigation strategy is adopted. At the global level, semantically guided path planning is performed based on the semantic memory map, and at the local level, fine-grained positioning of the target location is performed based on real-time perception.

[0010] As an alternative implementation, the cross-task incremental semantic memory graph includes a set of nodes, a set of edges, a set of semantic attributes, and a set of common sense co-occurrences. Nodes represent locations visited by the agent, edges represent reachability relationships between locations, semantic attributes represent room types and object sets corresponding to locations, and common sense co-occurrences represent potential co-occurring objects obtained based on common sense reasoning.

[0011] As an alternative implementation, a multi-threshold-based update rule is used to dynamically update the cross-task incremental semantic memory graph, specifically: When an agent selects an optimal waypoint, it inserts this waypoint as a new node into the graph and establishes a reachable edge between the new node and the current node. When the distance between a new node and an existing node is less than the correction threshold, and they have the same room type, the new node completely covers the original node; When the distance between two nodes is less than the merging threshold and they are semantically identical, the two nodes are merged. If the distance between a new node and an existing node is less than the reachability threshold, a reachability edge is established between them. Among them, the merging threshold is greater than the correction threshold, and the achievable threshold is adaptively adjusted according to environmental information.

[0012] As an alternative implementation method, semantically guided path planning is performed at the global level based on semantic memory graphs. Specifically, natural language instructions are semantically parsed and decomposed into several sub-tasks. Each sub-task corresponds to a target semantic set. Based on the semantic memory graph, a global reference path from the starting point to the target node is generated by integrating the spatial location, semantic attributes, and common sense co-occurrence information of the nodes.

[0013] As an alternative implementation, during the execution of the global reference path, candidate waypoints at each step are selected based on a comprehensive score of directional consistency and semantic consistency, wherein: The directional consistency score measures the consistency between the candidate point's movement direction and the global path direction; Semantic consistency score measures the similarity between the semantics of candidate points and the semantics of the current subtask target.

[0014] As an alternative implementation, fine-grained local positioning is triggered when the agent reaches the vicinity of the global path endpoint, including: Generate a set of local candidate waypoints around the current location; The visual observation information corresponding to each local candidate waypoint is input together with the navigation instructions into the multimodal large language model; The candidate points that are most conducive to approaching the final target are evaluated and selected by a multimodal large language model for accurate localization.

[0015] Secondly, the present invention provides a visual language navigation system based on a cross-task incremental semantic memory graph, comprising: The visual language navigation module is configured to: enable the agent to perform zero-shot visual language navigation tasks in a continuous environment, perform cross-modal alignment of natural language commands and environmental observations based on a multimodal large language model, select candidate waypoints and update task progress; The semantic memory graph module is configured to: construct and dynamically update a cross-task incremental semantic memory graph during navigation, wherein the semantic memory graph is used to structure and reuse the scene semantic information and spatial topological relationships perceived by the agent in historical tasks across tasks; The collaborative navigation module is configured to: after the semantic memory map has accumulated environmental knowledge, adopt a global and local collaborative navigation strategy, perform semantically guided path planning based on the semantic memory map at the global level, and perform fine-grained positioning of the target location based on real-time perception at the local level.

[0016] Thirdly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0017] Fourthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0018] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0019] Compared with the prior art, the beneficial effects of the present invention are as follows: The visual language navigation method based on cross-task incremental semantic memory graph of the present invention, by constructing cross-task incremental semantic memory graph (CISMG), enables the intelligent agent to continuously accumulate and reuse environmental knowledge like a human, solves the problem of task-by-task forgetting, and significantly improves navigation efficiency and success rate in repetitive environments.

[0020] The visual language navigation method based on cross-task incremental semantic memory graph of the present invention integrates memory graphs of topological structure and semantic attributes, combined with common sense co-occurrence reasoning, to provide intelligent agents with powerful structured knowledge support, enabling them to perform effective reasoning and searching even when the target is not directly visible, thereby enhancing environmental understanding and reasoning capabilities.

[0021] The visual language navigation method based on cross-task incremental semantic memory graph of the present invention improves navigation accuracy and robustness. The global-local collaborative navigation strategy takes into account both the consistency of macro-task direction and the accuracy of micro-target positioning, effectively overcoming the deviation and error problems that are prone to occur in long command navigation.

[0022] The visual language navigation method based on cross-task incremental semantic memory graph of the present invention maintains the advantage of zero-shot generalization. The present invention is based entirely on zero-shot learning and does not require training for specific environments, which reduces deployment costs and ensures good generalization ability of the method in different scenarios.

[0023] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0024] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0025] Figure 1 This is a framework diagram of the visual language navigation method based on cross-task incremental semantic memory graph of the present invention; Figure 2 A schematic diagram of the cross-task incremental semantic memory graph construction and update process; Figure 3 Top-view visualization of global-local cooperative navigation for intelligent agents; Figure 4 Visualization of a visual language navigation method based on cross-task incremental semantic memory graphs in a real-world environment; Figure 5 This diagram illustrates the impact of merging rules on the number and redundancy of nodes in a cross-task incremental semantic memory graph. Detailed Implementation

[0026] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed description is exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0028] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments of the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but includes other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0029] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0030] Example 1 like Figures 1 to 5 As shown, this embodiment provides a visual language navigation method based on cross-task incremental semantic memory graphs, including the following steps: The S1 agent performs zero-shot visual-language navigation tasks in a continuous environment. It performs cross-modal alignment of natural language commands and environmental observations based on a multimodal large language model, selects candidate waypoints, and updates the task progress. During navigation, S2 constructs and dynamically updates a cross-task incremental semantic memory graph. The semantic memory graph is used to structure and reuse the scene semantic information and spatial topological relationships perceived by the agent in historical tasks across tasks. S3 After the semantic memory map accumulates environmental knowledge, a global and local collaborative navigation strategy is adopted. At the global level, semantically guided path planning is performed based on the semantic memory map, and at the local level, fine-grained positioning of the target location is performed based on real-time perception.

[0031] The specific solution of the present invention is as follows: The method framework structure proposed in this invention is as follows: Figure 1 As shown, this framework abandons the traditional short-term memory approach of task-by-task forgetting and instead achieves an evolutionary process from zero-shot reasoning to long-term memory and then to global-local collaboration through autonomous perception-driven semantic accumulation and hierarchical decision-making. The proposed method comprises three core components: First, zero-shot visual language navigation provides the perception-reasoning entry point. The agent utilizes a multimodal large language model to align natural language instructions with environmental observations across modalities, thereby selecting candidate waypoints and modeling task progress. Second, the construction and updating of the cross-task incremental semantic memory graph plays a role in long-term cognitive representation. The cross-task incremental semantic memory graph (CISMG) abstracts panoramic observations into nodes with semantic attributes and common-sense co-occurrence, forming a topological structure through reachability relationships; to avoid redundancy and respond promptly to changes in objects in the environment, a dynamic update mechanism is designed, allowing memory to gradually accumulate and optimize across tasks. Finally, the global and local collaborative navigation strategy enables the agent to achieve hierarchical planning with the support of the semantic memory graph. The global layer relies on CISMG to decompose subtasks and reason about paths, ensuring that the direction and semantics are consistent; the local layer performs fine-grained localization based on real-time perception when the target is near.

[0032] 1. Zero-sample visual language navigation.

[0033] Learning-based visual language navigation is typically limited by large-scale manually labeled trajectory-instruction pairs. This supervised paradigm not only faces high data acquisition costs but also lacks the ability to generalize to unknown environments, making it difficult to deploy in practice. Breakthroughs in cross-modal reasoning and scene understanding using large language models provide key enabling technologies for zero-shot visual language navigation. This invention leverages the cross-modal reasoning capabilities of large multimodal models (MLLMs) to achieve navigation decisions without additional supervision. Simultaneously, it provides crucial input for constructing cross-task incremental semantic memory graphs, enabling agents to accumulate long-term semantic memory while performing tasks in new environments.

[0034] Our task is to operate in a continuous visual-language navigation environment, unlike discrete environments where agents can only move between pre-defined nodes. This invention employs methods commonly used in continuous environments, utilizing a pre-trained waypoint predictor to generate candidate waypoints reachable from the agent's current position. At time step t, the agent acquires a 360° RGB-D observation of its current position, collecting 12 viewpoints at 30° intervals. Each viewpoint simultaneously acquires RGB and depth images. The collected observation set is represented as follows: (1); Waypoint predictors analyze current observations of an agent to predict candidate locations it can reach next. RGB color images collected by the intelligent agent. This is a depth image. The predictor uses the observation... Given input, output a set of K candidate waypoints: (2); For each candidate point The waypoint predictor also returns a local view image of that location. As supplementary information to semantic awareness, it is used for navigation planning.

[0035] Traditional zero-shot visual language navigation methods such as NavGPT and Open-Nav typically rely on multiple expert models working collaboratively, such as handling instruction understanding, scene parsing, and action prediction separately. This inevitably leads to information loss and semantic fragmentation during cross-modal information transfer. Therefore, this invention employs a multimodal large language model (MLLM) as a unified inference and decision engine. MLLM possesses end-to-end cross-modal understanding capabilities and can directly receive images and text as input. Specifically, the model's input includes: natural language instructions. Navigation history text description Current task progress and the local image set corresponding to the candidate waypoints. MLLM integrates these inputs, performs inference with the support of a cross-modal attention mechanism, and ultimately outputs an optimal waypoint, that is, the position the agent should reach next: (3); During navigation, the agent's real-time state assessment of the task's progress is crucial. Existing visual-language navigation methods often fail to explicitly describe progress information, instead relying on historical trajectories or implicit states to infer the task's execution stage. This approach can lead to navigation strategies easily repeating or missing sub-goals when clear progress cues are lacking. Therefore, this invention introduces instruction progress into the navigation framework, dividing the task into "completed parts" and "parts to be executed," which are then used as input constraints in each round of decision-making. While selecting the optimal waypoint, the MLLM also outputs an updated instruction progress description. The instruction progress output is presented in natural language text, directly describing the completed instruction portion, for example: "I have completed 'Walk around the brown leather ottoman, angling slightly righttowards the clock on the wall.'". This text, along with the complete instruction, navigation history, and candidate waypoint images, is then input into the MLLM, becoming an important factor influencing the decision.

[0036] Zero-shot visual-language navigation enables agents to complete complex tasks without supervision or external prior knowledge. Simultaneously, it allows agents to reach optimal candidate waypoints. Then, a panoramic view of the current location will be displayed. The information is input into the semantic parsing module, where semantic information is extracted and stored in the semantic memory graph (CISMG) in the form of nodes, gradually promoting the accumulation of long-term semantic memory.

[0037] 2. Construction and updating of cross-task incremental semantic memory graphs.

[0038] Inspired by human navigation, if an intelligent agent can establish a long-term structured representation similar to "semantic memory," it can not only improve its environmental understanding but also significantly enhance generalization and reuse performance in multi-task and complex scenarios. Therefore, this invention proposes a cross-task incremental semantic memory graph (CISMG). During zero-shot navigation, the agent continuously collects panoramic observations and parses semantic information such as room types and main objects while performing tasks, and generates potential co-occurring objects based on this semantic information. The goal of CISMG is to transform real-time perception results into a reusable, scalable, and reasonable structured representation across tasks, enabling the agent to progressively build and update its cognitive map of the environment, much like a human.

[0039] The design of CISMG is inspired by two modeling approaches: topological graphs and knowledge graphs. Topological graphs abstract the structure of the environment in the form of nodes and edges, effectively describing the spatial reachability between locations; knowledge graphs emphasize representing entities and their semantic relationships in structured triples, supporting complex semantic reasoning. The long-term semantic memory graph proposed in this paper combines the advantages of both, preserving the spatial reachability of topological graphs while inheriting the semantic interpretability of knowledge graphs.

[0040] The core idea of ​​CISMG is to encode the location, semantic attributes, and latent information obtained by the agent during navigation into a graph structure. This allows each node to represent not only spatial location but also semantic knowledge derived from explicit observation and implicit reasoning. The semantic memory graph proposed in this invention maintains continuity within the same scene and is not reset due to task switching; instead, it accumulates and expands across tasks. This long-term mechanism means that as tasks increase, the agent can gradually improve its cognitive representation of the environment, thereby reusing existing semantic and structural information in subsequent navigation.

[0041] At time step t, the cross-task incremental semantic memory graph is defined as: (4); 1) Node set Each node corresponds to a location that the agent has visited. These locations are generated by a pre-trained waypoint predictor.

[0042] (5); 2) Edge set ,in This represents the reachability determined by the waypoint predictor, when an agent can reach a location from a given location. Move directly to the location When an edge is established between corresponding nodes, the topological connectivity and navigability of CISMG are guaranteed.

[0043] (6); 3) Semantic attribute set To express semantic information more efficiently, this paper borrows the triple representation from knowledge graphs and models the scene semantic information (room type and main object set) as follows: (7); in , It is the intelligent agent that will conduct panoramic observations during zero-shot navigation. The result is obtained by inputting into the semantic parsing module. This represents the set of key objects observed at this location. For example, the current node. If the room type is living room and its main object set is {sofa, coffee table}, then its semantic attributes can be represented as: (living room, contains, {sofa, coffee table}).

[0044] 4) Common Sense Co-occurrence Set This is used to supplement latent semantic information beyond direct observation. After parsing the panoramic observations and obtaining the semantic attributes of the nodes, the large language model generates 2-3 latent co-occurring objects highly related to the current room type or observed objects based on its commonsense reasoning. Formalized as: (8); in, This represents the set of potentially co-occurring objects obtained based on common sense reasoning. For example, when bed is observed, the large model infers that the set of objects that may co-occur is {pillow, blanket}.

[0045] Through this modeling approach, CISMG unifies spatial topology and semantic attributes under a single structure. The set of semantic attributes provides an environmental representation based on real observations, while the set of common-sense co-occurrences supplements the potential semantic cues. The combination of the two enables the agent to make decisions based on actual perception, and to reason and search using common-sense cues when the target is not yet in sight.

[0046] During task execution, the cross-task incremental semantic memory graph (CISMG) not only needs to continuously expand with new information but also must suppress redundancy, avoid semantic conflicts, and update object information in real time to ensure the compactness, semantic consistency, and accuracy of the CISMG. Without effective constraints, the number of nodes will expand exponentially with the increase of tasks, and redundant semantics will weaken the stability and usability of the graph structure. Furthermore, objects in the environment may change, and timely correction is necessary to ensure that the semantic memory graph reflects the true state of the current environment. Therefore, this invention proposes a threshold-based dynamic update strategy. By setting multiple hierarchical thresholds and combining node insertion, node correction, node merging, and reachability establishment rules, the self-organizing evolution and long-term stable optimization of the CISMG are achieved.

[0047] 1) Node insertion: This rule ensures that CISMG continuously expands and enriches its semantic structure. In the current node... At this point, the agent selects the optimal waypoint from the set of candidate waypoints. The agent then inserts the node into the CISMG as a new node. Next, the agent moves to the waypoint and... and Establish a reachable edge between them to ensure the connectivity of the graph. Then analyze... Extract semantic attributes from panoramic images at the location. Co-occurrence attribute with common sense Insert node Then, correction, merging, and reachability rules are used for analysis. Relationships with existing nodes in CISMG.

[0048] 2) Node correction: Correction threshold This is used to determine whether the semantic information of existing nodes in the CISMG needs to be corrected. Since some objects in the environment may be moved by human intervention—for example, a "chair" might be moved from the living room to the dining room—the semantic attributes of the corresponding nodes in the semantic memory graph need to be updated promptly to ensure the accuracy of the graph structure. When a new node... With existing nodes When the distance between them is less than the correction threshold and the two rooms are of the same type: (9); These two nodes can be considered to represent the same physical location, that is, the agent has reached a previously visited location. At this time, the semantic information of the node will be updated, the new node completely replaces the original node, and inherits its connectivity.

[0049] 3) Node merging: Merge threshold This represents the minimum aggregation radius for semantically identical regions, used to detect local redundancy. When two nodes are spatially close and share the same semantics, they can be merged into a single node, thus avoiding duplicate records.

[0050] (10); The semantic attributes and commonsense co-occurrence attributes of the merged nodes are updated as follows: (11); in, , It is based on the merged A newly generated set of co-occurring objects.

[0051] 4) Reachability rules: Reachable threshold The spatial connectivity range between nodes is defined. In this paper, the threshold is not a fixed constant, but rather adaptively adjusted based on the average distance between nodes of the same room type. For nodes belonging to the same room type, reachability relationships are dynamically established according to this rule. For nodes of different room types, connectivity is generated only through node insertion rules. Specifically, after inserting a new node, the system first calculates the average reachable distance between existing nodes of the same room type in the graph and uses it as the reachability threshold for that category. When a new node... With existing nodes The Euclidean distance between them is less than In this case, we assume that there is potential traversability between them and establish connecting edges to maintain the overall connectivity of the graph.

[0052] (12); Insertion rules are used to introduce new semantic nodes, correction thresholds determine whether the semantic information of existing nodes needs to be updated, merging thresholds focus on eliminating local redundancy, and reachability thresholds determine potential traversable relationships. These thresholds satisfy the following relationship: CISMG's design simulates how human cognitive maps are formed: unfamiliar areas are added and recorded, pathways are gradually established, repeated locations are updated based on the latest observations, and neighboring areas are integrated into unified memory units. Through this dynamic update mechanism, CISMG can continuously expand environmental coverage in zero-shot navigation while avoiding redundancy, giving the semantic memory map self-organization and incremental optimization capabilities. Figure 2 This demonstrates the semantic and common-sense reasoning information at each step of the navigation process, as well as the construction and updating process of the CISMG. Yellow nodes represent historical nodes accumulated from previous tasks; thanks to our proposed cross-task memory mechanism, these nodes can be preserved and integrated into the semantic memory graph of the current task, thereby achieving continuous accumulation and reuse of knowledge.

[0053] 3. Global-local collaborative navigation strategy.

[0054] To efficiently complete long-command navigation tasks in complex environments, this invention proposes a global-local collaborative navigation strategy. Relying on the gradual construction of a cross-task incremental semantic memory graph (CISMG), it achieves progressive environmental cognition. Combined with a mechanism of global path planning and local dynamic fine-tuning, it effectively improves the accuracy and precision of navigation. Specifically, the agent's strategy transitions from zero-shot reasoning to global-local collaborative navigation. Initially relying on zero-shot reasoning to complete tasks, it shifts to path planning based on environmental information provided by CISMG, ultimately forming a global and local collaborative navigation strategy.

[0055] When an agent first enters a new environment, its semantic memory map is blank, unable to provide any readily available environmental information for navigation. Therefore, at this stage, the agent relies on zero-shot visual-language navigation methods to complete the navigation task. During execution, the agent acquires environmental information through panoramic observation and extracts semantic information from each observation frame using the semantic parsing module. Leveraging the common-sense reasoning capabilities of the large model, it obtains common-sense co-occurring objects and gradually inserts this information into the CISMG, providing progressively accumulated environmental knowledge for the execution of subsequent tasks.

[0056] As the task is executed, the agent's CISMG (Corporate Identification and Assistive Technology) gradually improves, its environmental awareness becomes richer, and its navigation strategy transitions from zero-shot reasoning to path planning based on environmental information provided by the CISMG, forming a global-local cooperative navigation strategy. This strategy combines global path planning with local dynamic fine-tuning to ensure global consistency and local accuracy in task execution. Figure 3 This demonstrates the process by which an agent generates a global reference path based on a constructed CISMG and completes precise local localization.

[0057] Global path planning: Long instructions or complex tasks may involve multiple objectives or span multiple rooms. Directly planning a global path from the start point to the finish line can easily lead to unstable planning or difficulties in local execution. Therefore, when an agent receives a task instruction, it first performs semantic parsing of the task, decomposing it into several subtasks. Each subtask corresponds to a target semantic set. This provides semantic constraints for subsequent node selection. Based on these subtasks, the agent, by querying the CISMG (Global Reference Graph), integrates the spatial location, semantic attributes, and common-sense co-occurrence of nodes to generate a global reference path from the starting point to the target node. (13); Global path planning not only considers the geometric connectivity between paths, but also needs to ensure the semantic consistency of the paths. For example, when performing the subtask of "reaching the pillow in the bedroom", the AI ​​will first query nodes with semantic attributes (bedroom, contains, {pillow}) in the CISMG. If "pillow" has not yet been observed in the current environment, the system can use potential candidate objects (such as bed, blanket) that are highly related to pillow in the common sense co-occurrence set to reason, thus still having global planning capabilities even when the target is missing.

[0058] Node selection strategy: During navigation tasks, the global path only provides the macroscopic direction. Moving node by node along the global path may lead to overly rigid navigation, making it unable to flexibly adjust when encountering local obstacles or environmental changes. Therefore, this paper proposes a comprehensive scoring system combining directional consistency and semantic consistency. This system scores each candidate waypoint and selects the candidate with the highest score as the target for the next step.

[0059] Directional consistency ensures that the agent's local movement direction is consistent with the global plan, avoiding deviation from the overall path. The current node is... Candidate nodes The direction of movement is: The direction of the global path within the current local window is: , It is distance For the nearest node, the directional consistency score is calculated as follows: (14); Semantic consistency ensures that candidate nodes are semantically close to the target of the current subtask, preventing the agent from entering regions irrelevant to the task. A unified multimodal encoder is used to extract feature representations of subtask instructions and candidate pathpoint images, and their semantic similarity is calculated using cosine similarity. (15); in, Candidate waypoints The corresponding image, It is a subtask The target semantic set, It is a cosine similarity function that measures the degree of similarity between the semantics of candidate waypoints and the semantics of the target waypoint.

[0060] The comprehensive score of candidate waypoints is obtained by weighted fusion of directional consistency and semantic consistency. (16); in It's a balancing parameter. The agent selects the candidate waypoint with the highest score. As the next target location: (17); Local fine-tuning: In complex indoor environments, when an agent performs long-sequence navigation tasks, the CISMG (Corporate Identity Module) provided by global path planning can satisfy the overall semantics of the instructions. However, the granularity of the semantic information in the CISMG is limited, which may lead to some deviation between the agent and the final goal. Therefore, we designed a local fine-tuning module to activate after the global path is completed. A set of local candidate waypoints is generated near the endpoint of the global path. These candidate points cover the reachable area around the agent's current position. The visual observation information corresponding to the candidate nodes and the navigation instructions are input into the MLLM (Multi-Level Model), which comprehensively evaluates each candidate point and selects the candidate point that is most conducive to approaching the final goal. The MLLM can flexibly adjust according to visual information and instructions, enabling the agent to approach the goal more accurately in the last step. Fine-tuning is only performed at the endpoint of the path, reducing the model inference cost and avoiding navigation deviations caused by insufficient granularity of CISMG information.

[0061] 4. Conclusions and Verification.

[0062] To demonstrate that the merging rule can effectively simplify CISMG and reduce redundant nodes, we randomly selected a scenario from the dataset and compared the CISMG before and after enabling the merging rule. Specific comparison metrics included NodeCount, Local Duplicate Semantic Rate (LDSR), and Local Duplicate Density (LDD). LDSR defines the percentage of semantically duplicated nodes in a node's nearest neighbors, and LDD defines the percentage of semantically duplicated nodes within the neighborhood of the merging threshold. The results are as follows: Figure 5 As shown, after enabling the merging rule, the number of nodes decreased from 128 to 71, a reduction of approximately 44.5%. LDSR decreased from 31% to 11%, and LDD decreased from 15% to 3%, indicating a significant reduction in semantic redundancy within the node neighborhood. Our designed merging rule significantly reduces redundant nodes in CISMG while improving the semantic accuracy and spatial efficiency of the graph.

[0063] This invention proposes a visual language navigation framework based on a cross-task incremental semantic memory graph (CISMG), aiming to effectively address the "task-by-task forgetting" problem and the lack of structured memory in existing visual language navigation methods. By constructing and maintaining a structured semantic memory graph called CISMG, the agent's navigation memory can continuously accumulate across tasks and effectively utilize experience from past tasks in the current task. CISMG structurally combines the spatial representation capability of topological graphs with the semantic interpretability of knowledge graphs. This structured memory representation significantly enhances the agent's navigation reasoning ability. Furthermore, this invention proposes a global-local cooperative navigation mechanism, enabling the agent to achieve fine-grained target localization while considering the global task direction. Experiments in simulated and real-world environments verify that this method effectively improves the performance of zero-shot navigation. In future work, we plan to integrate robot manipulation capabilities into the navigation system and add object interaction functionality, enabling the agent to perform truly embodied tasks.

[0064] Example 2 This embodiment provides a visual language navigation system based on cross-task incremental semantic memory graphs, including: The visual language navigation module is configured to: enable the agent to perform zero-shot visual language navigation tasks in a continuous environment, perform cross-modal alignment of natural language commands and environmental observations based on a multimodal large language model, select candidate waypoints and update task progress; The semantic memory graph module is configured to: construct and dynamically update a cross-task incremental semantic memory graph during navigation, wherein the semantic memory graph is used to structure and reuse the scene semantic information and spatial topological relationships perceived by the agent in historical tasks across tasks; The collaborative navigation module is configured to: after the semantic memory map has accumulated environmental knowledge, adopt a global and local collaborative navigation strategy, perform semantically guided path planning based on the semantic memory map at the global level, and perform fine-grained positioning of the target location based on real-time perception at the local level.

[0065] It should be noted that the above modules correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the above modules and their corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules can be executed in a computer system as part of the system.

[0066] In further embodiments, the following is also provided: An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0067] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0068] A computer-readable storage medium for storing computer instructions that, when executed by a processor, perform the method of Embodiment 1.

[0069] The method in Example 1 can be directly executed by a hardware processor, or it can be executed by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0070] A computer program product includes a computer program that, when executed by a processor, implements the method in Embodiment 1.

[0071] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0072] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0073] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0074] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0075] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A visual language navigation method based on cross-task incremental semantic memory graphs, characterized in that, Includes the following steps: The S1 agent performs zero-shot visual-language navigation tasks in a continuous environment. It performs cross-modal alignment of natural language commands and environmental observations based on a multimodal large language model, selects candidate waypoints, and updates the task progress. During navigation, S2 constructs and dynamically updates a cross-task incremental semantic memory graph. The semantic memory graph is used to structure and reuse the scene semantic information and spatial topological relationships perceived by the agent in historical tasks across tasks. S3 After the semantic memory map accumulates environmental knowledge, a global and local collaborative navigation strategy is adopted. At the global level, semantically guided path planning is performed based on the semantic memory map, and at the local level, fine-grained positioning of the target location is performed based on real-time perception.

2. The visual language navigation method based on cross-task incremental semantic memory graphs as described in claim 1, characterized in that, The cross-task incremental semantic memory graph includes a set of nodes, a set of edges, a set of semantic attributes, and a set of common sense co-occurrence. Nodes represent locations visited by the agent, edges represent reachability relationships between locations, semantic attributes represent room types and object sets corresponding to locations, and common sense co-occurrence represents potential co-occurring objects obtained based on common sense reasoning.

3. The visual language navigation method based on cross-task incremental semantic memory graphs as described in claim 1, characterized in that, The cross-task incremental semantic memory graph is dynamically updated using a multi-threshold-based update rule, specifically as follows: When an agent selects an optimal waypoint, it inserts this waypoint as a new node into the graph and establishes a reachable edge between the new node and the current node. When the distance between a new node and an existing node is less than the correction threshold, and they have the same room type, the new node completely covers the original node; When the distance between two nodes is less than the merging threshold and they are semantically identical, the two nodes are merged. If the distance between a new node and an existing node is less than the reachability threshold, a reachability edge is established between them. Among them, the merging threshold is greater than the correction threshold, and the achievable threshold is adaptively adjusted according to environmental information.

4. The visual language navigation method based on cross-task incremental semantic memory graphs as described in claim 1, characterized in that, At the global level, semantically guided path planning based on semantic memory graphs is specifically performed as follows: natural language instructions are semantically parsed and decomposed into several sub-tasks, each sub-task corresponding to a target semantic set. Based on the semantic memory graph, the spatial location, semantic attributes, and common sense co-occurrence information of nodes are integrated to generate a global reference path from the starting point to the target node.

5. The visual language navigation method based on cross-task incremental semantic memory graphs as described in claim 4, characterized in that, During the execution of the global reference path, candidate waypoints at each step are selected based on a comprehensive score of directional consistency and semantic consistency, wherein: The directional consistency score measures the consistency between the candidate point's movement direction and the global path direction; Semantic consistency score measures the similarity between the semantics of candidate points and the semantics of the current subtask target.

6. The visual language navigation method based on cross-task incremental semantic memory graph as described in claim 1, characterized in that, Fine-grained localization is triggered when the agent reaches the vicinity of the global path endpoint, including: Generate a set of local candidate waypoints around the current location; The visual observation information corresponding to each local candidate waypoint is input together with the navigation instructions into the multimodal large language model; The candidate points that are most conducive to approaching the final target are evaluated and selected by a multimodal large language model for accurate localization.

7. A visual language navigation system based on cross-task incremental semantic memory graphs, characterized in that, include: The visual language navigation module is configured to: enable the agent to perform zero-shot visual language navigation tasks in a continuous environment, perform cross-modal alignment of natural language commands and environmental observations based on a multimodal large language model, select candidate waypoints and update task progress; The semantic memory graph module is configured to: construct and dynamically update a cross-task incremental semantic memory graph during navigation, wherein the semantic memory graph is used to structure and reuse the scene semantic information and spatial topological relationships perceived by the agent in historical tasks across tasks; The collaborative navigation module is configured to: after the semantic memory map has accumulated environmental knowledge, adopt a global and local collaborative navigation strategy, perform semantically guided path planning based on the semantic memory map at the global level, and perform fine-grained alignment of the target position based on real-time perception at the local level.

8. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-6.

Images