A zero-shot visual navigation method and system based on spatial understanding

Abstract

The application discloses a kind of zero sample visual navigation method and system based on space understanding, comprising constructing target-centered space layout priori;Design observation evaluation and correction module, identify new object using basic vision model, then filter unreliable detection through temporal consistency and semantic correlation verification, construct scene posteriori according to reliable detection;Design knowledge fusion module, fuse space layout priori and scene observation posteriori, to form priori and posteriori knowledge that can express the spatial constraints between objects;Propose dynamic programming module, calculate the optimal path in potential experience space based on current location and space knowledge graph, narrow the planning search range, effectively reduce redundant exploration.By mining the inherent spatial relationship between objects and regions in the scene, the reliability and efficiency of environment perception and path planning are improved, and it has good universality and application value.

Description

Technical Field

[0001] This invention generally relates to the field of intelligent robot navigation technology, and more specifically, to a zero-sample visual navigation method and system based on spatial understanding. Background Technology

[0002] Zero-shot target navigation, a core problem of embodied intelligence, aims to enable intelligent agents to autonomously locate and reach user-specified targets in unknown environments, relying solely on limited first-person observations, without requiring specific training for the target. This capability not only represents a leap from "passive execution" to "autonomous understanding and reasoning" for intelligent agents but is also a crucial prerequisite for service robots, drones, and intelligent assistants to perform tasks in real-world, complex environments. Its development has significant strategic importance and application value for promoting natural human-machine interaction and enhancing the adaptability and generalization capabilities of autonomous systems.

[0003] With the development of deep learning technology, intelligent agent target navigation algorithms based on convolutional neural networks have achieved good results. Existing technical solutions are mainly divided into end-to-end methods and modular methods: end-to-end methods, driven by large-scale data, can learn the mapping from visual input to action sequences to a certain extent; modular methods, by decoupling perception and planning and combining semantic and geometric information, effectively alleviate overfitting problems and improve generalization ability in unknown environments. However, in complex or large-scale environments, existing methods still have significant shortcomings. On the one hand, when the target is invisible, occluded, or far away, the perception results relying solely on visual input are often incomplete, easily leading to false positives and false negatives; on the other hand, due to the lack of global map information, existing local greedy planning strategies are inefficient in long-range exploration, often falling into blind search and path backtracking, making it difficult to guarantee the stability and efficiency of navigation.

[0004] Therefore, there is an urgent need to propose a new technical solution that can fully explore and utilize the semantic and spatial information in the environment, thereby improving the perception robustness and path planning efficiency of the agent when there is a lack of direct prior observation of the target. Summary of the Invention

[0005] To address the technical problem that existing zero-shot target navigation algorithms struggle to support reliable perception and planning reasoning when the viewpoint is limited or semantic cues are insufficient, leading to low navigation efficiency or even failure, this invention provides a zero-shot visual navigation method and system based on spatial understanding.

[0006] According to a first aspect of the present invention, a zero-shot visual navigation method based on spatial understanding is provided, the method comprising the following steps:

[0007] S100: The target to be searched forms a general target spatial layout based on empirical knowledge through reasoning using a large language model, so as to construct a spatial layout prior and initialize the spatial knowledge graph;

[0008] S200: Based on the color image and depth image collected in the current scene, the detection results of the target to be found in the scene are obtained through the basic visual detection model, and the spatial layout posterior is generated through evaluation and correction, and the spatial layout posterior or the spatial coordinates of the target are output.

[0009] S300: Layered knowledge fusion of spatial layout priors and spatial layout posteriors to generate a spatial knowledge graph that simultaneously expresses empirical spatial layout and actual spatial layout.

[0010] S400: Guides navigation path planning through spatial knowledge graphs and decodes the planned path into verbal guidance prompts;

[0011] S500: The above-mentioned language-based guidance prompts are used to perform semantic similarity calculations with images from multiple perspectives through a visual language model to determine and select the leading edge region corresponding to the best perspective, and the center position of the leading edge region is extracted as waypoint coordinates.

[0012] S600: Performs low-level path planning on spatial coordinates or waypoint coordinates to obtain an executable and precise navigation path;

[0013] S700: The agent moves according to the precise navigation path, gradually approaching the target from the current position and completing the overall navigation task.

[0014] According to some embodiments of the present invention, S100 specifically involves reasoning based on empirical knowledge using a large language model to determine the name or description of the target to be searched, extracting a general target spatial layout from the large language model, and then constructing common placement positions of the target in typical environments and the distribution of surrounding objects to form a spatial layout prior, thereby initializing a spatial knowledge graph; the spatial knowledge graph is a graph structure constructed with the target as the center, where nodes represent objects or regions and edges represent spatial relationships between objects.

[0015] According to some embodiments of the present invention, the target detection result in step S200 is evaluated and corrected to generate a spatial layout posterior specifically as follows:

[0016] S210: Perform confidence fusion on the detection results of the target to be searched. Confidence fusion includes temporal consistency fusion and semantic consistency fusion to generate fusion confidence. Then, the detection results are filtered through effective detection gating. When the fusion confidence score is higher than the set threshold, the result is written into the spatial layout posterior.

[0017] S220: Determine whether the target to be searched exists. If the target exists, extract the spatial coordinates of the target and jump to S600 for subsequent operations.

[0018] According to some embodiments of the present invention, in step S200, the target detection result includes the category name, confidence level, and target location. Before generating the spatial layout posterior, the detection result of the current frame is associated with the detections of similar locations in the previous frames to establish a continuous observation record of the same candidate targets in time, thereby forming a temporal detection trajectory as the basis for subsequent fusion of various confidence levels. The specific steps are as follows:

[0019] Accumulated detection results of the target across multiple frames:

[0020] ;

[0021] in, This indicates a confidence-weighted average for continuous testing. Indicates the confidence level of a single test. L It represents the cumulative number of tests;

[0022] Timing consistency is calculated as follows:

[0023] ;

[0024] in, = , Represents the frame rate scaling factor. K It is a threshold for the number of detections, which can rapidly improve the temporal confidence when there are multiple stable detections, and decay exponentially when there are missing detections, thereby obtaining a more stable judgment of the existence of the target;

[0025] The visual language model is invoked to perform layout rationality reasoning. Combining scene semantics, room type, object attributes, and prior knowledge, the system judges whether the detected spatial distribution conforms to a reasonable distribution and derives a semantic consistency score.

[0026] ;

[0027] in, This indicates that reasoning is performed using a visual language model. The prompt indicates that the VLM is activated for layout rationality reasoning. This refers to the current position of the agent in the semantic space. Indicates the detected object;

[0028] Credibility fusion: The temporal consistency confidence score and semantic consistency confidence score are fused to obtain the final detection confidence score, which is used for subsequent node confirmation and relationship construction.

[0029] ;

[0030] in, The adjustable weights are used to control whether the credibility leans more towards temporal consistency or semantic consistency.

[0031] According to some embodiments of the present invention, the layered fusion in S300 is specifically as follows: based on whether the prior and posterior contents of the spatial layout overlap, it is divided into two types: consistent fusion and incremental fusion. When a new node or new relationship does not have a corresponding entity or edge in the spatial knowledge graph, that is, when the prior and posterior have no intersection, it is directly added to the spatial knowledge graph in an incremental manner to expand the spatial knowledge. The specific formula is as follows:

[0032] ;

[0033] in, It represents the integration of knowledge. Representing scene observation, incremental nodes contain their category, confidence level, and spatial inference label information, while incremental edges store their three types of spatial constraints;

[0034] When a node or edge for new inference already exists in the prior graph, the posterior is used to cover the prior to correct any priors that conflict with the scenario, since the posterior has undergone multiple verifications and is scenario-dependent.

[0035] According to some embodiments of the present invention, S400 specifically includes the following steps:

[0036] S410: Utilizes VLM in conjunction with images observed in the current scene to perform reasoning, determine the agent's position node in the spatial knowledge graph, enabling accurate self-localization in the spatial knowledge graph and providing a starting point for path planning;

[0037] S420: Based on the spatial relationship between the target node and the current location node in the spatial knowledge graph, VLM is used to plan potential candidate paths to the target, and semantic information about the next key step is extracted from the candidate paths to generate corresponding guidance words, thereby constructing a language-based guidance prompt.

[0038] According to some embodiments of the present invention, the step of determining the optimal leading edge region in S500 includes:

[0039] S510: Input the language-based guidance prompt into the visual language model, perform semantic matching with multiple viewpoint images, calculate the semantic similarity between each viewpoint image and the language-based guidance prompt, and map the semantic similarity onto the traversable area top-down map of the current viewpoint coverage area;

[0040] S520: If there is an overlap in perspectives, the semantic similarity of the overlapping regions is weighted to obtain a new semantic similarity score of the overlapping regions that can simultaneously reflect the two perspectives and the target. The perspective with the highest similarity is selected as the best frontier region.

[0041] According to some embodiments of the present invention, the maximum value of the similarity score is selected as follows:

[0042] ;

[0043] in, Indicates the highest value score; Represents the set of candidate waypoints at time step t; This indicates that the semantic similarity value between the two is calculated. Represents the current viewpoint image; function Convert relationship paths and the agent's current position into verbal guidance prompts; Represents potential paths leading to a target on a spatial relationship map; This indicates the location of the agent in the spatial relationship graph.

[0044] According to some embodiments of the present invention, in S600, the underlying path planner is trained using a point-to-point reinforcement learning strategy. During the training process, the agent relies on visual observation and continuously optimizes the decision-making process through reinforcement learning algorithms in a simulated environment, learning the control strategy for moving from the current position to the specified target point, so as to improve the ability to follow paths and avoid obstacles.

[0045] According to a second aspect of the present invention, a zero-shot visual navigation system based on spatial understanding is provided, the system being used to implement the zero-shot visual navigation method based on spatial understanding. The system includes:

[0046] The input module is used for inputting the target to be searched and acquiring image information;

[0047] The large language model module is used to process the input target to generate spatial layout priors;

[0048] The basic visual model module is used to obtain the detection results of the target to be found in the scene through image information;

[0049] The evaluation and correction module is used to generate spatial layout posterior or target spatial coordinates from the detection results through evaluation and correction.

[0050] The knowledge fusion module is used to perform hierarchical knowledge fusion between spatial layout priors and spatial layout posteriors in order to generate a spatial knowledge graph.

[0051] The dynamic programming module is used to generate waypoint coordinates from a spatial knowledge graph;

[0052] The point navigation strategy module is used to perform low-level path planning on spatial coordinates or waypoint coordinates to obtain an executable and accurate navigation path.

[0053] As can be seen from the above technical solution, the advantages and positive effects of the zero-shot visual navigation method and system based on spatial understanding of the present invention are as follows:

[0054] 1. This invention constructs a spatial knowledge graph and introduces an evaluation and correction module and a knowledge fusion module. In cases where the target viewpoint is poor, obstructed, or the distance is far, it can use temporal consistency and scene semantic distribution to verify and correct the detection results, effectively suppress false detections and compensate for missed detections, thereby improving the robustness and reliability of target detection. At the same time, the knowledge fusion module constructs scene priors and posteriors, thereby better forming spatial constraint guidance related to specific scenes.

[0055] 2. This invention proposes a dynamic relationship planning module that combines a visual language model with prior information on spatial layout. In long-distance exploration, it can dynamically generate guiding potential paths, reduce the search space, reduce ineffective exploration, and avoid getting trapped in local optima, thereby improving planning efficiency.

[0056] 3. This invention achieves an overall improvement in target navigation performance in unknown environments, significantly increasing the success rate of finding targets and path efficiency, reducing redundant exploration, and has strong universality and practical value. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a flowchart illustrating a zero-shot visual navigation method based on spatial understanding in one embodiment of the present invention.

[0059] Figure 2 This is a schematic diagram illustrating the evaluation and correction process of a zero-shot visual navigation method based on spatial understanding in one embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the dynamic programming process of a zero-shot visual navigation method based on spatial understanding in one embodiment of the present invention. Detailed Implementation

[0061] The embodiments of this application will be further described in detail below with reference to the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to illustrate the principles of this application by way of example, but should not be used to limit the scope of this application. This application can be implemented in many different forms and is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

[0062] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, they should be considered part of the specification.

[0063] like Figure 1 As shown, according to a first aspect provided by the present invention, a zero-shot visual navigation method based on spatial understanding is provided, the method comprising the following steps:

[0064] S100: Construct a goal-oriented spatial layout prior, which is obtained by reasoning from a large language model trained on massive Internet data. It statistically summarizes or generalizes information such as typical room types, typical location areas, and common co-occurring objects that may appear in the target object, forming a general target space based on empirical knowledge.

[0065] S200: Construct an observation evaluation and correction module to generate a layout posterior. This module first uses a basic visual detection model to obtain the detection results of candidate objects in the scene and evaluates their credibility. It calculates the temporal consistency score and semantic consistency score respectively, and obtains the comprehensive credibility score through weighted fusion. Then, it removes false positive results with low credibility through effective detection gating, generates a layout posterior, determines whether the target exists, and outputs the spatial layout posterior or the spatial coordinates of the target.

[0066] S300: Design a prior and posterior knowledge fusion module to perform hierarchical fusion of the obtained target center prior and the obtained scene posterior. This fusion includes two types of operations: consistency fusion and incremental fusion, to generate spatial constraints that simultaneously express the empirical spatial layout prior and the actual spatial layout posterior.

[0067] S400: Design a dynamic programming module to generate the optimal path that satisfies spatial constraints. This module uses the fused spatial constraint knowledge to guide navigation path planning, thereby reducing invalid exploration. When no target or its co-occurring related objects are detected, the dynamic relationship planning module is activated. It uses a visual language model to fuse the semantic information of the current position with the spatial knowledge graph topology knowledge graph to generate the optimal path in the potential relationship space to reach the target, and decodes the path into a language-based guidance prompt.

[0068] S500: Calculates the semantic similarity score between the linguistic guidance prompts and each frontier region through a visual language model. Then, it uses the score to determine the similarity between the image information from each perspective and the target spatial information. The frontier region corresponding to the perspective with the highest similarity is the most promising frontier region to explore, and the center of the frontier region is selected as the waypoint coordinate.

[0069] S600: Performs low-level path planning on spatial coordinates or waypoint coordinates to obtain an executable and precise navigation path;

[0070] S700: The agent moves according to the precise navigation path, gradually approaching the target from the current position and completing the overall navigation task.

[0071] In some embodiments, zero-shot visual navigation relies solely on self-visual observation in unknown environments, making it susceptible to poor perspectives or insufficient semantic cues. This leads to instability in the reasoning of the underlying model during perception and planning, resulting in inefficient or even failed navigation. To address this issue, this invention proposes a structured scene prior-driven navigation framework. This framework dynamically constructs and updates a spatial knowledge graph during exploration, introducing the inherent relationships between objects and regions into perception and planning reasoning. Specifically, by introducing an evaluation and correction module, the target detection results are verified and corrected from both temporal and semantic consistency perspectives, thereby enhancing the robustness of visual perception. Simultaneously, a relationship-guided planning module is designed to reduce the planning search space in real time based on the scene relationship graph, dynamically calculate the optimal path, and reduce redundant exploration. This design improves perception accuracy and planning efficiency under weak semantics and complex scene conditions. It effectively alleviates the problem of unreliable reasoning in existing methods under conditions of insufficient semantics and limited perspectives, significantly improving the success rate and path efficiency of navigation tasks, and demonstrating stronger generalization ability and robustness in zero-shot object target navigation.

[0072] In some embodiments of the present invention, S100 includes the following steps:

[0073] S110: Utilize a large-scale language model to extract general spatial layout priors about the target object, thereby initializing a spatial knowledge graph. The spatial knowledge graph is a graph structure built around the target, where nodes represent objects or regions and edges represent spatial relationships between objects. This graph contains topological relationships, distance relationships, and orientation relationships, which fully describe the spatial distribution information between items and between items and space.

[0074] In this invention, the spatial relationship modeling module adopts a multi-level relationship representation, namely distance relationship unit, direction relationship unit, and topological relationship unit. By introducing distance relationship to measure the proximity and reachability between objects, it helps to prioritize the exploration of nearby targets and eliminate irrelevant nodes during navigation. By introducing direction relationship to characterize the geometric layout and orientation information of the environment, it can effectively eliminate ambiguity between instances of the same type of object and provide orientation index to improve navigation efficiency when the viewpoint is limited. By introducing topological relationship to model the connectivity semantics of the region hierarchy, it can establish reasonable region abstraction and high-level semantic reasoning in complex indoor environments with structural obstacles. This invention combines three types of relationships to achieve refined, semantically reasonable, and scalable spatial modeling, improving the perception, reasoning, and action capabilities of intelligent agents in complex environments.

[0075] This invention conducts simulation experiments using HM3D and Matterport3D datasets. The initial spatial relationship graph is stored in JSON format, containing the following content: "Object": {"Region": Room / Region Name, "Confidence": 0~1, "Associated Object": [{"Object Name": **, "Distance": ** meters, "Region": Room / Region Name, "Direction": Front / Back / Left / Right / Up / Down, "Confidence": 0~1}]}.

[0076] In this embodiment, the initial spatial relationship graph in S110 refers to the use of a large language model, specifically the Deepseek R1 model, to extract spatial relationships from all target objects in the two simulation datasets. Specific prompts include: "You are a spatial common-sense reasoner focused on indoor environments. Your task is to generate physically plausible prior knowledge about spatial relationships based on the cognitive principle of spatial memory. For a given target object, please provide its typical topological, directional, and distance relationships." After generating the spatial relationships using the large model, these initial spatial relationships are recorded in a JSON file, including various target names, their regions, associated objects, and relationships.

[0077] like Figure 2 As shown, in some embodiments of the present invention, S200 includes the following steps:

[0078] S210: Perform confidence fusion on the detection results of the target to be searched. Confidence fusion includes temporal consistency fusion and semantic consistency fusion to generate fusion confidence. Then, the detection results are filtered through effective detection gating. When the fusion confidence score is higher than the set threshold, the result is written into the spatial layout posterior.

[0079] S220: Determine whether the target to be searched exists. If the target exists, extract the spatial coordinates of the target and jump to S600 for subsequent operations.

[0080] Object detection is performed on color images using the basic visual detection model YOLOv7 and the open-set model Grounding DINO to obtain initial detection results. Subsequently, confidence fusion is performed on the detection results, including multi-frame detection confidence fusion and layout rationality inference based on a visual language model, to generate a fused confidence score. Finally, the detection results are filtered through effective detection gating; only results with a fused confidence score higher than a set threshold (0.8 for COCO categories and 0.6 for open-set categories) are included in the layout posterior.

[0081] To determine the existence of a target, multi-frame detection confidence fusion is based on the principle that "single-frame detection is unreliable, only continuous detection is reliable," maintaining a temporal detection counter for each object category. An object is considered to exist only if it is detected within several consecutive frames; if it is not observed in subsequent frames, its confidence level is lowered. Specifically, the continuous detection results within the agent's field of vision are accumulated to form a weighted temporal detection score.

[0082] Accumulated detection results of objects within the agent's field of vision:

[0083] ;

[0084] in, This indicates a confidence-weighted average for continuous testing. Indicates the confidence level of a single test. L It represents the cumulative number of tests;

[0085] Then, the timing consistency calculation is as follows:

[0086] ;

[0087] in, = , Indicates rate of rise control. K It is the threshold for the number of detections.

[0088] VLM Layout Rationality Reasoning: The Visual Language Model (VLM) is invoked to perform layout rationality reasoning. VLM combines scene semantics, room type, object attributes, and prior knowledge to determine whether the detected layout conforms to a reasonable spatial distribution, deriving a semantic consistency score.

[0089] ;

[0090] in, This indicates that reasoning is performed using a visual language model. The prompt indicates that the VLM is activated for layout rationality reasoning. This refers to the current position of the agent in the semantic space. This indicates the detected object.

[0091] Credibility fusion: The temporal consistency confidence score and semantic consistency confidence score are fused to obtain the final detection confidence score, which is used for subsequent node confirmation and relationship construction.

[0092] ;

[0093] in, The confidence level is fused into a score. The adjustable weights are used to control whether the credibility leans more towards temporal consistency or semantic consistency.

[0094] Specifically, such as Figure 2 As shown, this method effectively suppresses the problems of false detection and false negative detection of targets caused by limited viewpoint, occlusion or long distance, thereby improving the robustness and accuracy of target detection and knowledge graph update.

[0095] According to some embodiments of the present invention, step S300 specifically includes the following steps:

[0096] S310. The system calls VLM at a fixed period (every 10 steps) to perform batch reasoning on the spatial relationships of the currently confirmed nodes; while avoiding the high computational overhead of reasoning per frame, it improves the real-time performance and stability of the system.

[0097] S320. The hierarchical fusion strategy is divided into two types based on whether the prior and posterior contents of spatial layout overlap: consistency fusion and incremental fusion. When a new node or relationship does not have a corresponding entity or edge in the spatial knowledge graph, i.e., the prior and posterior have no intersection, it is directly added to the spatial knowledge graph incrementally to expand the spatial knowledge of the target center. The specific formula is as follows:

[0098] ;

[0099] in It represents the integration of knowledge. Representing scene observation, incremental nodes contain their category, confidence level, and spatial inference label information, while incremental edges store their three types of spatial constraints.

[0100] When a node or edge for new inference already exists in the prior graph, the posterior is overridden by the prior to correct any priors that conflict with the scenario, since the posterior has undergone multiple verifications and is scenario-dependent.

[0101] The coverage includes: replacing unreliable or coarse estimated positions in the prior with posterior node positions; updating the prior node confidence with posterior confidence; and correcting spatial relationships in the prior that conflict with the real scene with edge relationships inferred from the posterior.

[0102] This approach ensures that the merged spatial knowledge graph retains both the experiential structure and faithfully reflects the current environment.

[0103] In some embodiments of the present invention, S400 includes the following steps:

[0104] S410: The dynamic relationship planning module, based on a visual language model, matches the scene images collected by the current agent with the spatial knowledge graph to determine the position node of the agent in the spatial relationship space;

[0105] S420: Based on the spatial relationship between the target object and the current location, use a visual language model to plan potential candidate paths to the target, extract semantic information about the key areas of the next step from the candidate paths, generate corresponding guidance words, and construct language-based guidance prompts.

[0106] In some embodiments of the present invention, the guidance word generation is based on the following system prompt: "You are a potential relational spatial path inference assistant for an agent, and need to generate executable navigation instructions by combining visual context and spatial relational graph." The framework first locates the agent in the spatial relational graph through visual context (prompt: "Map visual features to relational graph nodes to determine the agent's precise position in the graph"); then, combined with the agent's current position, it uses spatial layout priors to infer potential paths to the target and extracts key guidance words from them. The prompt words are set as follows: "Generate action-oriented navigation cues in the form of keywords", output: ("area", "object"); finally, these cues are encoded into targeted prompts, such as "Entering {area} ahead" or "{object} may exist nearby".

[0107] like Figure 3 As shown, in some embodiments of the present invention, S500 includes the following steps:

[0108] S510: The linguistic guidance prompt is input into the visual language model. The prompt is semantically matched with multiple viewpoint images, and the semantic similarity between each viewpoint image and the guidance word is calculated. This semantic similarity is then mapped onto a top-down view map of the traversable region covered by the current viewpoint. Simultaneously, the midpoint of the leading edge corresponding to each viewpoint is marked as a candidate waypoint on the traversable region. The leading edge region is the boundary between explored and unexplored areas in the current environment, representing the boundary area that the agent has not yet observed but can continue to traverse and expand its cognitive range. The top-down view map of the traversable region is a top-down two-dimensional representation of the traversable area generated based on obstacles and free space in the current environment, used to indicate the range within which the agent can safely move.

[0109] S520: If there is an overlap in the viewpoints, the semantic similarity of the overlapping areas is weighted to obtain a new score for the overlapping areas that can simultaneously reflect the semantic similarity between the two viewpoints and the target. The position corresponding to the viewpoint with the highest similarity is selected as the navigation waypoint coordinates.

[0110] In some embodiments of the present invention, the similarity score and the selection of the maximum value are specifically as follows:

[0111] ;

[0112] in, Indicates the highest value score; Represents the set of candidate waypoints at time step t; This indicates that the semantic similarity value between the two is calculated. Represents the current viewpoint image; function Convert the relationship path and the agent's current location into natural language prompts; Represents potential paths leading to a target on a spatial relationship map; This indicates the location of the agent in the spatial relationship graph.

[0113] By introducing a spatial relationship graph, the agent is guided to select the next most advantageous semantic clue, enabling it to gradually narrow the search range based on relational priors and approach the target step by step during the exploration process. This method avoids blind searching and inefficient backtracking, improving the rationality of path selection and the overall efficiency of navigation.

[0114] In some embodiments of the present invention, S600 includes the following steps:

[0115] S610: The bottom-level path planner is trained using a point-to-point reinforcement learning strategy. During the training process, the agent relies on visual observation and odometry information. In a simulated environment, the decision-making process is continuously optimized through reinforcement learning algorithms. The agent learns the control strategy to move from the current position to the specified target point, thereby improving the ability to follow paths and avoid obstacles.

[0116] S620: When the navigation task is executed, the selected waypoint coordinates are input to the underlying path planner. The path planner generates a locally optimal path based on the current pose and the input target point, and converts the path into executable motion control commands. The underlying path planner drives the agent to complete continuous actions according to the generated control commands, ensuring that it can stably move from the current position to the selected waypoint position, thereby realizing the accurate execution of the target navigation task.

[0117] The method proposed in this embodiment is compared with greedy zero-shot target navigation methods based on current information decision-making. Existing local information decision-making methods often use greedy algorithms. As can be seen from the navigation path comparison results, the method of this invention can guide the agent to the target position more stably under various target conditions. In the same environment, compared with the SG-Nav method and VLFM method, this invention can effectively utilize the fusion information of spatial prior and scene posterior, reduce invalid exploration paths, improve the success rate of target localization, and still complete the navigation task in complex layouts or when the target is relatively concealed.

[0118] According to a second aspect of the present invention, a zero-shot visual navigation system based on spatial understanding is provided, the zero-shot visual navigation system based on spatial understanding being used to implement the zero-shot visual navigation method based on spatial understanding. The system includes:

[0119] The input module is used for inputting the target to be searched and acquiring image information;

[0120] The large language model module is used to process the input target to generate spatial layout priors;

[0121] The basic visual model module is used to obtain the detection results of the target to be found in the scene through image information;

[0122] The evaluation and correction module is used to generate spatial layout posterior or target spatial coordinates from the detection results through evaluation and correction.

[0123] The knowledge fusion module is used to perform hierarchical knowledge fusion between spatial layout priors and spatial layout posteriors in order to generate a spatial knowledge graph.

[0124] The dynamic programming module is used to generate waypoint coordinates from a spatial knowledge graph;

[0125] The point navigation strategy module is used to perform low-level path planning on spatial coordinates or waypoint coordinates to obtain an executable and accurate navigation path.

[0126] The embodiments of this application have now been described in detail. To avoid obscuring the concept of this application, some details known in the art have not been described. Those skilled in the art can fully understand how to implement the technical solutions disclosed herein based on the above description.

[0127] While specific embodiments of this application have been described in detail by way of examples, those skilled in the art should understand that the above examples are for illustrative purposes only and are not intended to limit the scope of this application. Those skilled in the art should understand that modifications can be made to the above embodiments or equivalent substitutions can be made to some technical features without departing from the scope and spirit of this application. In particular, as long as there is no structural conflict, the various technical features mentioned in the embodiments can be combined in any manner.

Claims

1. A zero-shot visual navigation method based on spatial understanding, characterized in that, The method includes the following steps: S100: The target to be searched forms a general target spatial layout based on empirical knowledge through reasoning using a large language model, so as to construct a spatial layout prior and initialize the spatial knowledge graph; S200: Based on the color image and depth image collected in the current scene, the detection results of the target to be found in the scene are obtained through the basic visual detection model, and the spatial layout posterior is generated through evaluation and correction, and the spatial layout posterior or the spatial coordinates of the target are output. S300: Layered knowledge fusion of spatial layout priors and spatial layout posteriors to generate a spatial knowledge graph that simultaneously expresses empirical spatial layout and actual spatial layout. S400: Guides navigation path planning through spatial knowledge graphs and decodes the planned path into verbal guidance prompts; S500: The above-mentioned language-based guidance prompts are used to perform semantic similarity calculations with images from multiple perspectives through a visual language model to determine and select the leading edge region corresponding to the best perspective, and the center position of the leading edge region is extracted as waypoint coordinates. S600: Performs low-level path planning on spatial coordinates or waypoint coordinates to obtain an executable and precise navigation path; S700: The agent moves according to the precise navigation path, gradually approaching the target from the current position and completing the overall navigation task.

2. The space-understanding-based zero-shot visual navigation method according to claim 1, characterized in that, Specifically, S100 involves reasoning based on empirical knowledge using a large language model to determine the name or description of the target to be searched, extracting a general target spatial layout from the large language model, and then constructing common placement positions of the target in typical environments and the distribution of surrounding objects to form a spatial layout prior, thereby initializing a spatial knowledge graph. The spatial knowledge graph is a graph structure constructed with the target as the center, where nodes represent objects or regions and edges represent spatial relationships between objects.

3. The space understanding based zero-shot visual navigation method of claim 1, wherein, The target detection result in S200 is evaluated and corrected to generate a spatial layout posterior, specifically as follows: S210: Perform confidence fusion on the detection results of the target to be searched. Confidence fusion includes temporal consistency fusion and semantic consistency fusion to generate fusion confidence. Then, the detection results are filtered through effective detection gating. When the fusion confidence score is higher than the set threshold, the result is written into the spatial layout posterior. S220: Determine whether the target to be searched exists. If the target exists, extract the spatial coordinates of the target and jump to S600 for subsequent operations.

4. The space understanding based zero-shot visual navigation method of claim 1, wherein, In step S200, the target detection result includes the category name, confidence level, and target location. Before generating the spatial layout posterior, the detection result of the current frame is associated with the detections of similar locations in the previous frames to establish a continuous observation record of the same candidate targets in time, thereby forming a temporal detection trajectory as the basis for subsequent fusion of various confidence levels. The specific steps are as follows: Accumulated detection results of the target across multiple frames: ； wherein, represents a confidence weighting of a sustained detection, represents a confidence of a single detection, L is the number of cumulative detections; Timing consistency is calculated as follows: ； wherein, = 1 / 2 , represents the frame number scale factor, K is the detection number threshold value, which can quickly increase the timing confidence when multiple stable detections are made, and exponentially decay when missing detections are made, thereby obtaining a more stable target existence judgment; The visual language model is invoked to perform layout rationality reasoning. Combining scene semantics, room type, object attributes, and prior knowledge, the system judges whether the detected spatial distribution conforms to a reasonable distribution and derives a semantic consistency score. ； wherein, represents reasoning with a visual language model, represents a prompt word that activates a VLM for layout plausibility reasoning, refers to the location on the semantic space of the agent at present, represents a detected object; Credibility fusion: The temporal consistency confidence score and semantic consistency confidence score are fused to obtain the final detection confidence score, which is used for subsequent node confirmation and relationship construction. ； in, The adjustable weights are used to control whether the credibility leans more towards temporal consistency or semantic consistency.

5. The zero-shot visual navigation method based on spatial understanding according to claim 1, characterized in that, The layered fusion in S300 is specifically divided into two types based on whether the prior and posterior contents of spatial layout overlap: consistency fusion and incremental fusion. When a new node or relationship does not have a corresponding entity or edge in the spatial knowledge graph, i.e., the prior and posterior have no intersection, it is directly added to the spatial knowledge graph incrementally to expand spatial knowledge. The specific formula is as follows: ； in, It represents the integration of knowledge. Representing scene observation, incremental nodes contain their category, confidence level, and spatial inference label information, while incremental edges store their three types of spatial constraints; When a node or edge for new inference already exists in the prior graph, the posterior is used to cover the prior to correct any priors that conflict with the scenario, since the posterior has undergone multiple verifications and is scenario-dependent.

6. The zero-shot visual navigation method based on spatial understanding according to claim 1, characterized in that, The S400 specifically includes the following steps: S410: Utilizes VLM in conjunction with images observed in the current scene to perform reasoning, determine the agent's position node in the spatial knowledge graph, enabling accurate self-localization in the spatial knowledge graph and providing a starting point for path planning; S420: Based on the spatial relationship between the target node and the current location node in the spatial knowledge graph, VLM is used to plan potential candidate paths to the target, and semantic information about the next key step is extracted from the candidate paths to generate corresponding guidance words, thereby constructing a language-based guidance prompt.

7. The zero-shot visual navigation method based on spatial understanding according to claim 1, characterized in that, The steps involved in determining the optimal leading edge region by S500 include: S510: Input the language-based guidance prompt into the visual language model, perform semantic matching with multiple viewpoint images, calculate the semantic similarity between each viewpoint image and the language-based guidance prompt, and map the semantic similarity onto the traversable area top-down map of the current viewpoint coverage area; S520: If there is an overlap in perspectives, the semantic similarity of the overlapping regions is weighted to obtain a new semantic similarity score of the overlapping regions that can simultaneously reflect the two perspectives and the target. The perspective with the highest similarity is selected as the best frontier region.

8. The zero-shot visual navigation method based on spatial understanding according to claim 7, characterized in that, The maximum value of the similarity score is selected as follows: ； in, Indicates the highest value score; Represents the set of candidate waypoints at time step t; This indicates that the semantic similarity value between the two is calculated. Represents the current viewpoint image; function Convert relationship paths and the agent's current position into verbal guidance prompts; Represents potential paths leading to a target on a spatial relationship map; This indicates the location of the agent in the spatial relationship graph.

9. The zero-shot visual navigation method based on spatial understanding according to claim 1, characterized in that, In the S600, the underlying path planner is trained using a point-to-point reinforcement learning strategy. During the training process, the agent relies on visual observation and continuously optimizes the decision-making process through reinforcement learning algorithms in a simulated environment. It learns the control strategy for moving from the current position to the specified target point, thereby improving the ability to follow paths and avoid obstacles.

10. A zero-shot visual navigation system based on spatial understanding, characterized in that, The system is used to implement the zero-shot visual navigation method based on spatial understanding as described in any one of claims 1 to 9, the system comprising: The input module is used for inputting the target to be searched and acquiring image information; The large language model module is used to process the input target to generate spatial layout priors; The basic visual model module is used to obtain the detection results of the target to be found in the scene through image information; The evaluation and correction module is used to generate spatial layout posterior or target spatial coordinates from the detection results through evaluation and correction. The knowledge fusion module is used to perform hierarchical knowledge fusion between spatial layout priors and spatial layout posteriors in order to generate a spatial knowledge graph. The dynamic programming module is used to generate waypoint coordinates from a spatial knowledge graph; The point navigation strategy module is used to perform low-level path planning on spatial coordinates or waypoint coordinates to obtain an executable and accurate navigation path.

Images