Efficient zero-shot visual language navigation method and system based on spatial perception waypoints

Abstract

The application discloses a kind of efficient zero sample visual language navigation method and system based on spatial perception waypoint, method includes: according to current environment observation data, obtain candidate waypoint set;Based on the candidate waypoint set, obtain abstract visual observation image;According to the abstract visual observation image, natural language navigation instruction and historical execution record, by single call multimodal large language model, obtain high-level action instruction and target space guide information;According to the target space guide information, obtain corresponding global world coordinate target position;According to the global world coordinate target position, obtain collision-free path and drive navigation intelligent agent to execute physical movement, to complete current decision step and enter next decision step.The application reduces the delay and the computing overhead while ensuring the navigation accuracy, meets the real-time deployment requirements of physical robot platform.

Description

Technical Field

[0001] This invention belongs to the field of embodied intelligence and robot navigation technology, and particularly relates to an efficient zero-sample visual language navigation method and system based on spatially perceived waypoints. Background Technology

[0002] The Visual Language Navigation in Continuous Environments (VLN-CE) task primarily aims to enable an embodied agent to navigate in an unknown, continuous environment based on natural language instructions. Existing zero-shot navigation methods mainly rely on repeatedly calling large models in a single decision step or depending on locally pre-trained waypoint predictors. These methods suffer from drawbacks such as high inference latency, high computational cost, and lack of global spatial awareness. Currently, there is a lack of a method for zero-shot visual language navigation that offers fast inference speed, low computational cost, and strong spatial awareness. Summary of the Invention

[0003] To address the aforementioned technical challenges, this invention proposes an efficient zero-sample visual language navigation method and system based on spatially perceptible waypoints. This method introduces an abstract waypoint visualization marking mechanism onto panoramic color images, enabling the navigation agent to acquire specific spatial perception and clear motion memory by calling a multimodal large language model only once in each decision-making step. This ensures navigation accuracy while reducing computational latency and overhead, meeting the real-time deployment requirements of physical robot platforms.

[0004] To achieve the above objectives, this invention provides an efficient zero-shot visual-language navigation method based on spatially aware waypoints, comprising: Obtain natural language navigation instructions and, based on current environmental observation data, obtain a set of candidate waypoints; Based on the candidate waypoint set, an abstract and visualized observation image is obtained; Based on the abstract and visualized observation images, natural language navigation instructions, and historical execution records, high-level action instructions and target space guidance information are obtained through a single call to the multimodal large language model. Based on the target space guidance information, obtain the corresponding global world coordinate target position; Based on the target position in the global world coordinates, a collision-free path is obtained and the navigation agent is driven to perform physical movement to complete the current decision step and proceed to the next decision step.

[0005] Optionally, based on current environmental observation data, the candidate waypoint set can be obtained as follows: Based on the multi-view color images, depth images, and real-time pose data output by the underlying odometry collected at the current time step of the navigation agent, multimodal perception information and physical positioning reference of the environment are obtained. Based on the multimodal perception information and physical positioning reference, a passable area mask is obtained by back-projecting depth information into a three-dimensional physical space and constructing a two-dimensional semantic grid map. Based on the passable region mask, the candidate waypoint set is obtained through topological skeletonization extraction and uniform sampling.

[0006] Optionally, based on the multimodal perception information and physical positioning reference, obtaining a passable area mask by back-projecting depth information into a three-dimensional physical space and constructing a two-dimensional semantic grid map includes: Optionally, based on the passable region mask, the candidate waypoint set is obtained through topological skeletonization extraction and uniform sampling, including: Based on the passable area mask, small gaps are filled by morphological closing operations, and a global topological skeleton with a single pixel width is extracted by the median transformation algorithm. Based on the global topology skeleton, the corridor endpoints and key topology nodes of multi-directional intersections are obtained by calculating the neighborhood connectivity of the skeleton pixels. Based on the key topology nodes and the global topology skeleton, while retaining all key topology nodes, the candidate waypoint set is obtained by uniformly sampling along the topology skeleton at fixed physical distance intervals using a non-maximum suppression strategy.

[0007] Optionally, obtaining an abstract visual observation image based on the candidate waypoint set includes: Map the two-dimensional raster map coordinates of the candidate waypoints in the candidate waypoint set to three-dimensional physical coordinates in the camera coordinate system; Based on the camera's internal parameters and height information, the three-dimensional physical coordinates are projected onto the two-dimensional image plane to obtain the image pixel coordinates; Based on the image pixel coordinates, circular visual markers are rendered on the original multi-view color image and accompanied by unique numerical identifiers. Unvisited candidate waypoints and visited historical waypoints are distinguished by different colors to obtain the abstract visual observation image.

[0008] Optionally, based on the abstract visual observation image, natural language navigation instructions, and historical execution records, obtaining high-level action instructions and target space guidance information through a single call to the multimodal large language model includes: Optionally, the target space guidance information includes: When there is a rendered candidate waypoint in the movement direction corresponding to the advanced action command, obtain the numerical identifier of the target waypoint; When the movement direction corresponding to the advanced action command does not generate waypoint markers due to occlusion or edge conditions, obtain the two-dimensional pixel bounding box of the target drivable area; When the advanced action command is a stop action, the navigation process is terminated.

[0009] Optionally, obtaining the corresponding global world coordinate target position based on the target space guidance information includes: Based on the target space guidance information, the target pixel coordinates are parsed and obtained. Based on the camera's inverse intrinsic parameter matrix and the current real-time two-dimensional pose of the navigation agent, the target pixel coordinates are back-projected and transformed into a two-dimensional global world coordinate system to obtain the global world coordinate target position.

[0010] Optionally, obtaining a collision-free path and driving the navigation agent to perform physical movement based on the target position in global world coordinates includes: Based on the global world coordinate target location and combined with the real-time traversable cost map, the collision-free shortest path from the current location to the global world coordinate target location is calculated using a fast traversal planner. Based on the collision-free shortest path, discrete motion commands are decomposed and obtained, and the chassis motor is driven to perform physical displacement to complete the current decision step and trigger the next round of perception decision cycle.

[0011] The present invention also provides an efficient zero-shot visual language navigation system based on spatially aware waypoints, comprising: a data acquisition module, an abstract representation module, a single large model decision module, and a path execution module; The data acquisition module is used to acquire natural language navigation instructions and obtain a set of candidate waypoints based on current environmental observation data. The abstract representation module is used to obtain an abstract visual observation image based on the candidate waypoint set; The single-call large model decision module is used to obtain advanced action instructions and target space guidance information by calling the multimodal large language model once, based on the abstract visual observation image, natural language navigation instructions and historical execution records. The path execution module is used to obtain the corresponding global world coordinate target position based on the target space guidance information; obtain a collision-free path based on the global world coordinate target position and drive the navigation agent to perform physical movement to complete the current decision step and enter the next decision step.

[0012] Compared with the prior art, the present invention has the following advantages and technical effects: This invention limits the number of queries for a multimodal large model to only once per step, reducing computational overhead and significantly lowering the average single-step inference time compared to existing structure optimization-based navigation methods.

[0013] By overlaying abstract waypoint representations onto RGB images, concrete spatial awareness and explicit spatial memory are provided to zero-sample agents that lack underlying metric space capabilities. Attached Figure Description

[0014] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of an efficient zero-shot visual language navigation method based on spatially aware waypoints according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the execution flow of the structured waypoint generation and abstract representation module according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the input-output interaction logic of the large model decision module in an embodiment of the present invention. Detailed Implementation

[0015] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0016] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0017] This embodiment proposes an efficient zero-shot visual-language navigation method based on spatially aware waypoints, such as... Figure 1 As shown, the specific steps include: Obtain natural language navigation instructions and, based on current environmental observation data, obtain a set of candidate waypoints; Based on the candidate waypoint set, an abstract and visualized observation image is obtained; Based on the abstract and visualized observation images, natural language navigation instructions, and historical execution records, high-level action instructions and target space guidance information are obtained through a single call to the multimodal large language model. Based on the target space guidance information, obtain the corresponding global world coordinate target position; Based on the target position in the global world coordinates, a collision-free path is obtained and the navigation agent is driven to perform physical movement to complete the current decision step and proceed to the next decision step.

[0018] Furthermore, based on current environmental observation data, the candidate waypoint set includes: Based on the multi-view color images, depth images, and real-time pose data output by the underlying odometry collected at the current time step of the navigation agent, multimodal perception information and physical positioning reference of the environment are obtained. Based on the multimodal perception information and physical positioning reference, a passable area mask is obtained by back-projecting depth information into a three-dimensional physical space and constructing a two-dimensional semantic grid map. Based on the passable region mask, the candidate waypoint set is obtained through topological skeletonization extraction and uniform sampling.

[0019] Specifically, acquiring real-time environmental observation data includes: at the current time step, the navigation agent acquires real-time color and depth images in four directions centered on itself, and simultaneously reads the real-time pose data output by the underlying odometry. This provides basic multimodal perception information and real-time physical positioning benchmarks for subsequent scene understanding and path planning.

[0020] Furthermore, based on the aforementioned multimodal perception information and physical positioning reference, the traversable area mask is obtained by back-projecting depth information into a three-dimensional physical space and constructing a two-dimensional semantic grid map, including: Furthermore, based on the passable region mask, the candidate waypoint set is obtained through topological skeletonization extraction and uniform sampling, including: Based on the passable area mask, small gaps are filled by morphological closing operations, and a global topological skeleton with a single pixel width is extracted by the median transformation algorithm. Based on the global topology skeleton, the corridor endpoints and key topology nodes of multi-directional intersections are obtained by calculating the neighborhood connectivity of the skeleton pixels. Based on the key topology nodes and the global topology skeleton, while retaining all key topology nodes, the candidate waypoint set is obtained by uniformly sampling along the topology skeleton at fixed physical distance intervals using a non-maximum suppression strategy.

[0021] Furthermore, based on the candidate waypoint set, obtaining the abstract visual observation image includes: Map the two-dimensional raster map coordinates of the candidate waypoints in the candidate waypoint set to three-dimensional physical coordinates in the camera coordinate system; Based on the camera's internal parameters and height information, the three-dimensional physical coordinates are projected onto the two-dimensional image plane to obtain the image pixel coordinates; Based on the image pixel coordinates, circular visual markers are rendered on the original multi-view color image and accompanied by unique numerical identifiers. Unvisited candidate waypoints and visited historical waypoints are distinguished by different colors to obtain the abstract visual observation image.

[0022] Specifically, structured waypoint generation and abstract representation includes: extracting a sparse set of candidate waypoints from the input depth data using a structured waypoint generation module, and directly projecting and rendering these waypoints onto a color image through coordinate transformation to generate a visually represented observation image containing intuitive visual markers. This step specifically includes: 1. Mapping and Walkable Region Extraction: Combining the camera's internal parameter matrix and the real-time pose of the navigation agent, the depth information of the current frame is back-projected to restore the 3D physical space, thereby constructing a 2D top-down semantic grid map for identifying obstacles and walkable areas in the environment. Next, semantic labels are filtered to extract binary masks representing only purely walkable areas. Simultaneously, the set of waypoints reached is incrementally updated using real-time pose. If the Euclidean distance between a newly detected location and an existing waypoint is less than a preset threshold, it is merged into the nearest existing waypoint, ensuring a compact trajectory history and a simplified structure.

[0023] 2. Topology Skeletonization and Waypoint Selection: Morphological closing operations are performed on the extracted binary mask of purely walkable regions to fill small gaps. Then, a global topology skeleton with a single pixel width is extracted using a median transformation algorithm. By calculating the neighborhood connectivity of each pixel on the skeleton, key topology nodes such as corridor endpoints and multi-way intersections are explicitly identified. To maintain the sparsity of the decision space and reduce the visual processing load, a non-maximum suppression strategy is applied along the topology skeleton while retaining all key topology nodes. Uniform sampling is performed at set spatial physical distance intervals, and finally, a final set of candidate waypoints is extracted. This method ensures complete coverage of the environmental geometry and topology while maintaining the sparsity of candidate points.

[0024] 3. Abstract Waypoint Representation and Rendering: The selected 2D raster map waypoint coordinates, combined with map resolution, origin offset, and current pose, are transformed and mapped into 3D physical coordinates in the camera coordinate system. Next, using camera intrinsic parameters and altitude information, these 3D points are projected onto the 2D image plane to obtain specific image pixel coordinates. Finally, these valid waypoints are rendered as visual markers and overlaid on the original four-way color image, with each marker accompanied by a unique numerical identifier. To provide clear spatial memory, unvisited candidate waypoints are strictly distinguished from historically visited waypoints by color; for example, unvisited points are marked with cyan dots, and visited points are marked with red dots.

[0025] Furthermore, based on the abstract visual observation image, natural language navigation instructions, and historical execution records, obtaining high-level action instructions and target space guidance information through a single call to the multimodal large language model includes: Further, the target space guidance information includes: When there is a rendered candidate waypoint in the movement direction corresponding to the advanced action command, obtain the numerical identifier of the target waypoint; When the movement direction corresponding to the advanced action command does not generate waypoint markers due to occlusion or edge conditions, obtain the two-dimensional pixel bounding box of the target drivable area; When the advanced action command is a stop action, the navigation process is terminated.

[0026] Specifically, the single large-scale model decision-making process involves employing a single multimodal large-scale language model as the robot's high-level planner, performing only one question-and-answer call at each decision step. The contextual information input to the multimodal large-scale language model includes: the initial natural language navigation instructions, the currently rendered four-way visualization image with color markers and numeric identifiers, a sliding window execution history consisting of the thoughts and actions of the previous few historical steps (when the system detects repeated visits to the same area via the access counter, it automatically adds defensive cyclical warning text to the history), and a list of semantic object categories identified in the four-way view using an object detection model. Based on the above inputs, the multimodal large-scale language model performs unified reasoning and progress analysis, predicting and outputting high-level action instructions and spatial targets. High-level action instructions include moving forward, backward, left, right, or stopping. If a rendered candidate waypoint exists in the selected direction of movement, the model directly outputs the numeric identifier of the target waypoint; if the selected direction fails to generate waypoint markers due to occlusion or edge conditions, the large model generates a two-dimensional pixel bounding box of the target drivable area as a complementary spatial target guide. When the model determines that the navigation task is successful and outputs a stop action, the entire navigation process is successfully terminated.

[0027] Furthermore, based on the target space guidance information, obtaining the corresponding global world coordinate target position includes: Based on the target space guidance information, the target pixel coordinates are parsed and obtained. Based on the camera's inverse intrinsic parameter matrix and the current real-time two-dimensional pose of the navigation agent, the target pixel coordinates are back-projected and transformed into a two-dimensional global world coordinate system to obtain the global world coordinate target position.

[0028] Furthermore, obtaining a collision-free path and driving the navigation agent to perform physical movement based on the global world coordinate target position includes: Based on the global world coordinate target location and combined with the real-time traversable cost map, the collision-free shortest path from the current location to the global world coordinate target location is calculated using a fast traversal planner. Based on the collision-free shortest path, discrete motion commands are decomposed and obtained, and the chassis motor is driven to perform physical displacement to complete the current decision step and trigger the next round of perception decision cycle.

[0029] Specifically, the underlying control and path execution include: after the agent extracts the target pixel coordinates given by the multimodal large language model (directly derived from the coordinates corresponding to the selected waypoint identifier, or the bottom center pixel coordinates of the bounding box), it uses the camera's inverse intrinsic parameter matrix and the robot's current 2D pose to back-project and transform the target pixel back into the 2D global world coordinate system. Subsequently, the fast travel planner of the underlying controller, combined with the real-time traversable cost map, calculates the collision-free shortest path from the current position to the target in the world coordinate system. This path is further decomposed into a series of discrete motion commands that can be directly executed by the chassis motors, driving the robot to perform actual physical movement, thereby completing the current perception-decision loop and entering the next time step.

[0030] This embodiment also provides an efficient zero-shot visual language navigation system based on spatially aware waypoints, including: a data acquisition module, an abstract representation module, a single large model decision module, and a path execution module; The data acquisition module is used to acquire natural language navigation instructions and obtain a set of candidate waypoints based on current environmental observation data. The abstract representation module is used to obtain an abstract visual observation image based on the candidate waypoint set; The single-call large model decision module is used to obtain advanced action instructions and target space guidance information by calling the multimodal large language model once, based on the abstract visual observation image, natural language navigation instructions and historical execution records. The path execution module is used to obtain the corresponding global world coordinate target position based on the target space guidance information; obtain a collision-free path based on the global world coordinate target position and drive the navigation agent to perform physical movement to complete the current decision step and enter the next decision step.

[0031] Specifically, the overall system consists of a real-time environmental observation data acquisition module, a structured waypoint generation and abstract representation module, a single large-scale model decision-making module, and a low-level control and path execution module. This method is primarily applied to embodied intelligent navigation tasks in unknown continuous environments and can be deployed on service robots, quadruped robots, or other physical platforms.

[0032] The system input includes real-time four-way color and depth images collected by the navigation agent at the current time step, as well as real-time two-dimensional pose data output by the underlying odometry. Color and depth images can be acquired by an RGB-D camera, and the four-way view usually covers the front, back, left, and right directions to provide a panoramic view. At the same time, it receives natural language navigation instructions given by humans. The overall execution process includes the following main steps: (1) Collect multimodal visual information and physical pose using the real-time environmental observation data acquisition module; (2) Extract sparse candidate waypoints through the structured waypoint generation and abstract representation module, and generate abstract observation images with visual markers; (3) Use the single large model decision module to perform single reasoning by integrating natural language instructions, abstract observation images, historical execution records, and semantic object lists, and output high-level action instructions and target space guidance information; (4) Through the underlying control and path execution module, convert the target space guidance information into physical coordinates, plan a collision-free path, and drive the agent to move.

[0033] Implement environmental observation data acquisition module: Multimodal data acquisition implementation: Using a sensor array mounted on the intelligent agent, real-time color and depth images of the surrounding environment from four perspectives: front, back, left, and right. This image data provides rich visual texture and geometric depth information.

[0034] Real-time pose reading implementation: Synchronously read the odometer data of the intelligent agent's chassis to obtain its real-time two-dimensional pose in the current environment, which serves as the basic physical reference for building a global map and path planning.

[0035] The execution flow diagram of the structured waypoint generation and abstract representation module is as follows: Figure 2 As shown, it specifically includes: Mapping and Accessible Area Extraction: Based on the camera's intrinsic parameter matrix and the agent's real-time pose, depth pixels in the depth image are back-projected into a 3D physical space. These 3D point clouds are then projected downwards to construct a top-down 2D semantic grid map. By filtering semantic labels representing obstacles such as walls and furniture, a binary mask of purely passable areas is extracted. Simultaneously, the set of waypoints reached is incrementally updated using real-time pose, and the Euclidean distance between new locations and existing waypoints is calculated and a threshold is set for merging, ensuring the compactness of historical trajectories.

[0036] Topological skeletonization and waypoint selection: Morphological closing operations are performed on the passable region mask to fill small holes. Next, a global topological skeleton with a single-pixel width is extracted using an axial transformation algorithm. By calculating the neighborhood connectivity of the skeleton pixels, key nodes such as corridor endpoints and multi-directional intersections are identified. To reduce visual redundancy while maintaining spatial representativeness, a non-maximum suppression strategy is applied along the topological skeleton, with uniform sampling at fixed physical distance intervals, while retaining all key topological nodes, ultimately obtaining a sparse set of candidate waypoints.

[0037] Abstract waypoint representation and rendering: Combining map resolution, origin offset, and the agent's current pose, the selected 2D waypoint coordinates are mapped to 3D coordinates in the camera coordinate system. Then, using camera intrinsics and altitude information, these coordinates are projected onto the 2D image plane. Finally, dots are overlaid as visual markers at the corresponding locations in the original four-way color image, each accompanied by a unique numerical identifier. To establish clear spatial memory, unvisited candidate waypoints and traversed historical waypoints are rendered differently (e.g., cyan and red) to distinguish them, generating the final abstract observation image for use in large-scale model inference.

[0038] The input-output interaction logic diagram of a single large model decision module is shown below. Figure 3 As shown: Context information construction: The system integrates human natural language commands, a four-way abstract observation image rendered with colored dots and numbers, a sliding window of thought and action history data from several past time steps, and a list of semantic object categories identified in the four-way view by an object detection model, into a unified prompt context. Specifically, when the system detects repeated access to the same area by the agent, it automatically adds defensive loop warning text to the history to prevent the model from entering an infinite loop.

[0039] Single Reasoning and Decision Output Module: The integrated contextual information is input into the multimodal large language model only once. By comprehensively analyzing the current progress, visual scene, historical trajectory, and semantic goal, the model directly outputs the next high-level action instruction (forward, backward, left, right, or stop) and the corresponding goal guidance information.

[0040] Dynamic target output and termination mechanism: If a rendered candidate waypoint exists in the selected direction of movement, the model directly outputs the numerical identifier of that waypoint. If the selected direction fails to generate an accurate waypoint marker due to occlusion or edge conditions, the model outputs a 2D pixel bounding box of the drivable area of ​​the target as a backup spatial target guide. When the model determines that the task is complete and outputs a "stop" action, the navigation process successfully terminates.

[0041] Low-level control and path execution module: Target pixel extraction: The output of the large model is analyzed. If it is a numeric identifier, the corresponding pixel position on the color image is found. If it is a bounding box, the bottom center pixel position of the bounding box is extracted and used as the two-dimensional image target for the current decision step.

[0042] Coordinate back projection and global positioning: By using the camera's inverse intrinsic parameter matrix and the agent's current real-time 2D pose, the extracted target pixel coordinates are back-projected back into the real 2D global world coordinate system to determine the absolute physical location that the agent needs to go to.

[0043] Path planning and motion execution: The underlying control system invokes a rapid travel method, combined with a real-time traversable cost map, to calculate the shortest, collision-free continuous trajectory from the current position to the target at global world coordinates. This path is further decomposed into a series of discrete motor drive commands, controlling the chassis to perform actual physical displacements, completing the current time step, and triggering the next round of perception and decision-making loop.

[0044] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An efficient zero-shot visual-language navigation method based on spatially aware waypoints, characterized in that, include: Obtain natural language navigation instructions and, based on current environmental observation data, obtain a set of candidate waypoints; Based on the candidate waypoint set, an abstract and visualized observation image is obtained; Based on the abstract visual observation image, the natural language navigation instructions, and the historical execution record, high-level action instructions and target space guidance information are obtained by a single call to the multimodal large language model; Based on the target space guidance information, obtain the corresponding global world coordinate target position; Based on the target position in the global world coordinates, a collision-free path is obtained and the navigation agent is driven to perform physical movement to complete the current decision step and proceed to the next decision step.

2. The efficient zero-shot visual-language navigation method based on spatially aware waypoints according to claim 1, characterized in that, Based on current environmental observation data, the candidate waypoint set includes: Based on the multi-view color images, depth images, and real-time pose data output by the underlying odometry collected at the current time step of the navigation agent, multimodal perception information and physical positioning reference of the environment are obtained. Based on the multimodal perception information and physical positioning reference, a passable area mask is obtained by back-projecting depth information into a three-dimensional physical space and constructing a two-dimensional semantic grid map. Based on the passable region mask, the candidate waypoint set is obtained through topological skeletonization extraction and uniform sampling.

3. The efficient zero-shot visual-language navigation method based on spatially aware waypoints according to claim 2, characterized in that, Based on the multimodal sensing information and physical positioning benchmark, the passable area mask is obtained by back-projecting depth information into a three-dimensional physical space and constructing a two-dimensional semantic grid map, including: Based on the internal parameter matrix of the camera, which serves as the physical positioning reference, and the real-time pose of the navigation agent, the depth pixels in the depth image of the multimodal perception information are back-projected to the three-dimensional physical space, and projected downwards to construct a top-down two-dimensional semantic grid map. Based on the two-dimensional semantic grid map, a passable area mask is obtained by filtering obstacle semantic labels.

4. The efficient zero-shot visual-language navigation method based on spatially aware waypoints according to claim 2, characterized in that, Based on the passable region mask, the candidate waypoint set is obtained through topological skeletonization extraction and uniform sampling, including: Based on the passable area mask, small gaps are filled by morphological closing operations, and a global topological skeleton with a single pixel width is extracted by the median transformation algorithm. Based on the global topology skeleton, the corridor endpoints and key topology nodes of multi-directional intersections are obtained by calculating the neighborhood connectivity of the skeleton pixels. Based on the key topology nodes and the global topology skeleton, while retaining all key topology nodes, the candidate waypoint set is obtained by uniformly sampling along the topology skeleton at fixed physical distance intervals using a non-maximum suppression strategy.

5. The efficient zero-shot visual-language navigation method based on spatially aware waypoints according to claim 2, characterized in that, Based on the candidate waypoint set, obtaining the abstract visual observation image includes: Map the two-dimensional raster map coordinates of the candidate waypoints in the candidate waypoint set to three-dimensional physical coordinates in the camera coordinate system; Based on the camera's internal parameters and height information, the three-dimensional physical coordinates are projected onto the two-dimensional image plane to obtain the image pixel coordinates; Based on the image pixel coordinates, circular visual markers are rendered on the original multi-view color image and accompanied by unique numerical identifiers. Unvisited candidate waypoints and visited historical waypoints are distinguished by different colors to obtain the abstract visual observation image.

6. The efficient zero-shot visual-language navigation method based on spatially aware waypoints according to claim 2, characterized in that, Based on the abstract visual observation image, natural language navigation instructions, and historical execution records, high-level action instructions and target space guidance information are obtained through a single call to the multimodal large language model, including: Based on the abstract visual observation image, natural language navigation instructions, historical execution records, and the list of semantic object categories obtained by recognizing the multi-view color image through the object detection model, a unified prompt word context is constructed. Based on the context of the prompt words, a unified reasoning process is performed by calling a multimodal large language model in a single instance to obtain advanced action instructions and target space guidance information; Specifically, when a navigation agent is detected to have repeatedly accessed the same area, a defensive cyclical warning text is added to the history.

7. The efficient zero-shot visual-language navigation method based on spatially aware waypoints according to claim 6, characterized in that, The target space guidance information includes: When there is a rendered candidate waypoint in the movement direction corresponding to the advanced action command, obtain the numerical identifier of the target waypoint; When the movement direction corresponding to the advanced action command does not generate waypoint markers due to occlusion or edge conditions, obtain the two-dimensional pixel bounding box of the target drivable area; When the advanced action command is a stop action, the navigation process is terminated.

8. The efficient zero-shot visual-language navigation method based on spatially aware waypoints according to claim 1, characterized in that, Based on the target space guidance information, the corresponding global world coordinate target position is obtained, including: Based on the target space guidance information, the target pixel coordinates are parsed and obtained. Based on the camera's inverse intrinsic parameter matrix and the current real-time two-dimensional pose of the navigation agent, the target pixel coordinates are back-projected and transformed into a two-dimensional global world coordinate system to obtain the global world coordinate target position.

9. The efficient zero-shot visual-language navigation method based on spatially aware waypoints according to claim 1, characterized in that, Based on the target position in the global world coordinates, obtaining a collision-free path and driving the navigation agent to perform physical movement includes: Based on the global world coordinate target location and combined with the real-time traversable cost map, the collision-free shortest path from the current location to the global world coordinate target location is calculated using a fast traversal planner. Based on the collision-free shortest path, discrete motion commands are decomposed and obtained, and the chassis motor is driven to perform physical displacement to complete the current decision step and trigger the next round of perception decision cycle.

10. A high-efficiency zero-shot visual-language navigation system based on spatially aware waypoints, used to implement the method as described in any one of claims 1-9, characterized in that, include: Data acquisition module, abstract representation module, single large model decision module, and path execution module; The data acquisition module is used to acquire natural language navigation instructions and obtain a set of candidate waypoints based on current environmental observation data. The abstract representation module is used to obtain an abstract visual observation image based on the candidate waypoint set; The single-call large model decision module is used to obtain advanced action instructions and target space guidance information by calling the multimodal large language model once, based on the abstract visual observation image, natural language navigation instructions and historical execution records. The path execution module is used to obtain the corresponding global world coordinate target position based on the target space guidance information; obtain a collision-free path based on the global world coordinate target position and drive the navigation agent to perform physical movement to complete the current decision step and enter the next decision step.

Images