An outdoor geographical semantic retrieval enhanced navigation and search method and system

Abstract

The application relates to an outdoor geographical semantic retrieval enhanced navigation and search method and system, which comprises the following steps: receiving a natural language instruction, and extracting navigation target semantic elements and search target semantic elements; under the constraint of a structured index, performing beam search, and outputting the latitude and longitude and boundary information of the entity; decomposing the natural language instruction into a subtask sequence, and under the constraint of a predefined executable action and condition space, generating a structured behavior tree; calling a routing engine, generating a global topology path and a waypoint sequence based on the nearest reachable boundary point of the entity; based on the scheduling of the structured behavior tree, performing real-time navigation along the waypoint sequence; after reaching the vicinity of the target, searching for the target related to the search target semantic elements and performing clustering positioning under the constraint of a polygon boundary, obtaining a final target point and converging navigation. Compared with the prior art, the application has the advantages of high positioning accuracy and wide application scenarios.

Description

Technical Field

[0001] This invention relates to the field of visual language navigation technology, and in particular to a retrieval-enhanced navigation and search method and system for outdoor geographic semantics. Background Technology

[0002] In recent years, language-guided navigation has evolved from goal-conditional strategies to end-to-end visual / visual-language-action frameworks, driven by basic models. Attempts at long-distance outdoor navigation combining topological maps or aerial cues have emerged. However, these methods largely rely on local visual correspondence and short-term memory, lacking explicit geographic semantic implementation. They struggle to reliably map abstract commands like "near the library / cafeteria / hotel" into executable geographic targets, and are susceptible to repetitive scenes and lighting changes in large-scale open environments, resulting in insufficient reliability. To address this, some works introduce geographic data such as OSM and employ a layered architecture to decouple task planning from underlying control, even using large models for geographic question answering / retrieval. However, many solutions rely on cloud-based inference, leading to network dependencies and latency issues. Furthermore, free-generative spatial inference may generate unverifiable locations or entities (inconsistent with the actual OSM), rendering downstream path planning targets invalid. They also have limited coverage of open semantic targets such as "finding a place for lunch." In the "last mile" stage after reaching the target area, traditional frontier navigation... While exploration methods can expand maps, they are semantically blind and struggle to perform semantically driven searches based on language objectives. Existing semantic exploration / target navigation methods are often limited by cloud-based reasoning, indoor settings, or edge computing power, lacking a stable closed loop of "semantic mapping - candidate aggregation - target localization - navigation convergence." Consequently, they are unable to reliably complete the complex command of "reaching a building and locating a specific target object" in an open outdoor world.

[0003] The invention disclosed in CN116737899A presents a visual natural language navigation method and system based on common sense information assistance. This invention draws on the idea of ​​semantic reasoning, extracts features from the input instruction information, and uses these features to reason and add common sense information as additional input for decision-making. Utilizing a cross-modal Transformer, it aggregates the input language instructions, environmental input, and extended information to obtain target prediction points for agent action control, thereby improving the accuracy of visual natural language navigation. It solves the problem in existing technologies that "the agent cannot truly understand the relationship between semantic information in language and the actual scene." However, this solution is difficult to apply in the limited outdoor open world and struggles to achieve accurate navigation and ultimately find the target based on complex instruction requirements. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a retrieval enhancement navigation and search method and system for outdoor geographic semantics.

[0005] The objective of this invention can be achieved through the following technical solutions: An enhanced navigation and search method for outdoor geographic semantics, comprising the following steps: The system receives natural language instructions and extracts navigation target semantic elements and search target semantic elements from the instructions. Under the constraints of a pre-built structured index, it performs beam search to align the navigation target semantic elements to unique entities in the local knowledge base and outputs the latitude, longitude, and boundary information of the navigation target. The local knowledge base is a local geographic entity database. The natural language instructions are decomposed into a sequence of subtasks, and a structured behavior tree is generated based on the subtask sequences and the latitude, longitude and boundary information, under predefined executable actions and conditional space constraints. The navigation nodes in the structured behavior tree call the routing engine to generate a global topology path and waypoint sequence based on the nearest reachable boundary point of the entity. Multimodal sensor data is collected in real time, and a semantic voxel map is constructed based on the multimodal sensor data; real-time navigation is performed along the waypoint sequence based on the scheduling of the structured behavior tree; after reaching the vicinity of the target, targets related to the semantic elements of the search target are searched in the semantic voxel map with polygon boundaries as constraints, and clustered and located to obtain the final target point and converge navigation.

[0006] Furthermore, the process of building a structured index specifically includes: For each map entity in the local knowledge base, an entity ID is created by concatenating a low-level identifier and a high-level identifier, and a structured index is built based on all entity IDs.

[0007] Furthermore, the beam search process under the constraints of the structured index specifically includes: Chain-thinking prompts are used to extract structured clues from the semantic elements of the navigation target, including target categories and modifiers, as retrieval priors to narrow the search scope; Based on the retrieval prior, multiple semantic units are generated sequentially, and the generation of each semantic unit is constrained by a pre-built structured index: selection is only allowed from the set of legal child nodes in the structured index, and the probability of semantic units belonging to other child node sets is set to zero; the set of legal child nodes consists of semantic units that actually exist in the local knowledge base; the generated semantic units are concatenated to obtain the unique target entity ID, and a search is performed in the local knowledge base to obtain the latitude, longitude, and boundary information of the entity.

[0008] Furthermore, during the generation of the global topology path and waypoint sequence, the structured behavior tree periodically detects the status of the navigation nodes. If a blockage, planning failure, or prolonged inactivity occurs, a recovery strategy is triggered to reinitialize the path search and update the path planning until the navigation task is completed.

[0009] Furthermore, the multimodal sensor data specifically includes lidar point cloud data, inertial measurement unit data, RGB images, and depth image data; The real-time navigation process specifically includes: generating real-time positioning results based on the multimodal sensor data using a tightly coupled optimization algorithm; performing coordinate transformation based on the global waypoint sequence and the real-time positioning results to convert the waypoint coordinates in the global coordinate system to the local coordinate system, and adaptively sampling the waypoint sequence according to the path curvature to generate a control point sequence; finally, performing motion control based on the control point sequence until reaching the vicinity of the target entity boundary.

[0010] Furthermore, the coordinate transformation process specifically includes: After the GPS signal stabilizes, travel a preset distance along a straight line and record the positions of the starting and ending points; calculate the global heading vector and heading offset angle based on the positions of the starting and ending points, using the following formula: in, The coordinates of the endpoint are... The coordinates of the starting position, Indicates displacement in the north direction. Indicates displacement in the eastward direction; A rotation matrix is ​​constructed using the aforementioned heading offset angle, and the coordinates of the waypoint sequence in the global coordinate system are transformed to the local coordinate system. The transformation formula is as follows: ; in, These are the waypoint sequence coordinates in the local coordinate system. The coordinates of the waypoint sequence in the global coordinate system; This indicates that a vector in the global coordinate system is transformed to the local coordinate system.

[0011] Furthermore, adaptive sampling specifically includes: If the curvature is greater than a preset threshold, sampling is performed once every shorter path; if the curvature is less than a preset threshold, sampling is performed once every longer path, in order to improve corner tracking and reduce the computational load on straight sections.

[0012] Furthermore, the process of constructing the polygon boundary specifically includes: An initial exploration boundary is generated based on the semantic voxel map. If no target matching the semantic element of the search target is found in the initial exploration boundary, candidate boundary points are extracted from the occupied grid of the semantic voxel map, and polygon boundaries are extracted from the boundary information of the entity. Only candidate boundary points located within the polygon boundaries are retained to constrain the search range.

[0013] Furthermore, the clustering and localization process specifically includes: In the semantic voxel map, highly relevant voxels exceeding a pre-set similarity threshold are detected. When a highly relevant voxel is detected, the exploration is interrupted, and a density-based spatial clustering algorithm is used to aggregate adjacent highly relevant voxels. The cluster containing the most voxels is selected as the master cluster, the centroid coordinates of the master cluster are calculated and projected onto the ground to obtain the final two-dimensional target point and converge navigation.

[0014] The present invention also provides a system for an enhanced navigation and search method for outdoor geographic semantics, comprising a memory and a processor, wherein the memory stores a computer program, and the processor invokes the computer program to execute the steps of any of the methods described above.

[0015] Compared with the prior art, the present invention has the following advantages: (1) This invention proposes a generative geographic retrieval mechanism based on prefix tree constraints. Under the constraints of a pre-built structured index (prefix tree index), beam search is performed to ensure that the generated entity identifiers are always located within the real map entity set. This mechanism avoids generating false or invalid locations, making the geographic retrieval results verifiable and executable. It also ensures that downstream path planning always obtains legitimate targets, thus solving the problems of unreliable generation and format disorder in the process of aligning natural language to geographic entities.

[0016] (2) This invention proposes a region constraint exploration and three-dimensional semantic clustering localization method for the last mile. After reaching the target area, the boundary polygon constraint exploration is used to prevent search divergence. The target localization is achieved based on three-dimensional voxel semantic fusion and density clustering. This avoids the spatial information collapse and target loss problems of traditional two-dimensional projection, improves the stability and success rate of target search in complex outdoor environments, and the designed system has completed the closed-loop verification of long-distance navigation and open vocabulary target search on a real unmanned vehicle platform, verifying the engineering feasibility and scalability of the system.

[0017] (3) This invention proposes a local offline task planning and execution monitoring method, which restricts high-level task planning to a predefined action and condition space and generates a structured behavior tree as an executable task blueprint. At the same time, it introduces a status monitoring and automatic recovery mechanism during execution, realizes retrying when encountering obstacles and dynamic replanning, significantly reduces the dependence on external networks and cloud-based large models, reduces task delays and uncontrollable risks, and improves the robustness of the system in weak network or no network environment. Attached Figure Description

[0018] Figure 1 This is a flowchart of an outdoor geographic semantic retrieval enhancement navigation and search method provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0020] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0021] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0022] Definitions: OSM (Object-Related Mapping Model) is a global open-source collaborative map database and project. Its map data is collected, edited, and maintained by volunteers worldwide and is freely available to the public under the Open Data License (ODbL). OSM provides rich vector geographic data, including roads, buildings, natural features, and points of interest, supporting highly customizable map rendering and application development. In this invention, OSM serves as the primary data source for the underlying geographic knowledge base. Its derived entity, topological relationships, and coordinate information are structured and used to construct a locally queryable set of map entities and a path planning network, providing fundamental geographic support for geographic semantic retrieval in natural language commands and global robot navigation.

[0023] GeoQA is a retrieval-enhanced geographic question-answering module. Its core function is to accurately align semantic elements of navigation targets (such as "library") in natural language to unique entities in the local geographic knowledge base. By constructing hierarchical entity identifiers and introducing a generative retrieval mechanism with prefix tree constraints, GeoQA imposes hard constraints on each semantic unit during the decoding process, ensuring that the output result is a legitimate entity existing in the knowledge base, thereby avoiding the generation of false or invalid locations. This module ultimately outputs the latitude and longitude coordinates and geometric boundary information of the matched entities, providing reliable and verifiable geographic landing points for downstream path planning and task execution. It is a key technical component for realizing the structured mapping from natural language instructions to spatial locations.

[0024] RGB-D: RGB-D is an image or data stream that simultaneously contains color information (RGB) and depth information (Depth). Its RGB channels provide red, green, and blue primary color information, forming a color visual image; the D channel uses a depth sensor (such as structured light, time-of-flight, or stereo vision) to acquire the distance information between each pixel in the scene and the camera, forming a depth map. In this invention, the RGB-D camera, as one of the key sensors for multimodal perception, uses its depth data to construct a 3D environmental geometric model in real time, enhancing the accuracy of localization and mapping; its color images are input into an open-vocabulary visual model to achieve pixel-level semantic understanding and target search, thereby providing dense spatial and semantic information support for the construction of semantic voxel maps and last-mile visual exploration.

[0025] GNSS: A general term for satellite positioning systems including GPS (US), BeiDou (China), GLONASS (Russia), and Galileo (EU), which achieve three-dimensional position, velocity, and time calculations globally by receiving signals from multiple satellites. In this invention, the GNSS receiver provides absolute geographic coordinates (latitude, longitude, and elevation) for robots in large-scale outdoor environments, used for target positioning in global path planning, heading initialization calibration, and global reference in multi-sensor fusion. By tightly or loosely coupling with local odometry (such as LiDAR-IMU odometry), GNSS effectively constrains accumulated errors, improving global consistency and robustness in long-distance navigation, especially providing robots with a stable and absolute position reference in open environments.

[0026] RAPPER is a retrieval-enhanced executable task planner deployed locally on the robot. It transforms natural language instructions into a structured, monitorable, and reversible execution flow. This module first decomposes the instructions into an ordered sequence of subtasks. Combining this with geo-entity information provided by GeoQA, and under predefined action and conditional space constraints, it generates a hierarchical behavior tree representation. By invoking the local routing engine, RAPPER calculates the global topology path from the starting point to the reachable boundary points of the target entity and adaptively sampled waypoint sequences. It monitors the status in real time during task execution, triggering retries, recovery, or dynamic replanning when encountering blockages, deviations, or failures. This achieves offline, low-latency, and highly reliable task planning and execution control, reducing reliance on external networks and large cloud models.

[0027] NæVIS is a navigation and last-mile visual exploration execution system responsible for the complete execution loop of the robot, from long-distance path tracking to refined search of the target area. This system integrates multimodal sensors such as LiDAR, RGB-D, IMU, and GNSS, achieving real-time localization and state estimation through laser inertial odometry, and simultaneously constructing an open-vocabulary semantic voxel map. During the long-distance navigation phase, NæVIS accurately follows the global path to the target area through global-local coordinate bridging and curvature adaptive waypoint tracking technology. In the last-mile phase, the system switches to exploration mode, exploring the frontier constrained by the target entity boundary. Simultaneously, it uses a 3D semantic voxel map to retrieve and search for voxels related to the target's semantic elements, locating the target's spatial position through density clustering, and finally navigating to the target point to complete the task. This achieves seamless transition and robust execution from large-scale movement to refined search.

[0028] Behavior Tree (XML): A hierarchical, modular control structure used to describe the decision-making logic and task execution flow of intelligent agents (such as robots). It organizes behavior nodes in a tree structure, including action nodes (performing specific actions), condition nodes (detecting states), and control nodes (sequential, selection, parallel, etc.). Node execution and state propagation are driven by a top-down, periodic "tick" mechanism. In this invention, the RAPPER module transforms the parsed natural language task into a standardized XML-formatted behavior tree, where each node maps to a predefined executable action or state condition. This transforms high-level task instructions into a structured flow that can be directly parsed, executed, and monitored by the underlying system. It also supports runtime state monitoring, failure recovery, and dynamic replanning, ensuring the reliability, interpretability, and maintainability of task execution.

[0029] polygon: A polygon is a closed two-dimensional geometric shape enclosed by a series of ordered vertices, commonly used in Geographic Information Systems (GIS) and robot navigation to represent the boundary of a region or object. In this invention, "polygon" specifically refers to the outline boundary of a geographic entity (such as a building or park) extracted from the OSM knowledge base, typically represented by a sequence of latitude and longitude coordinates. This polygonal boundary is used in navigation planning to calculate the nearest reachable point of the entity, avoiding setting the target point in an inaccessible internal region. In the last-mile exploration phase, it serves as a hard spatial constraint, limiting the robot's search range and ensuring that the exploration behavior does not exceed the target area, thereby improving search efficiency and task safety. Through polygonal constraints, the system achieves a natural transition and reliable execution from point target navigation to regional target search.

[0030] Trie: This is a tree-like data structure for efficiently storing and retrieving string sets. Each node corresponds to a character, the path from the root node to any node represents a string prefix, and the complete path to a leaf node corresponds to a complete string. In this invention, a Trie is used as the core implementation of the structured index. The system pre-constructs a Trie with hierarchical identifiers of all geographic entities in the local knowledge base (such as "teaching area - building A"). During the generative geographic retrieval process, by applying a hard mask constraint based on the Trie to the candidate character set at each decoding step, it is ensured that the generated entity identifiers necessarily correspond to real entries in the knowledge base. This mechanism prevents the generation of invalid or fictitious locations, ensuring the reliability, uniqueness, and downstream executability of the geographic retrieval results.

[0031] token: A token is the smallest discrete unit in natural language processing and sequence generation, typically obtained by segmenting text at the word, sub-word, or character level using a tokenizer. Each token corresponds to an integer index in the vocabulary. In this invention, a token refers to each generation unit of the entity ID string in the generative geographic retrieval process. For example, the entity ID "Teaching Area-Building A" can be segmented into the sequence ["Teaching", "Learning", "Area", "Building"]. During the decoding process of the Trie constraint, the system applies a Trie-based mask filter to all possible subsequent tokens corresponding to the current prefix at each generation step, allowing only valid tokens that actually exist in the knowledge base entity IDs to be extended. This ensures that the final generated complete token sequence necessarily corresponds to a valid geographic entity, achieving a reliable transition from probabilistic generation to deterministic retrieval.

[0032] Beam search: Beam search is a heuristic search algorithm for sequence generation tasks. At each decoding step, it retains the top k candidate sequences with the highest probability (called beamwidth) and only expands these candidates in the next step, thus achieving a balance between computational efficiency and generation quality. In this invention, beam search is applied to the generative retrieval process of geographic entity IDs: under the structured constraints of a Trie, the system considers only subsequent tokens corresponding to valid prefixes at each generation step, searching for the globally optimal or near-optimal entity ID sequence in a constrained space through beam search, and finally outputting the unique entity identifier with the highest probability. This method retains the semantic understanding capability of generative models while ensuring the legality and retrieval accuracy of the output results through constraints and search strategies, avoiding invalid or ambiguous outputs that may occur with traditional generative models.

[0033] Sequence / Fallback: These are two fundamental and core control nodes in a behavior tree, used to organize the execution logic and flow control of child nodes. A Sequence node executes its child nodes sequentially in a predetermined order, returning overall success only when all child nodes return success; if any child node fails, subsequent execution is immediately interrupted and failure is returned. In this invention, Sequence is used to describe a sequence of tasks that must be executed strictly step-by-step, such as "first navigate to point A, then perform a search." A Fallback node also executes its child nodes sequentially, but stops execution and returns overall success only when any child node returns success; failure is returned only when all child nodes fail. This node is often used to implement priority attempts or fault recovery strategies, such as "first try path A, if it fails, try the backup path B."

[0034] LIO: This is an algorithm framework for real-time estimation of a robot's six-degree-of-freedom pose by tightly coupling and fusing 3D LiDAR point cloud and inertial measurement unit (IMU) data. It utilizes high-frequency angular velocity and acceleration data from the IMU for motion prediction and point cloud distortion correction, while simultaneously using environmental geometric features extracted by the LiDAR for scan matching and nonlinear optimization, achieving high-precision, low-drift continuous motion estimation. In this invention, LIO (LiDAR I / O) serves as the core technology for multi-sensor fusion localization, providing real-time position, attitude, and velocity state estimation for the NæVIS system. It supports the transformation from global waypoints to local coordinates, real-time path tracking, and the construction and updating of semantic voxel maps, making it a key underlying perception module ensuring long-distance navigation stability and last-mile exploration accuracy.

[0035] Semantic voxel graph: This invention is a 3D environment representation method that integrates geometric, semantic, and temporal information. It discretizes space into a regular cubic grid (voxels). Each voxel not only records geometric attributes such as occupancy probability and color, but also associates natural language semantic tags (such as "person," "red clothes," and "door") and confidence levels through an open-vocabulary visual model. In this invention, the semantic voxel map is constructed and updated online in real time by the NæVIS system through the fusion of multi-sensor data such as RGB-D and LiDAR. It serves as the basis for dynamic obstacle avoidance and semantic understanding during navigation, and supports target retrieval based on natural language queries in the last-mile exploration stage. By searching for voxels that match the semantic elements of the target in the query and instructions, the 3D position of the target is determined by density clustering, and finally projected onto the ground to form a navigable 2D target point, thereby achieving a precise closed loop from open-vocabulary description to spatial positioning.

[0036] DBSCAN clustering: DBSCAN is a density-based spatial clustering algorithm that does not require pre-specifying the number of clusters and can autonomously discover clusters of arbitrary shapes and effectively identify noise points. Based on two core parameters—neighborhood radius ε and minimum number of points MinPts—it divides samples into core points, boundary points, and noise points, and merges adjacent high-density regions into the same cluster through density reachability relationships. In this invention, DBSCAN is used to perform spatial clustering of highly relevant voxels retrieved from a semantic voxel map, aggregating neighboring voxels belonging to the same entity into the same cluster, eliminating interference from discrete noise voxels, and selecting the main cluster with the most voxels to calculate its 3D centroid as the target spatial location. Finally, it is projected onto 2D ground coordinates, thereby achieving stable and robust target localization in complex outdoor environments, avoiding the failure risk of clustering methods based on fixed thresholds or preset shapes when the target shape is variable and the background is cluttered.

[0037] Example 1 like Figure 1 As shown, this embodiment provides a retrieval-enhanced navigation and search method oriented towards outdoor geographic semantics. The method includes the following steps: S1: Receives natural language instructions and extracts navigation target semantic elements and search target semantic elements from the natural language instructions; under the constraints of a pre-built structured index, it performs beam search to align the navigation target semantic elements to unique entities in the local knowledge base, and outputs the latitude, longitude and boundary information of the navigation target; the local knowledge base is a local geographic entity database. Specifically, The process of building a structured index specifically includes: For each map entity in the local knowledge base, an entity ID is created by concatenating low-level and high-level identifiers, and a structured index is built based on all entity IDs.

[0038] The process of beam search under the constraints of structured indexes specifically includes: Chain-thinking prompts are used to extract structured clues from the semantic elements of the navigation target, including target categories and modifiers, to narrow the search scope as retrieval priors; Based on retrieval priors, multiple semantic units are generated sequentially, and the generation of each semantic unit is constrained by a pre-built structured index: selection is only allowed from the set of legal child nodes in the structured index, and the probability of semantic units belonging to other child node sets is set to zero; the generated semantic units are concatenated to obtain a unique target entity ID, and a search is performed in the local knowledge base to obtain the entity's latitude, longitude and boundary information.

[0039] S101: Receives natural language instruction I, such as "Go to the library and find the person in red at the entrance." The system parses the instruction into two semantic elements: navigation target and search target.

[0040] S102: GeoQA aligns navigation targets (such as "library") to a unique entity in the local OSM knowledge base, outputs the latitude, longitude, and boundary information of that entity, and uses it as input for global planning.

[0041] S1021: Task Formalization: Specifically, During the generation of global topology paths and waypoint sequences, the status of navigation nodes is periodically checked by a structured behavior tree. If a blockage, planning failure, or prolonged inactivity occurs, a recovery strategy is triggered to reinitialize the path search and update the path planning until the navigation task is completed.

[0042] Viewing natural language localization as a retrieval task: given a query q, select the best-matching entity e from the entity set E of the OSM derived knowledge base K, and output its coordinates for downstream planning. The challenges of this task include semantic diversity, limited training data, and the need to output unique entities, which makes freely generated entities prone to drift.

[0043] S1022: Entity ID Construction (hierarchical, distinguishable, and close to human expression): Construct a natural language ID for each map entity: • Lower-level signage: Original name of the entity (e.g., Building A) • High-level labeling: Coarse-grained categories (e.g., teaching area) • Combination: Category - Entity Name (e.g., "Teaching Area - Building A") This hierarchical naming is used to reduce ambiguity due to similar names and improve the stability of large model generation.

[0044] S1023: Trie constraint index: A Trie prefix tree is constructed for all legitimate entity IDs. During inference and decoding, the candidate set of tokens at each step is masked: expansion is only allowed to child nodes of the current prefix in the Trie, thus ensuring that the generated sequence corresponds to a real entity. The paper provides a formal definition: if a token does not belong to the child node set, its probability is set to zero.

[0045] S1024: Two-stage reasoning and retrieval: free reasoning + constrained implementation: Phase A (Free Reasoning): Lightweight chained thinking prompts are used on input q to extract structured clues, such as target category \hat{c} and modifiers (e.g., "teaching area - Building A"), as retrieval priors to narrow down the candidate range.

[0046] Phase B (Constrained Generation): Beam search is performed under Trie constraints to generate target entity IDs; since each step of expansion is constrained by the Trie, the final output is guaranteed to be a unique entity existing in K, thus avoiding "illusion locations / invalid entities".

[0047] S2: Decompose natural language instructions into sub-task sequences, and generate a structured behavior tree based on the sub-task sequences, latitude and longitude and boundary information, under predefined executable actions and condition space constraints; the navigation nodes in the structured behavior tree call the routing engine to generate a global topology path and waypoint sequence based on the nearest reachable boundary point of the entity; S201: The RAPPER, based on the locally deployed large model, integrates the task parsing results and GeoQA output to generate a structured and executable behavior tree (XML), and calls the routing engine to generate global topology paths and waypoint sequences, which are then sent to the execution layer.

[0048] S2011: Task breakdown and goal achievement: The RAPPER receives instruction I and breaks it down into a sequence of subtasks T, where T{navigation target} represents "where to go" and T_{exploration target} represents "what to find". For each T_{navigation target}, GeoQA is called to obtain its latitude and longitude coordinates.

[0049] S2012: Polygon target point refinement (avoiding centroid unreachable conditions): To ensure the navigation target is reachable, instead of directly taking the centroid of the polygon, the robot's position is projected onto the nearest point on the target polygon's outline to obtain the nearest target point.

[0050] Then, based on the OSM map node, the local routing engine is called to generate a global topology path.

[0051] S2013: Behavior Tree (BT) Generation: Constrained Action Space + XML Executable Representation: The RAPPER performs a one-time generation on the locally deployed Qwen2.5-14B: Under the constraints of a predefined action / condition space (skill pool), the task is mapped to a hierarchical XML behavior tree (Sequence / Fallback, etc.), ensuring that each node can be parsed into a specific action message by the executor.

[0052] S2014: Execution monitoring and recovery / replanning: During BT operation, the engine periodically ticks nodes and monitors the robot's status. If a blockage, planning failure, or prolonged inactivity occurs, the BT structure triggers a recovery strategy (such as re-initializing path search) and updates the plan in the closed loop until the task is completed.

[0053] S3: Real-time acquisition of multimodal sensor data, and construction of semantic voxel map based on multimodal sensor data; real-time navigation is performed along waypoint sequence based on structured behavior tree scheduling; after reaching the vicinity of the target, targets related to the semantic elements of the target are searched in the semantic voxel map with polygon boundaries as constraints, and clustered and located to obtain the final target point and converge navigation.

[0054] S301: NæVIS integrates multimodal sensors such as LiDAR / IMU / RGB-D / GNSS to perform long-distance navigation; after reaching the vicinity of the target polygon, it switches to the "last mile" mode, and obtains the final target point and converges navigation by exploring the target boundary constrained by the polygon, building semantic voxel maps with open vocabulary and clustering localization.

[0055] Specifically, The real-time navigation process specifically includes: Multimodal sensor data is collected, and a tight-coupled optimization algorithm is used to generate real-time positioning results. Based on the global waypoint sequence and the real-time positioning results, coordinate transformation is performed to convert the waypoint coordinates in the global coordinate system to the local coordinate system. The waypoint sequence is adaptively sampled according to the path curvature to generate a control point sequence. Finally, motion control is performed based on the control point sequence until the target entity boundary is reached.

[0056] Preferred, The coordinate transformation process specifically includes: After the GPS signal stabilizes, travel a preset distance in a straight line and record the positions of the starting and ending points; calculate the global heading vector and heading offset angle based on the positions of the starting and ending points, using the following formula: in, The coordinates of the endpoint are... The coordinates of the starting position, Indicates displacement in the north direction. Indicates displacement in the eastward direction; A rotation matrix is ​​constructed using the heading offset angle, and the coordinates of the waypoint sequence in the global coordinate system are transformed to the local coordinate system. The transformation formula is as follows: ; in, These are the waypoint sequence coordinates in the local coordinate system. The coordinates of the waypoint sequence in the global coordinate system; This indicates that a vector in the global coordinate system is transformed to the local coordinate system.

[0057] Specifically, Adaptive sampling specifically includes: If the curvature is greater than a preset threshold, sampling is performed once every shorter path; if the curvature is less than a preset threshold, sampling is performed once every longer path, in order to improve corner tracking and reduce the computational load on straight sections.

[0058] S3011: Building multi-sensor fusion and semantic voxel maps: The execution layer integrates RGB-D camera, LiDAR, and IMU (and combines GNSS), uses LIO to generate real-time state estimates, and updates the online semantic map with an open set semantic voxel mapping module to provide spatial memory for semantic search.

[0059] S3012: Global-Local Coordinate Bridging: Heading Initialization and Coordinate Transformation Once the GPS stabilizes, the robot performs a straight-line initialization with a length of d=5m and calculates the global heading vector using the GNSS at the start and end points (obtained via ENU conversion); And calculate the heading offset: Then use a rotation matrix to change the global waypoints Switching to local coordinates, the transformation formula is: This is to reduce drift and deviation caused by inconsistency between global waypoints and local odometers.

[0060] S3013: Perform curvature-adaptive waypoint sampling (balancing efficiency and turning accuracy): Adaptive sampling of the global path based on local curvature i: if the curvature is greater than the threshold, use a fine interval of 3m; otherwise, use a coarse interval of 20m to reduce the computational load on straight road segments and improve the tracingability of corners.

[0061] S3014: Conducting "last mile" exploration: Polygon constraint boundary exploration + semantic voxel clustering localization: Upon reaching the vicinity of the target, switch to exploration mode: 1. Initialization: Quickly fill the local semantic voxel map and generate initial boundaries using a 360° in-situ scan; 2. If no target is found: Enter boundary exploration. First, extract candidate boundary points from the occupied grid, then perform hard constraint filtering on the target polygon, retaining only the boundaries within the boundaries to prevent the exploration from going astray; 3. Simultaneously perform semantic segmentation based on natural language queries: Detect highly relevant voxels exceeding the similarity threshold in the voxel map; once triggered, interrupt exploration and switch to target navigation: Cluster highly relevant voxels using DBSCAN, take the centroid of the main cluster and project it onto the ground to obtain the final 2D target point, and the navigation converges to complete the task.

[0062] Specifically, The process of constructing polygon boundaries specifically includes: An initial exploration boundary is generated based on the semantic voxel map. If no target matching the semantic element of the search target is found in the initial exploration boundary, candidate boundary points are extracted from the occupied grid of the semantic voxel map, and polygon boundaries are extracted from the boundary information of the entity. Only candidate boundary points located within the polygon boundaries are retained to constrain the search range.

[0063] Preferred, The clustering and localization process specifically includes: In the semantic voxel map, highly relevant voxels exceeding a pre-set similarity threshold are detected. When a highly relevant voxel is detected, the exploration is interrupted, and a density-based spatial clustering algorithm is used to aggregate adjacent highly relevant voxels. The cluster containing the most voxels is selected as the master cluster, the centroid coordinates of the master cluster are calculated and projected onto the ground to obtain the final two-dimensional target point and converge navigation.

[0064] Example 2 This embodiment provides a system for an outdoor geographic semantic retrieval-enhanced navigation and search method, including a memory and a processor. The memory stores a computer program, and the processor calls the computer program to execute the steps of any of the methods in Embodiment 1.

[0065] The innovation and beneficial effects of this invention are as follows: Innovations include: First, a generative geographic retrieval mechanism based on prefix tree constraints is proposed. During the retrieval phase, structured constraints are applied to the generation process, ensuring that the output always falls within the map entity candidate set. This mechanism avoids generating non-existent locations or invalid identifiers, guaranteeing that the retrieval results are verifiable and directly applicable to subsequent path planning and task execution. Second, a local offline task planning and execution monitoring method is proposed. High-level planning is restricted to a predefined action and condition space, and a structured, executable task flow is generated in the form of a behavior tree. This makes the planning process interpretable and feasible. Simultaneously, during the execution phase, status monitoring and recovery logic enable automatic retrieval upon encountering obstacles and dynamic replanning when necessary, thereby reducing reliance on external networks and remote inference and mitigating latency risks. Third, a region constraint exploration and 3D semantic clustering localization method for the last mile is proposed. After reaching the target area, frontier exploration using region boundary constraints prevents search divergence, and stable target points are obtained through 3D voxel semantic fusion and clustering localization, avoiding information collapse and target loss caused by 2D projection, and improving the robustness and stability of target search in complex outdoor environments.

[0066] Beneficial effects: In simulation evaluation, this solution shows significant advantages over the baseline method in long-distance navigation and last-mile tasks, with better performance in metrics such as task success rate and path efficiency. In terms of geographic retrieval capabilities, it achieves higher retrieval accuracy and stability through constrained generative retrieval and possesses strict offline deployability, making it suitable for weak or no-network scenarios. In terms of real-world system verification, this solution has completed a real-world mission trajectory of approximately 500 meters on an unmanned ground vehicle platform, proving that the method is not only effective in terms of evaluation metrics but also has engineering feasibility and scalability.

[0067] Improvements: The "natural language to geographic entity" translation process has been transformed from unreliable, freely generated data prone to formatting errors to constrained and verifiable output, ensuring that downstream path planning always yields legitimate targets. Compared to cloud-based iterative large language model calls, RAPPER offers local offline inference, structured output, and is monitorable and recoverable, reducing latency and uncontrollable risks. Compared to common 2D bird's-eye view projection semantic solutions, this approach maintains spatial consistency through 3D voxel fusion, reducing the risk of distant targets being "flattened / lost" in 2D projection, and makes the search more focused through boundary constraints.

[0068] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.

Claims

1. A retrieval-enhanced navigation and search method oriented towards outdoor geographic semantics, characterized in that, include: Receive natural language instructions and extract navigation target semantic elements and search target semantic elements from the natural language instructions; Under the constraints of a pre-built structured index, beam search is performed to align the semantic elements of the navigation target to unique entities in the local knowledge base, and the latitude, longitude, and boundary information of the navigation target are output; the local knowledge base is a local geographic entity database. The natural language instructions are decomposed into a sequence of subtasks, and a structured behavior tree is generated based on the subtask sequences and the latitude, longitude and boundary information, under predefined executable actions and conditional space constraints. The navigation node in the structured behavior tree calls the routing engine to generate a global topology path and waypoint sequence based on the nearest reachable boundary point of the entity; Real-time acquisition of multimodal sensor data, and construction of a semantic voxel map based on the multimodal sensor data; Based on the scheduling of the structured behavior tree, real-time navigation is performed along the waypoint sequence; after reaching the vicinity of the target, with polygon boundaries as constraints, targets related to the semantic elements of the search target are searched in the semantic voxel map and clustered for localization to obtain the final target point and converge navigation.

2. The retrieval enhancement navigation and search method for outdoor geographic semantics as described in claim 1, characterized in that, The construction process of the structured index specifically includes: For each map entity in the local knowledge base, an entity ID is created by concatenating a low-level identifier and a high-level identifier, and a structured index is built based on all entity IDs.

3. The retrieval enhancement navigation and search method for outdoor geographic semantics as described in claim 2, characterized in that, The beam search process under the constraints of the structured index specifically includes: Chain-thinking prompts are used to extract structured clues from the semantic elements of the navigation target, including target categories and modifiers, as retrieval priors to narrow the search scope; Based on the retrieval prior, multiple semantic units are generated sequentially, and the generation of each semantic unit is constrained by a pre-built structured index: selection is only allowed from the set of legal child nodes in the structured index, and the probability of semantic units belonging to other child node sets is set to zero; the set of legal child nodes consists of semantic units that actually exist in the local knowledge base; the generated semantic units are concatenated to obtain the unique target entity ID, and a search is performed in the local knowledge base to obtain the latitude, longitude, and boundary information of the entity.

4. The retrieval enhancement navigation and search method for outdoor geographic semantics as described in claim 1, characterized in that, During the generation of the global topology path and waypoint sequence, the structured behavior tree periodically detects the status of the navigation nodes. If a blockage, planning failure, or prolonged inactivity occurs, a recovery strategy is triggered to reinitialize the path search and update the path planning until the navigation task is completed.

5. The retrieval enhancement navigation and search method for outdoor geographic semantics as described in claim 1, characterized in that, The multimodal sensor data specifically includes lidar point cloud data, inertial measurement unit data, RGB images, and depth image data; The real-time navigation process specifically includes: generating real-time positioning results based on the multimodal sensor data using a tightly coupled optimization algorithm; performing coordinate transformation based on the global waypoint sequence and the real-time positioning results to convert the waypoint coordinates in the global coordinate system to the local coordinate system, and adaptively sampling the waypoint sequence according to the path curvature to generate a control point sequence; finally, performing motion control based on the control point sequence until reaching the vicinity of the target entity boundary.

6. The retrieval enhancement navigation and search method for outdoor geographic semantics according to claim 5, characterized in that, The coordinate transformation process specifically includes: After the GPS signal stabilizes, travel a preset distance along a straight line and record the positions of the starting and ending points; calculate the global heading vector and heading offset angle based on the positions of the starting and ending points, using the following formula: in, The coordinates of the endpoint are... The coordinates of the starting position, Indicates displacement in the north direction. Indicates displacement in the eastward direction; A rotation matrix is ​​constructed using the aforementioned heading offset angle, and the coordinates of the waypoint sequence in the global coordinate system are transformed to the local coordinate system. The transformation formula is as follows: ； in, These are the waypoint sequence coordinates in the local coordinate system. The coordinates of the waypoint sequence in the global coordinate system; This indicates that a vector in the global coordinate system is transformed to the local coordinate system.

7. The retrieval enhancement navigation and search method for outdoor geographic semantics according to claim 5, characterized in that, The adaptive sampling specifically includes: If the curvature is greater than a preset threshold, sampling is performed once every shorter path; if the curvature is less than a preset threshold, sampling is performed once every longer path, in order to improve corner tracking and reduce the computational load on straight sections.

8. The retrieval enhancement navigation and search method for outdoor geographic semantics according to claim 1, characterized in that, The process of constructing the polygon boundary specifically includes: An initial exploration boundary is generated based on the semantic voxel map. If no target matching the semantic element of the search target is found in the initial exploration boundary, candidate boundary points are extracted from the occupied grid of the semantic voxel map, and polygon boundaries are extracted from the boundary information of the entity. Only candidate boundary points located within the polygon boundaries are retained to constrain the search range.

9. The retrieval enhancement navigation and search method for outdoor geographic semantics according to claim 1, characterized in that, The clustering and localization process specifically includes: In the semantic voxel map, highly relevant voxels exceeding a pre-set similarity threshold are detected. When a highly relevant voxel is detected, the exploration is interrupted, and a density-based spatial clustering algorithm is used to aggregate adjacent highly relevant voxels. The cluster containing the most voxels is selected as the master cluster, the centroid coordinates of the master cluster are calculated and projected onto the ground to obtain the final two-dimensional target point and converge navigation.

10. A system for enhanced navigation and search methods oriented towards outdoor geographic semantics, characterized in that, It includes a memory and a processor, the memory storing a computer program, the processor invoking the computer program to perform the steps of the method as described in any one of claims 1 to 9.

Images