Visual spatial reasoning enhanced zero-shot unmanned aerial vehicle visual language navigation method
By employing a zero-shot method enhanced by visual spatial reasoning, and utilizing landmark waypoint generation and multi-stage verification reasoning, the problems of inaccurate target detection and spatial relationship reasoning bias in UAV visual language navigation are solved, achieving efficient and accurate navigation task execution.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- TIANMUSHAN LABORATORY
- Filing Date
- 2026-06-04
- Publication Date
- 2026-06-30
AI Technical Summary
Existing UAV visual language navigation methods are greatly affected by changes in perspective and scale when dealing with complex environments. They are inaccurate in target detection, have biases and illusions in spatial relationship reasoning, and the semantic ambiguity of natural language commands leads to inconsistent navigation.
We employ a zero-shot method enhanced by visual spatial reasoning, which improves navigation accuracy and adaptability by combining visual cues and spatial reasoning techniques through landmark waypoint generation, visual cue generation in the perception stage, multi-stage verification reasoning, and a closed-loop feedback mechanism in the execution stage.
The system enables efficient execution of navigation tasks in unknown environments, significantly improving navigation success rate and accuracy, reducing deployment costs, adapting to different environments and tasks, and reducing erroneous decisions and positioning deviations in traditional methods.
Smart Images
Figure CN122306094A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of unmanned aerial vehicle (UAV) technology, and more specifically to a zero-sample UAV visual language navigation method enhanced by visual spatial reasoning. Background Technology
[0002] In recent years, Visual Language Navigation (VLN) for unmanned aerial vehicles (UAVs), as an innovative UAV navigation technology, has gradually demonstrated enormous application potential in fields such as autonomous UAV navigation, intelligent transportation, and environmental monitoring. Unlike traditional navigation methods, VLN utilizes natural language commands to guide UAVs in autonomous navigation within complex three-dimensional environments. Existing VLN methods typically rely on accurate maps and real-time positioning information to assist navigation; however, in practical applications, due to the complexity and variability of the environment, the execution of navigation tasks often faces significant challenges in the absence of accurate maps and real-time positioning information.
[0003] Traditional UAV visual-language navigation methods often employ a pipeline structure of detection and planning, converting natural language commands into discrete text scene graphs, and then completing navigation through target recognition and spatial reasoning. However, this approach suffers from several significant problems. First, image-based target detection models are greatly affected by changes in viewpoint and scale when processing aerial images, often resulting in inaccurate feature matching and degraded detection performance. Second, existing spatial reasoning methods often rely on discrete textual symbol representations, such as scene graphs, which struggle to accurately reflect continuous spatial layouts, easily leading to biases or illusions in spatial relationship reasoning. Furthermore, the natural language commands in existing methods inherently possess semantic ambiguity; for example, descriptions of spatial relationships such as "in front" or "on the left" often fail to accurately resolve these ambiguities based on visual context when handled in visual-language models, resulting in inconsistent reasoning results.
[0004] Therefore, it is necessary to propose a zero-shot UAV visual language navigation method enhanced by visual spatial reasoning to solve the above problems. Summary of the Invention
[0005] The purpose of this invention is to solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention specifically adopts the following technical solution:
[0007] A zero-shot UAV visual-language navigation method enhanced with visual-spatial reasoning includes the following steps:
[0008] a) Problem formulation and overall framework construction, used to define the UAV visual language navigation task and establish a navigation framework;
[0009] b) Landmark-based waypoint generation: Based on the outline geometry of the landmarks, the exploration waypoints of the UAV are pre-calculated;
[0010] c) The perception phase is executed by processing the raw images acquired by the UAV through a visual cue generator to generate a structured visual representation for spatial reasoning.
[0011] d) Execution of the verification phase: Based on the structured visual representation, multi-stage verification reasoning is performed through the verification module to increase the dependence of spatial reasoning on visual evidence, reduce the risk of reasoning inconsistent with visual evidence, and suppress the illusion of pure text reasoning detached from visual evidence.
[0012] e) Execution phase: The high-order semantic decisions of the verification phase are converted into specific flight actions of the UAV by the actuator;
[0013] In this process, steps b) to d) are executed sequentially in a tightly coupled manner at each time step after the UAV acquires the original image, and the output of the verification stage is fed back to the perception stage through a closed-loop feedback mechanism to guide the next round of detection.
[0014] Furthermore, the formulation of the problem and the overall framework construction include:
[0015] Define landmark prior information, which includes landmark name, centroid coordinates and contour point set;
[0016] Define a navigation scenario, which includes natural language navigation instructions, the initial pose of the UAV, and a set of all available landmark prior information.
[0017] Define a navigation target, which is to reach the target position within a finite number of steps, and use the Euclidean distance between the UAV's termination position and the target position satisfying a preset threshold as the criterion for successful navigation.
[0018] Furthermore, the landmark-based waypoint generation includes:
[0019] Contour fusion: Calculate the fused contour and determine the intersection area of the fused contour as the priority search area;
[0020] Mesh generation: Candidate observation points are generated within the bounding box of the fused contour, and the step size of the candidate observation points is calculated based on the field of view angle at the current altitude of the UAV.
[0021] Greedy selection: A greedy set coverage algorithm is used to select a subset of observation points to maximize the cumulative coverage of the fused contour while satisfying the field of view overlap constraint. The access order of the subset of observation points is optimized by the traveling salesman problem.
[0022] Furthermore, the perception phase includes the following:
[0023] Semantic region extraction: The visual cue generator uses the open vocabulary detection capability of the visual language model to extract semantic regions of interest from the original image;
[0024] The structured visual representation generation involves, after identifying candidate targets, the visual cue generator dividing the image into regions of different granularities and labeling them using an overlay tag set to generate the structured visual representation, which includes the labeled image and symbol mapping.
[0025] Candidate target passing: All detected potential candidate targets are passed to the verification phase.
[0026] Furthermore, the verification phase executes explicit three-stage verification reasoning through a visual language model, the three-stage reasoning including:
[0027] The verification module performs the following verification methods: literal attribute matching, which verifies whether the visible features of the candidate target are consistent with the description in the natural language instruction; spatial topology verification, which verifies the spatial relationship between the target mentioned in the navigation instruction and the surrounding environment to eliminate referential ambiguity; and geographic boundary verification, which checks the spatial relationship between the candidate target and known landmarks to ensure its geographic legitimacy.
[0028] Furthermore, in the literal attribute matching stage, when visual evidence is insufficient, the target is marked as "to be determined".
[0029] Furthermore, the closed-loop feedback mechanism is as follows: when the verification stage cannot obtain sufficient evidence, the verification module outputs a natural language guidance signal, which is directly fed back to the perception stage as an additional text prompt for the next round of detection to enhance perception without changing the original task semantics.
[0030] Furthermore, the execution phase includes:
[0031] Define an execution mapping that converts the current state of the UAV, high-order semantic task primitives, and landmark prior information into the UAV target pose.
[0032] Task primitive parsing and execution maps the semantic decisions of the verification module to UAV actions through dedicated task primitives.
[0033] Furthermore, the task primitive parsing and execution includes at least one of the following actions:
[0034] Once the verification module confirms the target, the actuator uses a back-projection method to back-project the target's two-dimensional pixel centroids into world coordinates and controls the UAV to navigate directly to that location.
[0035] The movement action involves the actuator controlling the drone to fly to the next path point in the pre-calculated path point set. Upon arrival, the perception phase is triggered to re-observe the scene and autonomously decide on the next action.
[0036] During ascent / descent, the actuator adjusts the drone's altitude and correspondingly adjusts the field of view to balance image detail resolution and scene coverage.
[0037] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0038] 1. The visual language model of this invention combines visual cues and spatial reasoning techniques, enabling efficient navigation tasks in unknown environments. Through a multi-stage feedback mechanism, it effectively eliminates the reasoning illusion of the visual language model, significantly improving success rate and navigation accuracy. For example, the three-stage verification reasoning module reduces erroneous decisions and positioning deviations found in traditional methods.
[0039] 2. This invention requires no additional data training: Unlike traditional methods that require extensive supervised training, the zero-shot method proposed in this invention utilizes visual and inference cues to enable effective inference directly without the need for large amounts of training data. This reduces deployment costs and allows for better adaptation to different environments and tasks.
[0040] 3. This invention offers enhanced spatial reasoning capabilities: Through a visual cue generator and a rigorous three-stage spatial reasoning verification module, the system can more accurately understand and identify complex spatial relationships from the drone's top-down perspective, avoiding reasoning errors caused by discrete spatial reasoning based on text-based scene graphs in traditional methods. Especially when dealing with complex urban scenes and spatial geographic relationships, the verification stage performs multi-level verification of visual evidence, ensuring high accuracy in spatial reasoning. Attached Figure Description
[0041] Figure 1 This is the overall flowchart of the present invention. Detailed Implementation
[0042] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0043] Combination Figure 1 As shown, a zero-shot UAV visual language navigation method enhanced by visual spatial reasoning includes the following steps:
[0044] Step 1: Formulating the Problem and Building the Overall Framework
[0045] This step is used to clarify the mathematical definition and core logic of the UAV visual language navigation task, providing a theoretical basis for subsequent navigation operations. Specifically, it includes the following:
[0046] 1.1 Definition of prior information about landmarks: The set of landmarks that can be acquired by drones Chinese landmarks Prior information, defined as triples:
[0047]
[0048] in: Indicates the name of the landmark. Indicates the coordinates of the centroid of the landmark. Represents a set of landmark outline points; given natural language navigation instructions (For example, "the house with a white roof on the left side of Broadway"), when the drone passes by Get the corresponding landmark Location and shape information.
[0049] 1.2 Navigation Scenario Definition: Each navigation round is formally defined as a triple:
[0050]
[0051] in: Natural language navigation instructions, This is the initial pose of the drone. As a world coordinate system, For flight altitude, (Heading angle) The set of prior information for all available landmarks.
[0052] 1.3 Real-time observation and action definition: at each time step UAVs acquire aerial view images and current pose The CityNav environment natively provides a low-order discrete motion space including forward, left turn, right turn, up, down, and stop. This invention employs a hierarchical control strategy, whereby the system outputs high-order semantic task primitives. Stop, move, rise, fall; the actuator then transforms this primitive into a series of low-order... Instructions (see step five for details).
[0053] 1.4 Definition of Navigation Target: A navigation target is a target that can be navigated in a finite number of steps. Reach the target location within The criterion for successful navigation is that the Euclidean distance between the termination position and the target position satisfies the following condition. ,in Meters is the success threshold. The world coordinates of the ending position.
[0054] 1.5 Framework Closed-Loop Logic: Given Navigation Instructions and landmark a priori The framework calls the landmark-based waypoint generation module to pre-calculate the exploration path based on the fused landmark outlines; at each time step... The drone acquires the original top-down image. Subsequently, the three tightly coupled stages from step two to step four are executed sequentially. This three-stage architecture suppresses the reasoning illusion of the visual language model, increases the reliance of spatial reasoning on visual evidence, and reduces the risk of reasoning inconsistent with visual evidence. Each stage is connected through a closed-loop feedback mechanism. When sufficient evidence cannot be obtained in the verification stage, a natural language guidance signal is output. Feedback is sent to the perception stage to enhance the next round of detection.
[0055] Step 2: Landmark-based waypoint generation
[0056] This step is used to efficiently explore waypoints by pre-calculating landmark outline geometry information for navigation commands with complex spatial constraints (such as "drive around the block"). The specific steps are as follows:
[0057] 2.1 Contour Blending: Calculating the fused contour The calculation formula is as follows:
[0058] The intersection area of the contours is then identified as the priority search area.
[0059] 2.2 Mesh Generation: Candidate observation points are generated within the bounding box of the fused contour, with a step size of [missing information]. The calculation formula is as follows:
[0060] Among them, height The formula for calculating the field of view (FOV) at a given location is:
[0061] Parameter settings are as follows , .
[0062] 2.3 Greedy Selection: A greedy set-cover algorithm is used to select a subset of observation points. Under the premise of satisfying the field of view overlap constraint, maximize the fusion contour. The cumulative coverage; the order of observation point access is optimized through the Traveling Salesman Problem (TSP), and the access path length is approximately minimized at the discrete observation point level, ignoring dynamics and environmental constraints; the actuator generates the path during the initialization phase and executes the "move" command sequentially along the pre-calculated path points.
[0063] Step 3: Execution of the Perception Phase
[0064] This step processes the original top-down image using a visual cue generator to provide a structured visual representation for subsequent spatial reasoning. The specific steps are as follows:
[0065] 3.1 Semantic Region Extraction: The visual cue generator utilizes the open-vocabulary target localization or recognition capabilities of the visual language model to extract semantic regions in images. Compared to the limitations of traditional open-vocabulary detection models, such as training distribution constraints, weak small target detection capabilities from the perspective of UAVs, support for isolated word queries, and lack of overall environmental contextual relevance, open-vocabulary detection based on the visual language model has stronger adaptability. It can locate multiple different targets simultaneously in a single processing step by combining contextual awareness through complex and free-form natural language cues.
[0066] 3.2 Structured Visual Representation Generation: After identifying candidate targets, the visual cue generator divides the image into regions of different granularities and labels them using an overlay tag set to generate structured visual representations. ,in For the labeled image, The symbol mapping is as follows:
[0067] In the formula, each digital identifier uniquely corresponds to a physical entity.
[0068] 3.3 Candidate target delivery: The structured visual representation is delivered to the verification stage to reduce the contextual interference between candidate generation in the perception stage and discrimination in the verification stage, thereby decoupling the detection recall rate from the verification accuracy.
[0069] Step 4: Verification Phase
[0070] This step, based on the labeled images and structured visual representations provided in the perception phase, executes explicit three-stage verification reasoning through the verification module. This increases the reliance of spatial reasoning on visual evidence and reduces the risk of inference inconsistent with visual evidence. The core implementation of the verification module relies on the thought chain capability of the visual language model. This model can process the relationship between image data and natural language through thought chains and can perform reasoning given natural language instructions and image data, generating relevant high-level decisions. The specific steps are as follows:
[0071] 4.1 Literal Attribute Matching: The verification module verifies whether the visible features of the candidate target are consistent with the natural language instruction by referencing numerical identifiers (e.g., ①); it strictly follows the literal observation principle, only evaluates directly visible attributes, and marks the target as "to be determined" when visual evidence is insufficient, thus postponing the decision.
[0072] 4.2 Spatial Topology Verification: The verification module verifies the spatial relationship between the target and its surrounding environment by referencing numerical identifiers (e.g., "① is opposite ②"), replacing the traditional discrete text scene diagram and eliminating ambiguity in reference.
[0073] 4.3 Geographic Boundary Verification: The verification module ensures the geographic legitimacy of candidate targets by checking their spatial relationship with known landmarks, thus avoiding the selection of visually matching targets that are geographically incorrect.
[0074] 4.4 Iterative Visual Cue Generator Prompt: When the current scene cannot provide sufficient spatial evidence, the verification module outputs a natural language guidance signal. (For example, “pay close attention to white vehicles near the intersection”), this signal is fed back directly to the perception stage as an additional textual cue for the next round of detection to enhance perception without changing the original task semantics.
[0075] Step 5: Execution Phase
[0076] This step serves as the interface between abstract semantic reasoning and physical control. Through a semantic-motion decoupling actuator, high-order semantic decisions are converted into specific flight actions of the UAV. The specific steps are as follows:
[0077] 5.1 Execution Mapping Definition: Define the execution mapping relationship to convert the current position, higher-order semantic task primitives, and landmark prior information into the UAV target pose. The mapping formula is as follows:
[0078] In the formula, The target pose of the drone. For high-order semantic tasks, It is a set of prior information about landmarks.
[0079] 5.2 Task Primitive Parsing and Execution: The semantic decisions of the verification module are mapped to UAV actions using three types of dedicated task primitives, as follows:
[0080] 5.2.1 Stop Action: After the verification module confirms the target, the actuator converts the visual semantic results into physical navigation targets; using the back projection formula, and leveraging the camera intrinsic parameter matrix and the known flight altitude of the UAV relative to the ground, the two-dimensional pixel centroids of the target are directly back-projected into world coordinates. It also controls the drone to navigate directly to that location, effectively avoiding the accumulation of errors over multiple steps.
[0081] 5.2.2 Movement Action: The actuator controls the UAV to fly to the pre-calculated set of waypoints. Upon reaching the next path point, the perception phase is triggered, allowing the scene to be re-observed and the next action to be autonomously decided.
[0082] 5.2.3 Ascent / Descent Action: The actuator adjusts the drone's altitude. The field of view is adjusted accordingly to balance the image detail resolution and scene coverage.
[0083] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. The scope of patent protection of the present invention shall be determined by the claims. Similarly, any equivalent structural changes made based on the content of the present invention's specification shall also be included within the scope of protection of the present invention.
Claims
1. A zero-shot visual-language navigation method for unmanned aerial vehicles (UAVs) enhanced with visual-spatial reasoning, characterized in that, Includes the following steps: a) Problem formulation and overall framework construction, used to define the UAV visual language navigation task and establish a navigation framework; b) Landmark-based waypoint generation: Based on the outline geometry of the landmarks, the exploration waypoints of the UAV are pre-calculated; c) The perception phase is executed by processing the raw images acquired by the UAV through a visual cue generator to generate a structured visual representation for spatial reasoning. d) Execution of the verification phase: Based on the structured visual representation, multi-stage verification reasoning is performed through the verification module; e) Execution phase: The high-order semantic decisions of the verification phase are converted into specific flight actions of the UAV by the actuator; In this process, steps b) to d) are executed sequentially in a tightly coupled manner at each time step after the UAV acquires the original image, and the output of the verification stage is fed back to the perception stage through a closed-loop feedback mechanism to guide the next round of detection.
2. The zero-shot UAV visual-language navigation method enhanced by visual-spatial reasoning according to claim 1, characterized in that, The formalization of the problem and the overall framework construction include: Define landmark prior information, which includes landmark name, centroid coordinates and contour point set; Define a navigation scenario, which includes natural language navigation instructions, the initial pose of the UAV, and a set of all available landmark prior information. Define a navigation target, which is to reach the target position within a finite number of steps, and use the Euclidean distance between the UAV's termination position and the target position satisfying a preset threshold as the criterion for successful navigation.
3. The zero-shot UAV visual-language navigation method enhanced by visual-spatial reasoning according to claim 1, characterized in that, The landmark-based waypoint generation includes: Contour fusion: Calculate the fused contour and determine the intersection area of the fused contour as the priority search area; Mesh generation: Candidate observation points are generated within the bounding box of the fused contour, and the step size of the candidate observation points is calculated based on the field of view angle at the current altitude of the UAV. Greedy selection: A greedy set coverage algorithm is used to select a subset of observation points to maximize the cumulative coverage of the fused contour while satisfying the field of view overlap constraint. The access order of the subset of observation points is optimized by the traveling salesman problem.
4. The zero-shot UAV visual-language navigation method enhanced by visual-spatial reasoning according to claim 1, characterized in that, The perception phase includes the following: Semantic region extraction: The visual cue generator uses the open vocabulary detection capability of the visual language model to extract semantic regions of interest from the original image; The structured visual representation generation involves, after identifying candidate targets, the visual cue generator dividing the image into regions of different granularities and labeling them using an overlay tag set to generate the structured visual representation, which includes the labeled image and symbol mapping. Candidate target passing: All detected potential candidate targets are passed to the verification phase.
5. The zero-shot UAV visual-language navigation method enhanced by visual-spatial reasoning according to claim 1, characterized in that, The verification phase executes explicit three-stage verification reasoning through a visual language model, the three-stage reasoning including: The verification module performs the following verification methods: literal attribute matching, which verifies whether the visible features of the candidate target are consistent with the description in the natural language instruction; spatial topology verification, which verifies the spatial relationship between the target mentioned in the navigation instruction and the surrounding environment to eliminate referential ambiguity; and geographic boundary verification, which checks the spatial relationship between the candidate target and known landmarks to ensure its geographic legitimacy.
6. The zero-shot UAV visual-language navigation method enhanced by visual-spatial reasoning according to claim 5, characterized in that, In the literal attribute matching stage, when visual evidence is insufficient, the target is marked as "to be determined".
7. The zero-shot UAV visual-language navigation method enhanced by visual-spatial reasoning according to claim 1, characterized in that, The closed-loop feedback mechanism is as follows: when the verification stage cannot obtain sufficient evidence, the verification module outputs a natural language guidance signal, which is directly fed back to the perception stage as an additional text prompt for the next round of detection.
8. The zero-shot UAV visual-language navigation method with enhanced visual-spatial reasoning according to claim 1, characterized in that, The execution phase includes: Define an execution mapping that converts the current state of the UAV, high-order semantic task primitives, and landmark prior information into the UAV target pose. Task primitive parsing and execution maps the semantic decisions of the verification module to UAV actions through dedicated task primitives.
9. The zero-shot UAV visual-language navigation method enhanced by visual-spatial reasoning according to claim 8, characterized in that, The task primitive parsing and execution includes at least one of the following actions: Once the verification module confirms the target, the actuator uses a back-projection method to back-project the target's two-dimensional pixel centroids into world coordinates and controls the UAV to navigate directly to that location. The movement action involves the actuator controlling the drone to fly to the next path point in the pre-calculated path point set. Upon arrival, the perception phase is triggered to re-observe the scene and autonomously decide on the next action. During ascent / descent, the actuator adjusts the drone's altitude and correspondingly adjusts the field of view to balance image detail resolution and scene coverage.