A low-altitude unmanned aerial vehicle zero-sample visual language navigation method for urban scenes

Abstract

A low-altitude unmanned aerial vehicle zero-sample visual language navigation method for urban scenes. Under the condition that the absolute position of the navigation target is unknown and GNSS signal is denied, the forward-looking image, depth image and overhead view of the unmanned aerial vehicle are obtained; based on the weighted recursive of the continuous historical track points before the signal loss, the relative direction of the target is calculated from the historical track and the heading correction information is generated; the forward-looking image is divided into multiple image regions, the semantic similarity of each region and the natural language task target is calculated by using a picture-text matching model, and the highest score region is selected to generate a physical position prompt; the multi-view image, task target, heading correction information and position prompt are jointly input into a multi-modal large model, the normalized decision instruction containing action type and amplitude parameter is constrained by the pre-defined action set, and the autonomous navigation closed loop of visual perception, language reasoning and flight control is formed after analysis and execution. The invention realizes zero-sample open vocabulary navigation without preset coordinates and special detectors under GNSS denial.

Description

Technical Field

[0001] This invention relates to autonomous navigation, intelligent flight, visual language understanding, and multimodal large model application technologies for low-altitude unmanned aerial vehicles (UAVs), and particularly to a zero-sample visual language navigation method for low-altitude UAVs in urban scenarios. Background Technology

[0002] With the rapid development of the low-altitude economy, the complexity of UAV missions and the intensity of environmental interaction are increasing, placing higher demands on the autonomy of UAVs. Traditional UAV navigation relies heavily on Global Navigation Satellite System (GNSS) signals, preset waypoints, manual remote control, dedicated target detectors, or fixed category recognition models, making it difficult to adapt to the actual operating conditions of changing target descriptions, frequent obstruction, drastic changes in perspective, and dense urban obstacles. Meanwhile, due to the inherent vulnerability of GNSS in urban scenarios (such as multipath effects, signal obstruction, and interference), UAVs urgently need the ability to achieve high-precision autonomous navigation under GNSS signal decay conditions. Furthermore, in the absence of high-precision prior maps or manually preset waypoints, UAVs also need to be able to understand targets described by open-ended terms in natural language, such as "hotel entrance," "restaurant across the river," "blue amusement park ride," and "park bench," and make continuous and robust online decisions.

[0003] Existing low-altitude navigation technologies for unmanned aerial vehicles (UAVs) mainly include manual remote control, GNSS / inertial navigation combined with navigation, pre-defined waypoint planning navigation, and visual odometry-based navigation. Manual remote control relies on the pilot continuously observing the image transmission and issuing control commands, making it difficult to support large-scale UAV swarms for multi-target search tasks. Waypoint planning methods require prior access to target coordinates or high-precision maps and lack understanding of natural language semantic information. Visual odometry methods depend on the stability of environmental texture and feature extraction, and are prone to tracking loss under conditions of weak texture, repetitive texture, or dynamic lighting. When GNSS signals weaken, GNSS / inertial navigation combined with navigation introduces significant positioning errors, failing to guarantee reliable navigation mission execution. Therefore, traditional solutions are ill-suited to the complex navigation needs of urban low-altitude scenarios, necessitating the introduction of new intelligent navigation mechanisms.

[0004] Most current solutions attempt to introduce a Vision Language Model (VLM) to replace GNSS for safe and autonomous navigation tasks. In this scenario, navigation targets are often not predefined fixed categories, but rather given ad hoc by natural language, such as "near the orange-striped parasol ahead," "near the restaurant entrance on the opposite bank of the river," or "near the art sculpture ahead." Traditional dedicated target detectors struggle to cover these open-vocabulary targets and are prone to false positives or false negatives when targets are far away, partially occluded, scale-changing, or viewpoint-switching. Meanwhile, traditional visual feature matching methods have limited model capacity and representational ability when faced with diverse targets described in natural language, significantly constraining their generalization capabilities. In summary, current technology lacks a feasible solution that can combine open-vocabulary semantic understanding with stable low-altitude obstacle avoidance navigation under conditions of lost satellite navigation signals.

[0005] Traditional UAV navigation technology relies on GNSS / inertial navigation combined technology and preset route strategies. In relatively open environments, it can achieve continuous, stable, and low-latency navigation control, but it is highly dependent on stable and continuous GNSS signals, such as... Figure 1 As shown. Figure 1 It demonstrates the existing traditional UAV navigation technology architecture, including the traditional UAV navigation process based on GNSS satellite positioning system and inertial measurement unit (IMU).

[0006] The basic process is as follows: an airborne GNSS receiver (supporting GPS, BeiDou, etc.) acquires the UAV's latitude, longitude, altitude, and speed information in real time, while an inertial measurement unit (IMU) simultaneously collects triaxial acceleration and angular velocity. The data from both sources are processed using a multi-source data fusion algorithm (such as Kalman filtering) for tight or loose coupling to obtain a high-frequency, smooth attitude estimate. Simultaneously, the flight control system loads a preset flight path as the desired trajectory, compares the fused navigation solution with the flight path, and generates lateral and longitudinal deviations. Finally, the navigation module outputs control commands to drive the actuators, forming a closed-loop adjustment. However, if the GNSS signal is blocked, experiences multipath effects, or is interfered with, the positioning accuracy and stability of the fusion system will significantly decrease, leading to flight path tracking failure.

[0007] The main drawbacks of existing technologies are as follows: First, insufficient adaptability to open-vocabulary targets. Traditional target detection and navigation systems heavily rely on predefined category labels and a large number of finely labeled samples, strictly limiting their perception range to the closed vocabulary of the training set. This results in a lack of basic semantic understanding and zero-shot generalization ability when faced with targets described temporarily by users in natural language that have not appeared in the training data (such as "the person wearing a red coat and carrying a blue package" or "a white car with a broken left rearview mirror"). Therefore, whenever a new target type is introduced, the system needs to retrain the model, severely restricting the flexible deployment and real-time interaction capabilities of UAVs in dynamic, open urban environments. Second, a lack of effective means to address GNSS vulnerability. The "urban canyon" effect formed by urban high-rise building clusters can cause severe obstruction, scattering, and multipath reflection of GNSS signals, seriously affecting navigation accuracy and directly threatening flight safety: at the moment of signal loss, if the UAV suddenly loses its heading reference and blindly executes the flight maneuvers of the previous moment, it is highly likely to collide with urban three-dimensional obstacles such as building exteriors, billboards, and lampposts. Therefore, the ability to maintain course and ensure safe navigation under GNSS decay conditions has become a key enabling technology for the large-scale deployment of low-altitude urban drones.

[0008] In summary, the fundamental shortcomings of existing technical solutions can be attributed to the following aspects: First, the high dependence on GNSS limits the application scenarios. In complex urban low-altitude environments, once the satellite signal is lost, the existing navigation methods immediately lose absolute reference, causing the flight direction to drift randomly and making it impossible to maintain the correct heading and calculate the relative position of the navigation target autonomously.

[0009] Second, there is a natural cross-modal disconnect between semantic output and low-altitude physical flight control commands. The original prediction results of multimodal large models are usually free-format natural language text with reasoning processes. However, UAV flight control commands require high real-time performance and determinism. Due to the lack of a safe and standardized set of actions, the text output of large models cannot be directly and reliably mapped to physical control parameters, making it difficult to form a closed-loop control of "semantic decision-physical flight".

[0010] Third, large end-to-end models are highly susceptible to "illusions" in complex scenarios. Existing open-language navigation methods tend to directly input the entire forward-facing image captured by the UAV into a large multimodal model for end-to-end action prediction. However, in low-altitude urban environments, the image contains high-density visual clutter such as buildings, trees, roads, and parasols, while the navigation target typically occupies only a tiny pixel area. Due to the lack of external prior guidance, the large model's attention is easily distracted by salient areas irrelevant to the task, leading to severe "illusion" behaviors in the output direction decisions, such as serious jitter, looping, or decision errors.

[0011] Fourth, there is a lack of scientific evaluation benchmarks for complex low-altitude visual-language navigation. Current evaluation benchmarks for autonomous driving or general-purpose robots cannot be directly transferred to the field of low-altitude UAV navigation. Existing evaluation methods are mostly limited to ideal environments, lacking a unified testing platform under extreme conditions such as GNSS rejection, open-vocabulary targets, and complex multi-scenario interactions. The lack of standardized evaluation benchmarks that include comprehensive quantitative indicators such as success rate, path efficiency, and navigation error hinders efficient and repeatable horizontal comparisons and iterative optimization in related research.

[0012] It should be noted that the information disclosed in the background section above is only for understanding the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0013] The main objective of this invention is to overcome the deficiencies in the aforementioned background technology and provide a zero-sample visual-language navigation method for low-altitude unmanned aerial vehicles (UAVs) in urban scenarios.

[0014] To achieve the above objectives, the present invention adopts the following technical solution: A zero-sample visual-language navigation method for low-altitude UAVs in urban scenarios includes the following steps: S1. Acquire multi-view images captured by the UAV's onboard camera, the multi-view images including a forward view, a forward depth view, and a top view; S2. Under the condition that the absolute position of the navigation target is unknown and GNSS signal is denied, the relative direction of the navigation target is calculated from the historical flight trajectory based on the weighted recursive prediction of the continuous historical track points before the loss of GNSS signal, and the heading correction information is generated. S3. Divide the forward graph into multiple image regions according to a predetermined partitioning strategy, calculate the semantic similarity between each image region and the natural language task target using an image-text matching model, select the region with the highest similarity and convert it into its corresponding physical location semantics, and generate semantic region prompts. S4. Input the multi-view image, the natural language task objective, the heading correction information and the semantic region prompt into the multimodal large model. Constrain the multimodal large model to output standardized action decision instructions through a predefined action set. The action decision instructions include action type and action amplitude parameters. S5. Parse and execute the action decision command to control the drone's flight, forming an autonomous navigation closed loop of visual perception, language reasoning, and flight control.

[0015] The present invention has the following beneficial effects: This invention provides a zero-sample visual-language navigation method for low-altitude UAVs in urban scenarios. This method integrates the forward-looking image, forward-looking depth image, and top-view image acquired by the UAV's onboard camera with the natural language task target, semantic region localization results, and UAV motion control commands into the same navigation process, constructing a closed-loop architecture of "visual perception - language command reasoning - flight control - autonomous navigation". This enables the UAV to autonomously search for, approach, and even dock targets based on open natural language commands without relying on external navigation information (such as GNSS signals, preset waypoints, or dedicated target detectors). This gives the UAV the ability to achieve stable autonomous navigation in low-altitude scenarios such as urban high-density obstruction, complex terrain, and frequent indoor-outdoor switching. To address the high dependence of traditional methods on GNSS signals, this invention proposes a heading maintenance mechanism based on weighted recursive prediction of historical tracks. Under conditions where the absolute position of the navigation target is unknown and GNSS signals are denied, the current heading and relative direction of the target can be estimated using only the existing historical positioning data of the airborne sensors. This enables the navigation system to maintain continuous and stable target search capabilities even in urban canyon environments with dense high-rise buildings and severe signal obstruction, significantly expanding the practical deployment scenarios and operational window of visual language navigation methods.

[0016] Meanwhile, this invention breaks through the closed-world limitation of traditional target detection methods, which require retraining for new targets. Users can directly describe navigation targets in any natural language. The system utilizes the image and text semantic understanding capabilities of the multimodal large model and the CLIP model to identify target types that have not appeared in the training data with zero samples during the inference stage, achieving true target localization under an open vocabulary. Addressing the "semantic overload" and hallucination behaviors (such as misjudgment of direction and looping) caused by task-irrelevant significant regions in the image during end-to-end decision-making in the multimodal large model, this invention innovatively introduces the CLIP nine-grid semantic region matching mechanism: the forward color image is divided into multiple regions corresponding to unique physical orientation semantics. The CLIP model calculates the image and text similarity between each region and the target text, selects the region with the highest score, and converts it into a natural language orientation cue, which is then added to the large model input as a lightweight, model-independent "large model hallucination suppressor". This mechanism does not enhance the model's own capabilities, but rather reduces the open-vocabulary target localization from a full-map search problem to a limited region selection problem by changing the structure of the input information. This fundamentally weakens the conditions that trigger illusions, causing the visual attention of the large model to shrink from the entire map to the high-probability region of the target, thus significantly improving the accuracy of direction judgment and decision stability.

[0017] Furthermore, to address the issue that the native free-format text output of multimodal large models cannot directly drive cross-modal mapping of physical flight control systems, this invention defines a standardized set of atomic actions (including translational, rotational, and stopping actions) covering all degrees of freedom of the UAV. It also strictly limits the output of large models to single action commands in the format of "action type + numerical amplitude" through prompt words, achieving an engineered interface for the deterministic conversion from natural language semantics to flight control. Simultaneously, it incorporates a two-layer safety closed-loop mechanism, including fault tolerance for continuous invalid output and safety obstacle avoidance verification based on depth maps, ensuring the legality of model output commands and flight safety. The five core modules of this invention (multi-view image acquisition, CLIP semantic region prompting, multimodal large model action decision-making, action parsing fault tolerance, and motion execution) all adopt a modular design, supporting flexible replacement and expansion of the image-text matching model, multimodal large model, and action set, adaptable to different UAV platforms and diverse urban scenario task requirements.

[0018] To systematically verify the technical effects of this invention, a standardized low-altitude urban airway evaluation benchmark covering multiple semantic categories, environmental complexities, and start / end conditions was constructed in a simulation platform. Multiple complementary indicators, including success rate, navigation error, path-weighted success rate, oracle success rate, and track length, were used for multi-dimensional quantitative evaluation. Experimental results show that the method of this invention achieves high success rates in target scenarios covering different semantic categories such as building entrances, urban furniture, public facilities, and artworks without any pre-training or fine-tuning. In particular, the introduction of the CLIP semantic region cueing mechanism significantly improves the navigation success rate and path efficiency of the multimodal large model, significantly reduces navigation error, and the actual flight path length does not increase and may even be shorter, fully demonstrating that this mechanism makes model decisions more accurate and efficient rather than simply increasing the number of flight steps.

[0019] In summary, this invention has achieved significant technological advancements in robust navigation under GNSS denial conditions, zero-sample open-vocabulary target localization capabilities, large-model illusion suppression effects, and engineering practicality and scalability, providing a complete and feasible solution for autonomous navigation of low-altitude urban UAVs.

[0020] Other beneficial effects of the embodiments of the present invention will be further described below. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the existing traditional UAV navigation technology architecture, illustrating the traditional UAV navigation process based on GNSS satellite positioning system and inertial measurement unit (IMU).

[0022] Figure 2This is an overall closed-loop flowchart of the zero-sample visual language navigation method for low-altitude unmanned aerial vehicles according to an embodiment of the present invention.

[0023] Figure 3 This is a front view of the UAV flying in a simulated environment in an embodiment of the present invention (the navigation target is a fountain).

[0024] Figure 4 This is a schematic diagram of CLIP nine-square grid semantic region localization and hallucination suppression in an embodiment of the present invention.

[0025] Figure 5 This is the overall flowchart of the zero-sample visual language navigation method for low-altitude UAVs in urban scenarios according to the present invention. Detailed Implementation

[0026] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0027] This invention aims to overcome the shortcomings of existing UAV navigation technologies, such as excessive reliance on GNSS signals, insufficient open-vocabulary semantic understanding capabilities, hallucination behavior of large multimodal models, and difficulties in cross-modal mapping. It proposes a zero-sample visual-language navigation method for low-altitude UAVs in urban scenarios. This method organically integrates multi-view visual perception, weighted heading prediction based on historical flight paths, CLIP semantic region cues, and standardized action decision-making of large models. It constructs a closed-loop navigation framework that can autonomously search and approach targets based on natural language commands without requiring preset target coordinates, dedicated target detectors, or continuous GNSS signals. By using lightweight external prior guidance rather than enhancing the model's own capabilities, it fundamentally suppresses hallucination behavior of large models in complex low-altitude scenes. Through a standardized interface of "action type + numerical amplitude," it achieves a deterministic mapping from semantic understanding to physical flight. Simultaneously, it includes standardized evaluation benchmarks and a multi-dimensional quantitative evaluation system, providing a repeatable, comparable, and iterative engineering solution for low-altitude urban visual-language navigation.

[0028] Specifically, in response to the above four technical bottlenecks, the embodiments of the present invention propose corresponding solutions: First, addressing the issues of heading maintenance and target orientation positioning under GNSS signal fading, this invention designs a heading maintenance mechanism based on weighted recursive prediction of historical tracks. When the absolute position of the navigation target is unknown and GNSS signal rejection occurs, the system utilizes only the historical flight trajectories recorded by onboard sensors before signal loss to extract multiple consecutive track points and calculate the track vectors between adjacent points. Considering that track information closer to the current time has greater reference value, an incrementally weighted summation of each track vector segment is used to obtain the predicted heading vector. By comparing this predicted heading with the current heading at the moment of GNSS failure, the required target azimuth correction angle and turning direction are calculated. This mechanism enables the UAV to maintain continuous estimation of the target orientation even after losing absolute positioning reference.

[0029] Second, to address the problem of large-scale illusion suppression in open-language navigation, this invention proposes a lightweight illusion suppression mechanism based on CLIP nine-square grid semantic region matching. The system divides the forward-facing color image of the UAV into multiple image patches in row-priority order, with each image patch corresponding to a unique physical orientation semantic (e.g., slightly left above, directly in front, slightly right in front, etc.). The CLIP image-text matching model is used to calculate the similarity between each image patch and the natural language task target, and the image patch with the highest score is selected, and its orientation is converted into a natural language prompt. This prompt is added to the input of the multimodal large model as a "large-scale illusion suppressor". Its role is not to enhance the model's own capabilities, but to reduce the dimensionality of open-vocabulary target localization from a full-image search problem to a limited region selection problem by changing the input information structure, thereby fundamentally weakening the "semantic overload" condition that causes model illusion.

[0030] Third, addressing the cross-modal mapping problem between large-scale model semantic decision-making and UAV control commands, this invention constructs a standardized UAV action set and safe execution mechanism based on "action type + numerical amplitude". This action set covers all degrees of freedom of the UAV, including translational, rotational, and stopping actions. Each translational action is accompanied by a distance parameter, and each rotational action is accompanied by a rotation angle parameter. The multimodal large model is strictly limited to outputting only single action commands conforming to this set's format through prompt words. Two layers of safety checks are also implemented: the first layer counts consecutive invalid outputs, automatically entering a safe hovering mode when a threshold is reached; the second layer reviews valid but potentially dangerous commands based on the forward depth map, implementing a pruning or rejection strategy. This mechanism achieves a reliable conversion from free-format natural language output to deterministic flight control commands.

[0031] Fourth, addressing the lack of evaluation benchmarks for visual language navigation in low-altitude cities, this invention constructs a standardized flight evaluation benchmark within a simulation platform. This benchmark covers multiple typical low-altitude urban air routes and encompasses various semantic categories such as building entrances, urban furniture, public facilities, and artworks, with environmental complexity gradients from simple to complex. The evaluation employs five complementary indicators—success rate, navigation error, path-weighted success rate, oracle success rate, and track length—for multi-dimensional quantitative assessment. Through paired control experiments such as "with / without CLIP," the independent gains of each module can be accurately quantified, providing a unified infrastructure for horizontal comparison and iterative optimization under different model configurations and scenarios.

[0032] See Figures 1 to 5 This invention provides a zero-sample visual-language navigation method for low-altitude unmanned aerial vehicles (UAVs) in urban scenarios, comprising the following steps: Step S1: Acquire multi-view images captured by the UAV's onboard camera. The multi-view images include a forward view, a forward depth view, and a top view.

[0033] In some embodiments, in the multi-view image in step S1: the forward view is used to provide semantic target and environmental appearance information in front; the forward depth visualization map is used to provide distance information of obstacles in front to assist obstacle avoidance; the top view is used to provide spatial layout information below and around the drone; the multimodal large model performs target search, distance judgment and action planning by fusing the above three types of image information and the semantic region prompts.

[0034] Step S2: Under the condition that the absolute position of the navigation target is unknown and the GNSS signal is denied, the relative direction of the navigation target is calculated from the historical flight trajectory based on the weighted recursive prediction of the continuous historical track points before the loss of the GNSS signal, and the heading correction information is generated.

[0035] In some embodiments, the weighted recursive prediction based on continuous historical track points before the loss of GNSS signal in step S2 includes: obtaining the coordinates of multiple historical track points set continuously before the loss of GNSS signal in the navigation coordinate system; sequentially calculating the track vector between adjacent historical track points; using an increasing weight allocation strategy where the weight increases as the distance from the current time increases, weighting and summing each segment of track vector to obtain the predicted heading vector; projecting the predicted heading vector and the current heading vector at the time of GNSS signal failure onto the horizontal plane respectively; calculating the required target azimuth correction angle based on the angle between the two projected vectors; and determining the turning direction based on the sign of the vertical component of the cross product of the two vectors.

[0036] Step S3: Divide the forward graph into multiple image regions according to a predetermined partitioning strategy, calculate the semantic similarity between each image region and the natural language task target using an image-text matching model, select the region with the highest similarity and convert it into its corresponding physical location semantics to generate semantic region prompts.

[0037] In some embodiments, generating semantic region prompts in step S3 includes: dividing the forward-facing color image of the UAV into multiple image patches with a preset number of rows and columns in row-priority order, each image patch corresponding to a uniquely defined physical orientation semantic of the UAV's field of view; calculating the image-text similarity between each image patch and the natural language task target using the CLIP image-text matching model; selecting the image patch with the highest similarity score from all image patches; and converting the physical orientation semantic of the selected image patch into orientation prompt text in natural language format, as part of the input of the multimodal large model.

[0038] In some embodiments, the semantic region cue in step S3 serves as a large model illusion suppressor. Its working mechanism is as follows: by changing the information structure of the multimodal large model input, the open vocabulary target localization is transformed from a full-image search problem into a limited region selection problem, thereby reducing the semantic overload and direction misjudgment caused by the distraction of model attention by significant task-irrelevant regions in the image.

[0039] In some embodiments, the semantic region prompt generated in step S3 and the multimodal large model in the following step S4 adopt the following "soft guidance, hard decision" interface design: the semantic region prompt, as auxiliary visual attention prior information, is added to the input prompt words of the multimodal large model in natural language form; the multimodal large model has complete final decision-making power, and its output action instructions may be inconsistent with the direction indicated by the semantic region prompt; when the similarity scores output by the image-text matching model for each image region are generally low and the dispersion is lower than a preset threshold, the reference weight of the semantic region prompt is automatically reduced, and the multimodal large model makes independent decisions based on multi-view images.

[0040] Step S4: Input the multi-view image, the natural language task objective, the heading correction information, and the semantic region prompt into the multimodal large model. Constrain the multimodal large model to output standardized action decision instructions through a predefined action set. The action decision instructions include action type and action amplitude parameters.

[0041] In some embodiments, the step S4 of constraining the multimodal large model to output standardized action decision instructions through a predefined set of actions includes: predefining a set of standardized atomic actions covering the degrees of freedom of the UAV's motion, wherein the set of atomic actions includes translational actions, rotational actions, and stopping actions; forcibly limiting the multimodal large model to output only a single action instruction in the input prompt, and the action instruction must belong to the set of atomic actions and be accompanied by a corresponding numerical amplitude parameter; performing format validation on the output of the multimodal large model, and only parsing it into an executable action instruction when the output format conforms to the preset specification.

[0042] In some embodiments, the motion amplitude parameter in step S4 is generated by the multimodal large model in real time based on multi-view visual input. The estimation basis includes: estimating the unobstructed available flight distance and obstacle distance based on the pixel-level distance information encoded in the forward depth map; assisting in judging the target distance based on the target's visual scale, texture details, and context in the forward map; determining the safe flight range based on the spatial layout below and around the UAV provided by the top view; the multimodal large model outputs a larger step size when the distance to the target is far to improve search efficiency, and outputs a smaller step size or stop command when approaching the target to ensure docking accuracy.

[0043] In some embodiments, the method further includes a safety verification and fault tolerance mechanism executed between step S4 and the following step S5. The mechanism includes: counting the consecutive invalid outputs of the multimodal large model; when the number of consecutive invalid outputs reaches a preset threshold, controlling the UAV to automatically enter a safe hovering mode and sending an alarm message; performing safety verification on dangerous commands that are valid in format but may cause collisions, the safety verification including range checks of motion amplitude parameters and obstacle avoidance safety threshold checks based on the forward depth map; when the forward distance of the forward command is detected to be greater than the distance of the obstacle directly in front in the forward depth map, executing a pruning strategy to correct the forward distance to a safe range, or executing a rejection strategy and requesting the model to regenerate the obstacle avoidance command.

[0044] Step S5: Parse and execute the action decision command to control the drone's flight, forming an autonomous navigation closed loop of visual perception, language reasoning, and flight control.

[0045] In some embodiments, the parsing and execution of the action decision command in step S5 includes: converting the parsed action type and action amplitude parameters into low-level control quantities based on the current flight state of the UAV; for translational actions, reading the current yaw angle of the UAV, mapping the relative motion speed in the body coordinate system to the world coordinate system, and controlling the UAV to fly a specified distance in the corresponding direction; for rotational actions, calculating the rotation time based on the set yaw angle velocity, and performing a fixed-angle turn; through the iterative decision-making and action execution closed loop of the multimodal large model, the UAV dynamically adjusts its flight step size as it gradually approaches the target, and finally outputs a stop command to complete the navigation task.

[0046] In some embodiments, the zero-sample visual language navigation method for low-altitude UAVs in urban scenarios further includes constructing a standardized flight evaluation benchmark for low-altitude urban environments. This benchmark specifically includes: constructing multiple typical low-altitude urban routes in a simulation platform, covering various semantic categories, environmental complexities, and start / end conditions. The starting point, target position, takeoff altitude, and environmental configuration of each route are precisely fixed. Multi-dimensional evaluation indicators are used to quantitatively assess navigation performance. These indicators include: success rate, navigation error, path-weighted success rate, oracle success rate, and track length. The success rate is the percentage of flights where the planar distance between the UAV's endpoint and the target is within a success threshold. The navigation error is the average planar distance between the UAV's endpoint and the target. The path-weighted success rate is a weighted indicator based on the success rate, calculated as the ratio of the shortest path length to the actual path length. The oracle success rate is the percentage of flights where the planar distance between any point in the flight track and the target is within a success threshold. The track length is the average total length of the actual flight path.

[0047] This invention proposes a zero-sample visual-language navigation method for low-altitude UAVs in urban scenarios. Under conditions where the absolute position of the navigation target is unknown and GNSS signals are denied, it can maintain heading and target orientation awareness solely by relying on weighted recursion of onboard historical flight paths. It generates lightweight orientation cues through CLIP nine-square grid semantic region matching, reducing the dimensionality of the large model's full-map search to region selection, effectively suppressing hallucination behavior at the information structure level. At the same time, it defines a standardized action set of "action type + numerical amplitude" and a safety verification mechanism to achieve deterministic mapping from natural language decision-making to physical flight control, and provides a standardized multi-dimensional evaluation benchmark. This endows UAVs with zero-sample open-vocabulary autonomous navigation capabilities in complex urban scenarios without the need for preset coordinates, dedicated detectors, or continuous GNSS signals.

[0048] The following further describes specific embodiments of the present invention, algorithm examples, and experimental verification.

[0049] A zero-sample visual-language navigation method for low-altitude UAVs in urban scenarios focuses on: 1. Weighted heading prediction based on historical tracks under urban low-altitude GNSS decay, solving the problem of UAVs losing absolute positioning reference and being unable to determine target location after GNSS signal loss; 2. Semantic region cues and large model illusion suppression based on CLIP, solving the problem of directional misjudgment and looping behavior caused by 'semantic overload' in multimodal large models during end-to-end decision-making on the whole map; 3. A set of UAV actions of "action + amplitude", solving the problem of cross-modal mapping where the free text output of large models cannot be directly and reliably converted into deterministic control commands of the flight control system; 4. Flight evaluation benchmarks based on multiple urban low-altitude environmental routes, solving the current problem of the lack of a unified and standardized evaluation system for repeated experiments on multiple scenarios and multiple models, acknowledging GNSS denial and open vocabulary targets.

[0050] The overall process includes task initialization, heading prediction based on historical tracks, multi-view perception, semantic region cues, multimodal action decision-making, and action parsing and execution. For the complete closed-loop process, see [link to full description]. Figure 2 This includes weighted heading prediction based on historical tracks under urban low-altitude GNSS decay conditions, semantic region cues and large-model illusion suppression based on CLIP, a set of "action + amplitude" UAV actions, and flight evaluation benchmarks based on 14 urban low-altitude airways. In a specific example, the system first sets the starting point, orientation, and takeoff altitude in the AirSim simulation environment, and then determines the navigation target based on the mission text, such as an orange and white striped parasol, a sculpture, the entrance to a restaurant on the opposite bank of the river, a hotel entrance, a blue amusement park ride, a fountain, a house entrance, a playground, a park bench, or an art sculpture, etc. Figure 3 As shown.

[0051] Weighted heading prediction based on historical flight paths In the "zero-sample open vocabulary navigation" scenario addressed by this invention, the absolute position of the navigation target is not known a priori at system startup. Users describe the target solely in natural language (e.g., "orange and white striped parasol ahead"), and the UAV relies entirely on its onboard visual sensors and the zero-sample semantic understanding capabilities of its multimodal large-scale model to search for and approach the target. This is fundamentally different from traditional navigation methods that rely on preset waypoints or target coordinates. When GNSS signals are lost, the core responsibility of the heading prediction module is precisely to predict "which direction the target is" solely from historical flight trajectories using weighted recursion, under conditions where the target coordinates are unknown—that is, to extract the target direction information implicit in historical flight tracks—in order to maintain navigation continuity under GNSS rejection conditions.

[0052] To address the issue of UAVs entering GNSS signal fading zones, this method innovatively proposes a heading maintenance mechanism based on historical track weighted prediction, enabling the UAV to maintain orientation awareness of the navigation target even after losing absolute positioning reference. Specifically, the system continuously records the position and orientation information in the flight trajectory before GNSS signal loss. When GNSS signal loss is detected, the system extracts the NED coordinates of the six most recent historical track points before signal loss, denoted as... ,in The trackpoint closest to the current moment, (NED coordinate system). From this, the vectors of 5 adjacent track segments can be calculated: Considering that track segments closer to the current time have greater reference value for the current heading, this method uses an increasing weight vector to perform a weighted summation of the above 5 track vector segments: The weights are set as follows: The values ​​increase progressively from far to near, summing to 1.0. To calculate the angle correction required on the XY plane for the UAV before resuming visual-language navigation after GNSS signals become unavailable, we will predict the heading vector. Projecting onto the XY plane yields a two-dimensional direction vector. At the same time, the fifth segment heading vector... XY plane components The direction the UAV is facing at the moment the GNSS signal fails. Then, the target azimuth correction angle required after GNSS fading. for: The direction of rotation is determined by the sign of the Z component of the cross product of vectors: if If the turn signal is positive, then it is a left turn; otherwise, it is a right turn.

[0053] The historical trajectory prediction mechanism has the following significant advantages: (1) It is based entirely on the existing historical positioning data of the airborne sensors, without the need for additional hardware; (2) The weighted recursive design makes the contribution of newer trajectory information greater, and has better tracking sensitivity to changes in flight direction. Multi-view perception The drone simultaneously acquires three types of images via its onboard camera: a forward-facing image for observing semantic targets and the appearance of the environment ahead; a forward-facing depth visualization image for determining the distance to obstacles and assisting in collision avoidance; and a top-down view for observing the spatial layout below and around the drone. The system encodes these images into image inputs that can be received by a multimodal large model, which, together with natural language task prompts, constitute the decision context, as shown below: in These represent the forward view, forward depth view, and top view, respectively, with T representing the target text description. The model's decision is... , Decision process of the representative model CLIP-based semantic region cues and large-model hallucination suppression When the entire forward-facing field of view of a drone is directly input into a large model for action decision-making without any spatial guidance, the model faces severe "semantic overload"—salient areas in the image unrelated to the navigation task (such as bright skies, brightly colored billboards, and large glass curtain walls of buildings) attract the model's visual attention, leading to a systematic bias in the model's judgment of target location. The essence of this "illusion" lies in the fact that large models, without spatial constraints, need to simultaneously perform multiple tasks such as target detection, location estimation, distance judgment, and action planning. The excessively long inference chain leads to the accumulation and amplification of errors in intermediate links.

[0054] The system divides the forward-facing color image into nine image patches in three rows and three columns, prioritizing each patch. These patches correspond to relative positions such as "slightly left above front," "slightly above front," "slightly right above front," "slightly left in front," "slightly right in front," "slightly left in front," "slightly left in front," "slightly below front," and "slightly right in front." The CLIP model then calculates the image-text similarity between each patch and the target text, selecting the patch with the highest score and converting its position into a verbal cue, such as "The navigation target is very likely slightly to the right in front of you." This cue does not directly replace the large model's decision but serves as a visual attention prior, helping the large model focus its reasoning on the local area most likely to contain the target.

[0055] In this invention, the CLIP semantic region cues and the Multimodal Large Model (MLLM) action decision-making adopt a "soft guidance, hard decision" interface design, with clearly defined responsibilities for each module. CLIP only provides lightweight spatial attention priors, and its output is appended to the end of the MLLM cues in natural language form, positioned as "assisting your judgment" rather than a mandatory instruction. MLLM, on the other hand, has complete final decision-making power: it simultaneously receives five types of information, including the forward RGB image, the forward depth image, the top view, the task description text, and the CLIP orientation cues, and autonomously decides the final action after comprehensively evaluating the global semantics of the scene. MLLM has the full right to output actions that are inconsistent with or even opposite to the CLIP cues—for example, when CLIP mistakenly identifies a non-target area as the highest similarity area due to background interference, MLLM can still output reasonable flight instructions through its own scene understanding capabilities (such as recognizing obstacles ahead that need to be avoided, or the top view showing that the target is actually on the other side).

[0056] For low-confidence scenarios, when the CLIP similarity scores of the nine regions are generally low and the dispersion is below a preset threshold (indicating that there are no regions in the image that significantly match the target text), the actual reference value of the CLIP location cues is limited, and its semantic weight in the cues naturally decreases. In this case, MLLM can still make autonomous decisions based on multi-view images and task descriptions, and the system will not be paralyzed due to CLIP failure. The above design ensures that CLIP is an "advisor" rather than a "commander," and MLLM is the final decision-maker. There is no mandatory compliance relationship between the two modules, and the system behavior is predictable.

[0057] Based on the above interface design, the CLIP model calculates the image-text similarity of the nine-grid region as follows: in, , indicating the first output of the CLIP model The degree of matching between the block image and the target text. The matching degree of the image patch with the highest matching degree is represented as follows: Figure 4 As shown. Figure 4 The paper demonstrates CLIP nine-square grid semantic region localization and illusion suppression, showing the process of forward image segmentation, image-text similarity calculation and orientation cue generation.

[0058] Multimodal large-scale model action decision-making and the establishment of "action + amplitude" UAV action sets In the multimodal large model action decision-making step, the system inputs the forward color map, forward depth map, top view, target task text, and the orientation cues into the multimodal large model. The cues predefine the UAV's mission objective, available action set, and output format, requiring the model to output only one action command without any explanatory statements. Preferred available actions include forward flight, backward flight, left flight, right flight, ascent, descent, left turn, right turn, and stop, with action parameters corresponding to movement distance or turning angle, respectively. The formal description of this mechanism is as follows: Let the multimodal large model be a function: in These represent the forward view, forward depth view, and top view, respectively, with T representing the target text description. Provides CLIP orientation hints. Model output. Must meet ,in The set of valid outputs under the above format constraints. The amplitude of the movement is measured in meters (when the drone is performing a flight maneuver) or degrees (when the drone is performing a turning maneuver). The system only operates when... If the output is valid, the corresponding action is executed; otherwise, the output is discarded and the model inference is requested again in the next decision cycle. The specific definitions are as follows: The motion amplitude parameters (such as forward distance and turning angle) in the above motion decision instructions are generated in real time by a multimodal large model based on multi-view visual input, and are not fixed step sizes. Specific estimation bases include: Ahead Depth Map: The depth visualization map presents pixel-level distance information in a color-coded manner. Multimodal large models can estimate the unobstructed available flight distance, the distance to obstacles, and the relative position of the target in the depth dimension from it. This is the core input source for amplitude estimation.

[0059] Forward color image: By observing the target's visual scale, texture details, and context, we can help determine the target's distance. If the target occupies a large proportion of the image, it means that the target is relatively close. In this case, the flight stride should be smaller to avoid overtaking the target. If the target is small, it means that the distance is far. The stride can be increased appropriately to speed up the arrival.

[0060] Top view: Provides the spatial layout below and around the drone, helping to determine the safe flight range and avoid flying out of the safe airspace due to excessive step size.

[0061] Cue word constraints: The cue words contain the task context ("It doesn't need to be too precise, just output stop when you get close to the target"), which guides the model to output larger steps when it is far from the target to improve efficiency, and to output smaller steps or stop instructions when it is close to the target to ensure accuracy.

[0062] In addition, amplitude accuracy is ensured through closed-loop iteration: the amplitude value output by the multimodal large model is not directly executed without verification. The system first performs physical feasibility verification on the parsed amplitude parameters (such as limiting the translation distance to a reasonable range, rejecting values ​​that exceed the envelope and re-requesting inference); even if the output is a safe but not accurate amplitude value, the subsequent "observation-decision-execution" closed loop will be dynamically corrected in the next iteration based on the new visual input, gradually converging to the target.

[0063] This step is the core of this method to achieve cross-modal mapping from "semantic understanding" to "physical flight". The native output of the multimodal large model is free-format natural language text. If it is directly connected to the flight control link without restriction, it will introduce at least three types of fatal risks: (a) unparseable output format - the model may add explanatory text such as "I think we should..." or "I suggest you..." before and after the action command, causing the parser to be unable to extract the effective command; (b) uncontrollable action space - the model may output ambiguous actions that the flight control system cannot execute (such as "move in the direction of the target"), or dangerous actions that exceed the physical envelope of the UAV; by establishing the above action set, the multimodal large model can output standardized control commands suitable for flight control system calls based on understanding the image scene and language target.

[0064] Security verification and fault tolerance mechanism This invention establishes a two-layer progressive security system: output validity verification (to deal with invalid formats) and physical security verification (to deal with valid but potentially dangerous instructions).

[0065] Layer 1: Fault Tolerance for Continuous Invalid Outputs. When the multimodal large model fails to validate the output format multiple times consecutively (action name not in the action set, missing parameters, or abnormal format), the system executes the following fault tolerance strategy: For the first 3 consecutive invalid outputs, the system discards each invalid output, inputs the same visual input in the next round with an error message ("The output format of the previous round is incorrect, please reconsider"), and the large model re-infers; For the 4th and subsequent consecutive invalid outputs (the consecutive invalid threshold is configurable), the system determines that the large model is currently in an abnormal state and automatically enters a safe hovering mode—the drone hovers at its current position while sending an alarm message to the ground station, waiting for manual takeover or a mission reset command.

[0066] The second layer: Review of dangerous commands that are valid in format but may cause collisions. Even if the large model outputs commands with correct format, the system still performs the following safety checks before execution: parameter range check—for example, translation distance is limited to 0.5 to 20.0 meters, rotation angle is limited to 5 to 180 degrees, parameters that exceed the physical envelope are rejected and re-inferenced; depth obstacle avoidance safety gate—before executing a forward command, the system reads the obstacle distance in the directly forward area of ​​the forward depth map. If the distance is less than the forward distance in the command, the system executes a pruning strategy (automatically pruning the forward distance to a safe distance before execution) or a rejection strategy (completely rejecting dangerous commands and adding obstacle warnings in the next round, requiring the model to regenerate obstacle avoidance commands).

[0067] In addition, the system also has timeout protection: a maximum decision-making step limit is set for each navigation task (e.g., 20 to 40 steps, depending on the complexity of the scenario). Once the limit is reached, the drone automatically hovers and issues an alarm to prevent infinite flight caused by the model not outputting a stop command for an extended period. These two fault-tolerance mechanisms constitute a complete safety closed-loop logic, ensuring that the drone will not collide or go out of control due to abnormal model output during actual flight.

[0068] Simulation environment action execution In the action execution steps, the system converts the parsed actions into low-level control variables based on the UAV's current flight state. For forward, backward, left, and right movements, the system first reads the UAV's current yaw angle, then maps the relative velocity in the body coordinate system to the world coordinate system, and then controls the UAV to fly a predetermined distance in the corresponding direction. For ascent and descent movements, the system converts the vertical velocity and duration according to the altitude definition of the low-altitude flight coordinate system. For left and right turns, the system calculates the turning time based on the set yaw rate and executes a fixed-angle turn. In this way, natural language-driven action decisions can be stably mapped to the continuous control process of a multi-rotor UAV. This execution process, in conjunction with the aforementioned amplitude closed-loop correction mechanism, achieves a stable mapping from semantic decisions to physical flight.

[0069] Flight evaluation benchmarks based on 14 urban low-altitude environment routes To systematically evaluate the navigation capabilities of this method under different urban low-altitude environments and target types, this invention constructs a standardized evaluation benchmark covering 14 typical low-altitude urban air routes in the AirSim high-fidelity multi-rotor UAV simulation platform. The design of this benchmark follows the following principles: (1) Openness and diversity of target types - covering building entrances (hotel main entrances, house entrances), urban furniture (parasols, park benches, fountains), public facilities (playgrounds, amusement facilities), artworks (figure sculptures, art sculptures) and naturally separated scenes (restaurants on the opposite bank of the river); (2) Gradual distribution of environmental complexity - including simple open scenes (such as in front of detached houses), moderately obstructed scenes (such as hotel entrances in building complexes) and highly complex scenes (such as restaurants that need to cross rivers, playgrounds with large open spaces); (3) Span of target scale and distance - including small targets at close range (such as park benches, with a starting distance of about 30m) and large scene targets at long distance (such as blue amusement facilities, with a starting distance of about 60m); (4) Differentiated setting of flight altitude - setting takeoff altitudes ranging from 1m to 18m according to the altitude and visual characteristics of different targets.

[0070] The above benchmark has the following characteristics: (1) Zero sample size - all target descriptions are free text in natural language and do not belong to any predefined fixed category set. The system has never seen the correspondence of these targets in any labeled data before the test, ensuring that the evaluation truly reflects the open vocabulary navigation capability; (2) Repeatability - the starting point, target position, take-off altitude and environmental configuration of each scenario are precisely fixed, ensuring fair comparison between different models and different configurations; (3) Statistical significance - each model configuration and each scenario combination is subjected to 10 independent repeated flights, which has sufficient statistical power.

[0071] To comprehensively quantify navigation performance, this benchmark adopts the following five evaluation indicators, assuming the total number of flights is N, and the actual waypoint sequence of the i-th flight is... The navigation target coordinates are The distance threshold δ was successfully determined.

[0072] (1) Success Rate (SR): The percentage of flights in which the 2D Euclidean distance between the UAV's endpoint and the navigation target is within a preset success threshold. This indicator directly reflects the capability to complete the navigation task. The formula is as follows: like , ;otherwise, .

[0073] in Represents the 2D Euclidean distance in the XY plane. SR directly reflects the ability to complete navigation tasks.

[0074] (2) Navigation Error (NE): The average 2D Euclidean distance between the UAV's final position and the navigation target, in meters. This metric quantifies the degree of deviation when the UAV fails to reach the target. The formula is as follows: NE is measured in meters. NE quantifies the average deviation between the UAV's final position and the target.

[0075] (3) Success weighted by path length (SPL): This is a weighting factor for path efficiency introduced on top of the success rate. The calculation method is as follows: ,in This is a success flag for the i-th flight (1 or 0). The shortest straight-line distance from the starting point to the target. This represents the actual flight path length for the i-th flight. This metric comprehensively evaluates the accuracy of navigation and the efficiency of path planning. The formula is as follows: in Let be the total actual track length of the i-th flight. The shortest straight-line distance from the starting point to the target. SPL incorporates path efficiency weighting on top of success rate, simultaneously measuring navigation accuracy and path rationality.

[0076] (4) Oracle Success Rate (OR): The percentage of flights in the entire flight path where at least one trajectory point falls within the success threshold. This metric distinguishes between two different failure modes: "approaching the target but failing to stop successfully" and "failing to approach the target at all," assisting in diagnosing the specific reasons for navigation failure. The formula is as follows: like , ;otherwise, .

[0077] OR measures whether at least one point in the flight path has entered the vicinity of the target. By comparing the difference between OR and SR, it is possible to diagnose whether the navigation failure is due to "approaching the target but failing to stop successfully" or "failing to approach the target at all".

[0078] (5) Trajectory Length (TL): The total length of the actual flight path of the UAV from the starting point to the ending point, in meters. This indicator is used in conjunction with SPL to determine whether the model has made an invalid detour due to misjudgment of direction. The formula is as follows: TL is measured in meters. TL is used in conjunction with SPL. If high SR is accompanied by high TL and low SPL, it indicates that the model has an invalid detour due to orientation misjudgment.

[0079] Based on the above analysis, CLIP semantic region suggestion can be regarded as a lightweight, model-agnostic "large model illusion suppressor". Its working mechanism is not to enhance the capabilities of the model itself, but to reduce the difficulty of the task by changing the information structure of the input - reducing the open vocabulary target localization from a full-graph search problem to a nine-to-one region selection problem, thereby weakening the "semantic overload" condition that causes model illusion from the root.

[0080] Experimental Example: Core Mechanism and Experimental Results (1) Navigation target orientation positioning mechanism based on historical track weighted prediction under GNSS rejection conditions.

[0081] When the absolute position of the navigation target is unknown and GNSS signal is lost, the system uses the six most recent historical waypoints from the preceding flight trajectory to perform a weighted summation of five track vectors with increasing weights (0.10, 0.15, 0.20, 0.25, 0.30) to predict the current heading and the target's relative bearing. In 14 test scenarios, the predicted turning angles covered 17.4° to 54.6°, including seven left-turn and seven right-turn scenarios. This mechanism enables the navigation system to maintain the continuity of target search and directional stability even under GNSS rejection conditions.

[0082] (2) Multimodal large model navigation illusion suppression mechanism based on CLIP nine-square grid semantic region matching.

[0083] The forward-facing color image of the drone is divided into a 3×3 grid, with each region corresponding to a unique physical orientation semantic (upper left, directly in front, lower right, etc.). The CLIP model calculates the image-text similarity between each region and the target text, selects the region with the highest score, and converts its orientation into a natural language prompt, which is then added to the input of the large model to suppress large model illusions. This mechanism reduces the visual attention of the large model from the entire image to high-probability target regions, reducing "semantic overload" at the source of information structure. Experiments show that after introducing CLIP, the navigation success rate of the Qwen3.5-35B-A3B model increased from 42.9% to 80.7% (+88%), the path efficiency (SPL) increased from 37.3% to 68.7% (+84%), and the trajectory length was shorter, verifying the significant suppression effect of this mechanism on large model orientation misjudgment and invalid detours.

[0084] (3) "Action type + numerical range" standardization of UAV action set and safety execution mechanism.

[0085] Nine atomic action sets covering all degrees of freedom of the UAV are defined: forward, backward, left, right, up, down, turnleft, turnright, and stop. Each translational action is followed by the distance traveled (in meters), and each rotational action is followed by the rotation angle (in degrees). Cue words are used to strictly limit the output of large models to only single action commands in a fixed format. This action set achieves a deterministic mapping from natural language commands to physical flight control.

[0086] (4) Standardized evaluation benchmarks and multi-dimensional evaluation system for 14 low-altitude urban air routes.

[0087] A standardized evaluation benchmark covering 14 typical low-altitude urban air routes was constructed in the AirSim simulation platform, with environmental complexity ranging from simple to high, takeoff distances from 15m to 60m, and takeoff altitudes from 1m to 18m. The evaluation employed five complementary indicators—SR (Success Rate), NE (Navigation Error), SPL (Path Weighted Success Rate), OR (Oracle Success Rate), and TL (Trajectory Length)—for multi-dimensional diagnostic analysis. Through a paired control experiment with and without CLIP (Clip Indicator), the independent gain contribution of CLIP semantic region cues to each multimodal large model could be accurately quantified.

[0088] Based on the experimental results, compared with traditional UAV navigation methods based on preset waypoints or dedicated detectors, the method of this invention demonstrates significant technical advantages, specifically in the following aspects: (1) Robust navigation capability under GNSS fading conditions. This method, through a weighted heading prediction mechanism based on 6 historical flight paths, can still maintain basic heading awareness and target direction estimation even when GNSS signals are completely lost. The turning angles of the 14 test scenarios cover a wide range from 17.4° to 54.6°, and include both left and right turns, verifying the universality of the heading prediction mechanism under diverse flight paths. This capability enables the navigation system to maintain continuous operation in urban canyon environments with dense high-rise buildings and severe signal obstruction, significantly expanding the practical deployment scenarios and operational window of visual language navigation methods.

[0089] (2) Zero-shot open-vocabulary target localization capability. Users can directly describe navigation targets in any natural language. The system utilizes the textual and semantic understanding capabilities of the multimodal large model and CLIP model to identify target types that have not appeared in the training data with zero samples during the inference stage. This completely breaks the closed-world limitation of traditional detection and navigation methods that "identify fixed categories and require re-collection, labeling, and retraining of the model for new targets". In 14 target scenarios covering different semantic categories and visual features such as building entrances, urban furniture, public facilities, and artworks, this method achieved an overall success rate of 80.7% without any pre-training or fine-tuning, fully demonstrating its generalization and deployment capabilities in dynamic and open urban environments.

[0090] (3) The large model illusion suppression effect guided by CLIP spatial prior. Compared with directly handing the entire image to the large model for end-to-end decision-making, this method provides explicit orientation cues through CLIP matching of nine-grid areas, which significantly improves the accuracy of orientation judgment of the large model in complex low-altitude urban scenes. Experimental data show that the introduction of CLIP semantic cues increases the success rate of the 35B model from 42.9% to 80.7% (+88%), the path efficiency SPL from 37.3% to 68.7% (+84%), and the navigation error from 15.94m to 9.81m (-38.5%), while the trajectory length is shorter (41.75m vs 44.01m), which fully demonstrates that the spatial cues of CLIP make the model decision more accurate and efficient, rather than simply increasing the number of flight steps.

[0091] (4) Standardized engineering interface for "action + amplitude". This method strictly limits the output of the large model to a fixed format of 9 atomic actions, ensuring that only legal, safe and executable instructions enter the flight control link. This structured output design of "action type + numerical parameters" realizes the deterministic conversion from natural language semantics to flight control, making this method feasible for direct integration into actual flight control systems.

[0092] (5) Systematic multi-dimensional experimental verification. This method constructed 14 standardized low-altitude urban airway benchmarks and used five complementary evaluation indicators (SR / NE / SPL / OR / TL) for multi-dimensional cross-analysis. The evaluation results formed a complete chain of evidence, comprehensively verifying the technical effectiveness and engineering practical value of this method from multiple dimensions such as success rate (80.7%), navigation accuracy (9.81m), path efficiency (68.7%), and stopping reliability (SR-OR difference of only 6.4%).

[0093] (6) Modular and Scalable Architecture. The five core modules of this method—multi-view image acquisition, CLIP semantic region prompting, MLLM action decision-making, action parsing fault tolerance, and motion execution—all adopt a modular design. The CLIP model can be replaced with other image-text matching models, the MLLM can be upgraded to a more powerful version, the action set can be expanded as needed (such as adding hovering, return, etc.), and new scene types can be added to the evaluation benchmark. This architecture design supports continuous iteration of technology and incremental expansion of scenarios, and can be adapted to different UAV platforms, different urban environments, and different mission requirements.

[0094] In summary, this invention provides a visual-language navigation method for UAVs in low-altitude urban environments. This method utilizes the forward-looking image, forward-looking depth image, and top-view image acquired by the UAV's onboard camera to unify the natural language task objective, multi-view visual observation, semantic region localization results, and UAV motion control commands into the navigation process, forming a closed loop of UAV visual perception-language command reasoning-flight control-autonomous navigation. This enables the UAV to autonomously search for, approach, and even dock targets based on open language commands without relying on external navigation information, thereby possessing the ability to autonomously navigate in low-altitude scenarios with high-density obstruction, complex terrain, and frequent indoor-outdoor transitions in urban environments.

[0095] This invention also provides a storage medium for storing a computer program, which, when executed, performs at least the methods described above.

[0096] This invention also provides a control device, including a processor and a storage medium for storing a computer program; wherein the processor executes the computer program by performing at least the method described above.

[0097] This invention also provides a processor that executes a computer program, at least performing the methods described above.

[0098] The storage medium can be implemented by any type of non-volatile storage device, or a combination thereof. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc or CD-ROM; magnetic surface memory can be disk storage or magnetic tape storage. The storage media described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0099] In the several embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0100] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0101] In addition, in the various embodiments of the present invention, each functional unit can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0102] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0103] Alternatively, if the integrated units of this invention are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, RAM, magnetic disks, or optical disks.

[0104] The methods disclosed in the several method embodiments provided by this invention can be arbitrarily combined without conflict to obtain new method embodiments.

[0105] The features disclosed in the several product embodiments provided by this invention can be arbitrarily combined without conflict to obtain new product embodiments.

[0106] The features disclosed in the several method or device embodiments provided by the present invention can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0107] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various equivalent substitutions or obvious modifications can be made without departing from the concept of the present invention, and all such modifications, achieving the same performance or application, should be considered within the scope of protection of the present invention.

Claims

1. A zero-sample visual-language navigation method for low-altitude unmanned aerial vehicles (UAVs) in urban scenarios, characterized in that, Includes the following steps: S1. Determine the navigation target based on the task text, wherein the navigation target is a natural language task target described in natural language, and acquire multi-view images collected by the UAV's onboard camera, wherein the multi-view images include a forward view, a forward depth view, and a top view. S2. Under the condition that the absolute position of the navigation target is unknown and GNSS signal is denied, the relative direction of the navigation target is calculated from the historical flight trajectory based on the weighted recursive prediction of the continuous historical track points before the loss of GNSS signal, and the heading correction information is generated. S3. Divide the forward graph into multiple image regions according to a predetermined partitioning strategy, calculate the semantic similarity between each image region and the natural language task target using an image-text matching model, select the region with the highest similarity and convert it into its corresponding physical location semantics, and generate semantic region prompts. S4. Input the multi-view image, the natural language task objective, the heading correction information and the semantic region prompt into the multimodal large model. Constrain the multimodal large model to output standardized action decision instructions through a predefined action set. The action decision instructions include action type and action amplitude parameters. S5. Parse and execute the action decision command to control the drone's flight, forming an autonomous navigation closed loop of visual perception, language reasoning, and flight control.

2. The method according to claim 1, characterized in that, In the multi-view image described in step S1: The forward graph is used to provide semantic target and environmental appearance information in front; The forward depth map is used to provide distance information to obstacles ahead to assist in obstacle avoidance; The top view is used to provide information on the spatial layout below and around the drone; The multimodal large model integrates the above three types of image information and the semantic region prompts to perform target search, distance judgment and action planning.

3. The method according to claim 1, characterized in that, The weighted recursive prediction based on continuous historical track points before GNSS signal loss in step S2 includes: Obtain the coordinates of multiple historical waypoints set consecutively before the GNSS signal was lost in the navigation coordinate system; Calculate the track vectors between adjacent historical waypoints in sequence; An incremental weighting strategy is adopted to assign greater weight to segments of the trajectory vector by weighted summation, which is closer to the current time; The predicted heading vector and the current heading vector at the moment the GNSS signal fails are projected onto the horizontal plane respectively. The required target azimuth correction angle is calculated based on the angle between the two projected vectors, and the turning direction is determined based on the sign of the vertical component of the cross product of the two vectors.

4. The method according to claim 1, characterized in that, The generation of semantic region hints in step S3 includes: The forward-facing color image of the UAV is divided into multiple image blocks with a preset number of rows and columns in row-priority order. Each image block corresponds to a uniquely defined physical orientation semantic of the UAV's field of view. The CLIP image-text matching model is used to calculate the image-text similarity between each image patch and the natural language task target. Select the image patch with the highest similarity score from all image patches; The physical orientation semantics corresponding to the selected image patch are converted into orientation prompt text in natural language format and used as part of the input of the multimodal large model.

5. The method according to claim 1, characterized in that, The semantic region prompts mentioned in step S3 serve as a large model illusion suppressor. Their working mechanism includes: by changing the information structure of the multimodal large model input, the open vocabulary target localization is transformed from a full-image search problem into a limited region selection problem, thereby reducing the semantic overload and direction misjudgment caused by the distraction of model attention by significant task-irrelevant regions in the image.

6. The method according to claim 1, characterized in that, The semantic region hints generated in step S3 and the multimodal large model in step S4 adopt the following interface design, specifically including: The semantic region cues, as auxiliary visual attention prior information, are appended to the input cues of the multimodal large model in the form of natural language. The multimodal large model has complete final decision-making power, and its output action instructions may not be consistent with the direction indicated by the semantic region prompts; When the similarity scores output by the image-text matching model for each image region are generally low and the dispersion is lower than a preset threshold, the reference weight of the semantic region prompt is automatically reduced, and the multimodal large model makes independent decisions based on multi-view images.

7. The method according to claim 1, characterized in that, Step S4, which involves constraining the multimodal large model to output standardized action decision instructions using a predefined set of actions, includes: A standardized set of atomic actions covering the motion degrees of freedom of the UAV is predefined, the set of atomic actions including translational actions, rotational actions and stopping actions; The input prompts force the multimodal large model to output only a single action command, and the action command must belong to the atomic action set and be accompanied by a corresponding numerical amplitude parameter. The output of the multimodal large model is format-validated, and only when the output format conforms to the preset specification is it parsed into an executable action instruction; Furthermore, the motion amplitude parameter is generated by the multimodal large model in real time based on multi-view visual input, and its estimation basis includes: Based on the pixel-level distance information encoded in the forward depth map, the available flight distance without obstacles ahead and the distance to obstacles are estimated. Based on the visual scale, texture details, and context of the target in the forward image, the distance to the target is determined. Based on the spatial layout of the area below and around the drone provided in the top view, the safe flight range is determined; The multimodal large model outputs a larger step size when it is far from the target to improve search efficiency, and outputs a smaller step size or a stop command when it is close to the target to ensure docking accuracy.

8. The method according to claim 1, characterized in that, It also includes a security verification and fault tolerance mechanism executed between step S4 and step S5, the mechanism including: The continuous invalid outputs of the multimodal large model are counted. When the number of consecutive invalid outputs reaches a preset threshold, the drone is controlled to automatically enter the safe hovering mode and send an alarm message. Safety checks are performed on dangerous commands that are valid in format but may cause collisions. The safety checks include range checks of action amplitude parameters and obstacle avoidance safety threshold checks based on forward depth maps. When the forward distance of the forward command is detected to be greater than the distance to the obstacle directly in front in the forward depth map, a pruning strategy is executed to correct the forward distance to a safe range, or a rejection strategy is executed and the model is requested to regenerate the obstacle avoidance command.

9. The method according to claim 1, characterized in that, The parsing and execution of the action decision instruction in step S5 includes: Based on the current flight status of the drone, the parsed action type and action amplitude parameters are converted into low-level control variables; For translational movements, the current yaw angle of the UAV is read, the relative motion speed in the body coordinate system is mapped to the world coordinate system, and the UAV is controlled to fly a specified distance in the corresponding direction. For rotational movements, the rotation time is calculated based on the set yaw rate, and a fixed-angle turn is executed. Through the iterative decision-making and action execution closed loop of the multimodal large model, the UAV dynamically adjusts its flight step size as it gradually approaches the target, and finally outputs a stop command to complete the navigation task.

10. The method according to any one of claims 1 to 9, characterized in that, It also includes establishing standardized flight evaluation benchmarks for low-altitude urban environments, the benchmarks including: Multiple typical low-altitude urban air routes covering various semantic categories, environmental complexities, and start and end conditions are constructed in the simulation platform. The starting point, target position, take-off altitude, and environmental configuration of each air route are precisely fixed. A multi-dimensional evaluation index is used to quantitatively evaluate navigation performance. The multi-dimensional evaluation index includes: success rate, navigation error, path-weighted success rate, oracle success rate, and track length. Among them, the success rate is the percentage of flights where the plane distance between the drone's endpoint and the target is within the success threshold; the navigation error is the average plane distance between the drone's endpoint and the target; the path-weighted success rate is an indicator that is weighted based on the success rate by the ratio of the shortest path length to the actual path length; the oracle success rate is the percentage of flights where the plane distance between any point in the flight path and the target is within the success threshold; and the path length is the average of the total length of the actual flight path.

Images