Unmanned aerial vehicle visual language navigation zero fine-tuning method based on fine-grained cognitive function module integration

Abstract

The application provides a zero fine-tuning method for unmanned aerial vehicle visual language navigation based on fine-grained cognitive function module integration, which comprises an instruction analysis process, a judgment and attention process, a perception process, a judgment and sub-target extraction process, a judgment and imagination process, a sub-target completion judgment, a decision and multi-level memory process. The application integrates various biological inspired functions, including perception, attention, memory, imagination, problem solving, reasoning and decision making and the like. By constructing each functional component based on a medium-sized language large model and a visual-language large model and interacting, the framework does not need to be trained for a specific task, and exhibits strong performance and human-like exploration behavior. Experimental results show that the application performs superiorly in various zero sample fine-tuning methods, highlighting the potential and prospect of the modular architecture framework design in solving multi-modal embodied intelligent agent tasks.

Description

Technical Field

[0001] This invention relates to the field of UAV visual language navigation technology, specifically a zero-fine-tuning method for UAV visual language navigation based on the integration of fine-grained cognitive function modules. Background Technology

[0002] Vision-Language Navigation (VLN) is one of the core challenges in the field of embodied intelligence, requiring agents to generate dynamic trajectories and execute sequential actions in complex environments based on multimodal perception (visual and linguistic input). The rapid development of UAV technology has made aerial VLN a research hotspot, with unique challenges including: 1. Navigation in a continuous three-dimensional space with high degrees of freedom; 2. Limited global perception based on an egocentric perspective; 3. Long-term path planning requirements; 4. Lower landmark differentiation in aerial scenes compared to indoor environments. These inherent characteristics pose challenges to traditional data-driven methods. While expanding the scale of training data has proven effective in indoor ground-based VLN, this strategy is unsustainable in UAV navigation because the cost of dataset construction and model training in UAV application scenarios increases exponentially, necessitating the development of solutions to mitigate data dependence.

[0003] Thanks to breakthroughs in large-scale foundation models (LFMs, including large language models (LLMs) and multimodal large models (MLLMs), zero-shot, no-training frameworks driven by these technologies have achieved good generalization capabilities in ground-based VLN. However, even when using state-of-the-art LFMs (such as the GPT-4 series) as foundational modules, existing frameworks still struggle with aerial VLN tasks. This indicates that developing effective zero-fine-tuning UAV VLN frameworks requires breaking through existing paradigms rather than simply relying on powerful LFMs. Summary of the Invention

[0004] To address the problems of existing technologies, this invention proposes a zero-shot method for UAV visual-language navigation (VLN). This method integrates fine-grained cognitive modules driven by large-scale foundational models (LFMs). The structure of these modules simulates core human cognitive functions, including language processing, perception, attention, memory, imagination, problem-solving, reasoning, and planning. Through the construction of sophisticated functional modules, collaborative interaction between modules, designed cueing engineering, and structured information flow, the UAV visual-language navigation task is completed based solely on a medium-sized language model (LLM) and visual-language model (VLM).

[0005] The technical solution of this invention is as follows:

[0006] A method for zero-fine-tuning UAV visual language navigation based on fine-grained cognitive function module integration includes the following steps:

[0007] Step 1: Input the navigation commands and the set command parsing prompts into the pre-trained large model to realize command parsing and obtain a list of several command sentences and the corresponding landmarks for each command sentence;

[0008] Step 2: If the current instruction is the first instruction or the instruction has changed at the current time step, select the current instruction and the next instruction, as well as the landmark corresponding to the current instruction and the landmark corresponding to the next instruction at the current time step. Combine the set attention cue words and input them into the pre-trained large model to obtain the attention problem corresponding to the current instruction and proceed to step 3. Otherwise, proceed directly to step 3.

[0009] Step 3: Input the attention problem corresponding to the current instruction and the image acquired at the current time step, along with the set perception prompts, into the pre-trained large model to obtain the scene description at the current time step; and perform collision warning based on the depth map at the current time step.

[0010] Step 4: If the current instruction is the first instruction or the instruction changes at the current time step, input the current instruction and the scene description of the current time step obtained in Step 3, along with the sub-target extraction prompts, into the pre-trained large model to obtain the sub-target list, and proceed to Step 5; otherwise, proceed directly to Step 5.

[0011] Step 5: If the current sub-target is the first sub-target or a sub-target switch occurs at the current time step, select the current sub-target and the next sub-target, as well as the landmark corresponding to the current command sentence, and input them into the pre-trained large model in combination with the set imagination prompt words to obtain the possible visual state after the current sub-target is completed, and proceed to step 6; otherwise, proceed directly to step 6.

[0012] Step 6: Input the current sub-target and the next sub-target, the possible visual state after the current sub-target is completed, the scene description at the current time step, and the original sub-target memory sequence of the current sub-target, along with the set sub-target completion judgment prompt words, into the pre-trained large model to determine whether the current sub-target is completed;

[0013] If the current sub-target is not completed, a decision-making process is performed: the instruction memory sequence, the current time step scene description, collision warning, the preset list of valid actions, and the current sub-target are input into the pre-trained large model along with the set decision prompts to obtain the optimal action selection; the current time step scene description, the optimal action selection, and the landmark corresponding to the current instruction sentence are input into the pre-trained large model along with the set memory prompts to obtain the single-step memory; the single-step memory is inserted into the original sub-target memory sequence of the current sub-target, and the UAV executes the optimal action selection, the current time step ends, and the process returns to step 3 to proceed with the perception process of the next time step;

[0014] If the current sub-target has been completed, the original sub-target memory sequence, combined with the set compressed prompt words, is input into the pre-trained large model to obtain the compressed sub-target memory sequence, and the original sub-target memory sequence is cleared. Then, it is determined whether the current sub-target is the last sub-target in the sub-target list. If it is the last sub-target, it is further determined whether the current instruction sentence is the last instruction sentence of the current navigation instruction. If it is the last instruction sentence, the task ends. If it is not the last instruction sentence, it returns to step 2 to switch instruction sentences. If it is not the last sub-target, it returns to step 5 to switch sub-targets according to the sub-target list.

[0015] Furthermore, in step 6, when determining whether the current sub-target has been completed, if the current sub-target is the last sub-target in the sub-target list corresponding to the current instruction sentence, then the next sub-target is the next instruction sentence.

[0016] Furthermore, in step 1, the instruction parsing prompts include cognitive roles, overall tasks, precautions, input domains, output format constraints, and expected outputs; wherein the overall tasks include two tasks: instruction sentence segmentation and landmark extraction, the input domain is filled with navigation instructions, and the prompts require a standardized output format.

[0017] Furthermore, in step 2, the attention prompt includes cognitive role, task, input domain, output constraint, and expected output; wherein the input domain is filled with the current instruction sentence and its landmark, as well as the next instruction sentence and its landmark, and the prompt requires a standardized output format.

[0018] Furthermore, in step 3, the perceptual prompts include cognitive roles, tasks, input domains, output constraints, precautions, and expected outputs. The input domain includes a first-person RGB image and the attention problem corresponding to the current instruction. The output adopts a hierarchical structured output method of global information + detailed information. First, a concise summary description is used to quickly summarize the core features of the current scene. Then, the scene is divided into three regions according to spatial orientation: the center, the left side, and the right side. The visual content of each region is described in detail. All description results are output in a standardized format.

[0019] Furthermore, in step 4, the sub-goal extraction prompts include cognitive roles, overall tasks, input domains, detailed tasks, precautions, output constraints, and expected outputs; wherein the input domain includes the current instruction sentence and the scene description of the current time step; through means-ends analysis of the current instruction sentence using a large language model, all executable sub-goals or intermediate goals contained therein are identified, and combined with scene-aware information, unachievable or irrelevant content is eliminated to ensure that each sub-goal is executable in the current environment and to avoid generating false or unattainable goals; all sub-goals are output in a standardized format.

[0020] Furthermore, in step 5, the imagined prompts include cognitive roles, tasks, input domains, output constraints, precautions, and expected outputs; wherein the input domain is filled with the current sub-target and the next sub-target, as well as the landmark corresponding to the current command sentence, and the key visual states of the environment visible in front of the drone after the completion of the current sub-target are generated by the large language model; the output visual expectation description focuses on the spatial location and relative distance of the landmarks; the visual expectation description is output in a standardized format.

[0021] Furthermore, in step 6, the sub-goal completion judgment prompts include cognitive role, task, input domain, precautions, additional requirements, output constraints, and expected output; the output includes the judgment result of whether the sub-goal has been completed or not, and the reason for making the judgment of completion or non-completion.

[0022] Furthermore, in step 6, the decision prompts include cognitive roles, tasks, input domains, precautions, additional requirements, output constraints, and expected outputs; the outputs include three parts: a normalized probability distribution of candidate actions, which represents the UAV decision system's preference for each action in a quantitative way; the final action selected, i.e., the action number with the highest probability; and the decision-making process, i.e., the reasons and reasoning links for this action selection.

[0023] Furthermore, in step 6, the memory prompts include cognitive roles, tasks, input domains, precautions, output constraints, and expected outputs; the observations and actions at each time step are recorded through single-step memory; during the execution of a sub-goal, the system continuously accumulates the corresponding single-step memories to form the original sub-goal memory sequence of the current sub-goal; the original sub-goal memory sequence composed of single-step memories is summarized and compressed through a large language model to generate structured compressed sub-goal memories; the compressed memories of all completed sub-goals, as well as the original sub-goal memory sequence of the current sub-goal, are dynamically combined to form the current instruction memory sequence.

[0024] Beneficial effects

[0025] This invention focuses on the visual-language navigation (VLN) task for unmanned aerial vehicles (UAVs), which presents many unique challenges compared to terrestrial scenarios, such as high-degree-of-freedom 3D spatial exploration, long-term path planning, and navigation reference based on fuzzy landmarks from a first-person perspective. To address these challenges, this invention proposes FineCog-Nav, a training-free modular framework inspired by human cognitive processes. This framework integrates various biologically inspired functional modules, including perception, attention, memory, imagination, problem-solving, reasoning, and decision-making. By constructing and interacting with dedicated modules based on a medium-sized language model (LLM) and a visual-language model (VLM), our framework exhibits robust performance and human-like exploratory behavior without requiring task-specific training. Experimental results demonstrate that FineCog-Nav outperforms various zero-shot fine-tuning methods, highlighting the potential and prospects of modular architecture design in solving multimodal embodied agent tasks.

[0026] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0027] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which:

[0028] Figure 1 : A schematic diagram of the UAV visual language navigation zero-fine-tuning system framework (FineCog-Nav) based on fine-grained cognitive function module integration proposed in this invention;

[0029] Figure 2 : Prompt words during the instruction parsing process;

[0030] Figure 3 : The output of the instruction parsing process in the embodiment;

[0031] Figure 4 Cue words for the attention process;

[0032] Figure 5 : The output of the attention process in the embodiment;

[0033] Figure 6 : Cue words for the perception process;

[0034] Figure 7 In this embodiment, the input image for the sensing process;

[0035] Figure 8 : The output of the sensing process in the embodiment;

[0036] Figure 9 : Prompt words for the sub-target extraction process;

[0037] Figure 10 : Input to the sub-target extraction process in the embodiment;

[0038] Figure 11 : Cue words for the imagination process;

[0039] Figure 12 : Prompt words for the sub-goal completion judgment process;

[0040] Figure 13 In this embodiment, the input for the sub-target completion judgment process is...

[0041] Figure 14 In this embodiment, the output of the sub-target completion judgment process;

[0042] Figure 15 Cue words in the decision-making process;

[0043] Figure 16 : Inputs to the decision-making process in the embodiment;

[0044] Figure 17 : The output of the decision-making process in the example;

[0045] Figure 18 Clues for single-step memorization;

[0046] Figure 19 In the embodiment, the input of the single-step memory process;

[0047] Figure 20 : The output of the single-step memory process in the embodiment;

[0048] Figure 21 : Prompt words for the compression process;

[0049] Figure 22 : Input to the compression process in the embodiment;

[0050] Figure 23 : The output of the compression process in the embodiment;

[0051] Figure 24 : Instruction memory sequence illustration;

[0052] Figure 25 Comparison of baseline methods with different base models on AerialVLN-S-Fine;

[0053] Figure 26 Comparison of AerialVLN-S Val-Seen with baseline methods under different base models. Detailed Implementation

[0054] The embodiments of the present invention are described in detail below. These embodiments are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0055] Vision-Language Navigation (VLN) aims to enable agents to navigate in complex environments based on natural language instructions. Its core challenge lies in achieving accurate cross-modal visual-text alignment. Early VLN research focused on feature-level multimodal alignment, employing sequence-to-sequence language encoders and attention mechanisms to establish visual-language associations. Besides feature encoding, data augmentation, pre-training strategies, map building, and reinforcement memory mechanisms have also driven the field's development. As VLN tasks on ground platforms (especially quadruped robots) have matured, research focus has shifted to more challenging aerial scenarios, namely UAV (Unmanned Aerial Vehicle) visual-language navigation. LANI pioneered UAV navigation by constructing a closed, simple environment from a top-down perspective. ANDH reconstructed instructions into dialogue form, proposing a Transformer-based method to predict waypoints and areas of human interest. However, such bird's-eye-view (BEV) methods differ significantly from the perspective of a real UAV. To address this issue, AerialVLN pioneered the construction of the first first-person-view (FPV) drone VLN benchmark dataset. Since then, the drone VLN field has rapidly developed in areas such as map design, modular task decomposition, and temporal reasoning, driving the evolution of VLN towards continuous open-world scenarios.

[0056] While VLN methods have made significant progress in accuracy and efficiency, the proliferation of model parameters and data scale has led to a sharp increase in training costs. Therefore, zero-shot fine-tuning VLN techniques, based on open-source pre-trained base models and requiring no additional pre-training, have attracted attention. CLIP-NaV and NavGPT are early representative works in this direction, utilizing large language models to parse instructions and visual language models to parse the environment. To better handle unordered or ambiguous instructions, NavGPT introduces historical context into the zero-shot fine-tuning process and constructs modules for perception, summarization, and decision-making. DiscussNav further utilizes the GPT-4 large model to build expert modules, improving decision stability through multi-round discussion mechanisms among experts. MapGPT and InstructNav integrate map inference to transform language instructions into executable trajectories. Furthermore, traditional VLN tasks often rely on discrete waypoint selection, which differs significantly from real continuous environments. Therefore, some research has extended zero-shot VLN to continuous navigation spaces, pushing it towards practical deployment. Compared to ground environments with structured priors, aerial scenes are far more complex, and zero-shot fine-tuning methods for UAV VLN (Vehicle Level Navigation) are still in their early exploratory stages. Representative works such as STMR, OpenUAV, and NavAgent demonstrate considerable technical potential, but existing zero-shot methods often impose stringent requirements and dependencies on the input data (e.g., NavAgent requires fine-grained landmark annotation, and STMR relies on real poses to build maps). Despite their promising prospects, current zero-shot fine-tuning methods still struggle to meet the challenges of UAV VLN tasks and require further exploration.

[0057] To address the aforementioned issues, this embodiment proposes a zero-fine-tuning method and system for UAV visual-language navigation (VLN) tasks based on fine-grained cognitive function module integration. The method specifically includes the following steps:

[0058] Step 1: Instruction parsing process:

[0059] Navigation instructions The pre-trained large model is input with the set instruction parsing prompts to perform instruction parsing, resulting in a list of several instruction sentences and their corresponding landmarks. The pre-trained large model in this step is instantiated from a mature commercial large language model (LLM).

[0060] In visual-language navigation (VLN) tasks in a continuous three-dimensional space, UAVs often need to complete multi-step navigation based on long and complex natural language commands. However, directly processing the entire complex command increases the information load, reduces the efficiency of perception and decision-making, and leads to difficulties in multimodal alignment. To address this, we draw on the human heuristic "divide and conquer" problem-solving mechanism to automatically parse complex navigation commands into an ordered sequence of command sentences and extract a corresponding list of visual landmarks for each command sentence. This provides a structured and step-by-step input for subsequent sub-target extraction and perception processes.

[0061] The entire parsing process can be formally represented as:

[0062] Navigation instructions Break it down into several semantically independent and sequentially ordered instruction sentences. Semantic analysis is performed on each instruction sentence to automatically identify and extract a list of visual landmarks explicitly mentioned within it. , Indicates a command sentence A list of corresponding landmarks.

[0063] The prompts designed for the instruction parsing process include, for example: Figure 2 As shown, this includes the cognitive role, overall task, precautions, input domain, output format constraints, and expected output. The overall task comprises two parts: command segmentation and landmark extraction. The input domain contains navigation commands, and the prompts require a standardized JSON output format for subsequent processing. Here, using a single command-navigation trajectory pair as an example, the input and output process of the command parsing steps is demonstrated in detail:

[0064] Fill the input field of the prompt with the navigation instructions "Turn left and fly over the yellow truck. Turn left and turn left at the next intersection after the other truck. Land in front of the truck's rear wheel. Stop on the sidewalk." and the final output will be as follows. Figure 3 As shown.

[0065] Step 1 works only once before each navigation begins (i.e. when a navigation command is received), and the subsequent process is described according to each time step.

[0066] Step 2: Judgment and Attention Process:

[0067] If the current instruction is the first instruction or an instruction switch occurs at the current time step, then at the current time step, the current instruction and the next instruction, along with their corresponding landmarks, are selected and input into the pre-trained large model, along with the set attention cues, to obtain the attention problem corresponding to the current instruction, and proceed to step 3. Otherwise, proceed directly to step 3. The pre-trained large model in this step is instantiated from a mature commercial Large Language Model (LLM).

[0068] Since each instruction requires multiple time steps to complete, for each time step, we first need to determine if the current time step is the first time step of the current instruction. If it is, we then simulate the "attention" function in human cognition to obtain the attention problem corresponding to the current instruction, as follows:

[0069] In a 3D open-world environment, UAVs often face a large amount of interference from irrelevant details during visual perception, which can affect their focus on and understanding of key targets. To address this, we have drawn inspiration from human attention mechanisms and designed and introduced an attention step to proactively guide subsequent perception steps to focus on key landmarks relevant to the task, thereby improving perception efficiency and the accuracy of navigation decisions.

[0070] In the attention step, the current and next instruction sentences are used as the attention window. This is achieved by inputting the current and next instruction sentences, as well as the landmarks corresponding to the current and next instruction sentences: and A series of landmark-related questions are automatically generated using a large model. These questions focus on the current state of landmarks and the spatial relationships and attributes (such as location and distance) between objects and observers (the agent itself) from the agent's perspective, paying attention to their potential impact on navigation tasks. This question-driven attention mechanism guides the perception process to actively search for information closely related to the task, rather than passively receiving environmental data. This transforms the perception process from passive collection to active exploration, enhancing the system's focus on key targets and its environmental understanding.

[0071] Cue words designed for the attention process, such as Figure 4 As shown, this includes the cognitive role, task, input domain, output constraints, and expected output. The input domain contains the current instruction and its landmark, as well as the next instruction and its landmark. The prompt requires a standardized JSON format output for subsequent processing. Here, we use a specific moment in a navigation task as an example to demonstrate the input and output process of the attention step in detail:

[0072] The input domain of the prompts is filled with the following: "Current instruction: Pass through the shops and go past the small lift. Current instruction landmarks: shops, small lift. Next instruction: Rotate and rise at the bike parking area. Next instruction landmark: bike parking." A pre-trained large model generates attention questions: for each landmark mentioned in the current and next instructions (shops, small lift, bike parking), it proactively raises questions that guide perceptual attention. For example, it focuses on their visibility, relative position, or distance. The final output is as follows: Figure 5 As shown.

[0073] Through the above process, the attention step can proactively guide perception to focus on environmental elements that are highly relevant to the current and upcoming tasks, thereby providing key reference information for subsequent perception, judgment, and decision-making.

[0074] Step 3: Perception Process

[0075] The attention problem corresponding to the current instruction and the image acquired at the current time step are combined with the set perception cues and input into a pre-trained large model to obtain the scene description at the current time step. The pre-trained large model in this step is instantiated from a mature commercial multimodal vision-language model (VLM).

[0076] Visual perception capabilities are crucial for the autonomous navigation of drones in complex three-dimensional environments. The perception process, as the core link between the environment and the agent's decision-making, is responsible for efficiently and accurately understanding and representing the current scene, providing a solid perceptual foundation for subsequent intelligent actions. Without a reliable perception process, the agent will be unable to effectively cope with dynamic changes in the environment and diverse navigation tasks.

[0077] cue words designed for the perception process, such as Figure 6 As shown, it includes cognitive roles, tasks, input domains, output constraints, precautions, and expected outputs; the input domain includes first-person RGB images. ,like Figure 7 As shown, and the attention problem corresponding to the current instruction. ,like Figure 5 The attention problem is derived from this; RGB images are understood and textualized using a multimodal visual-language model (VLM), with a focus on the attention problem proposed in the attention step. The system strategically focuses on visual anchor points closely related to instructions and navigation, such as shops, small elevators, and bicycle parking areas, extracting the most valuable navigational information. To ensure a comprehensive and organized description of the current visual content, a hierarchical structure of "Overall + Details" is adopted: First, a concise overall description quickly summarizes the core features of the current scene; then, the scene is divided into three areas according to spatial orientation: At the Center, On the Left, and On the Right, with detailed descriptions of the visual content in each area. This highlights navigation-related key points while ensuring the completeness and spatial hierarchy of scene information. All descriptions are output in structured JSON text format for easy use in subsequent steps. The final output is as follows: Figure 8 As shown.

[0078] Through the above design, the perception step can efficiently and structurally acquire scene information, proactively respond to navigation-related concerns, and provide strong perception support for the autonomous intelligent navigation of UAVs.

[0079] Furthermore, during the perception process, collision warnings are also provided based on the depth map at the current time step. Collision warning is a critical safety feature in UAV autonomous navigation systems. It is primarily responsible for promptly identifying potential risks during navigation tasks and predicting and warning of collisions to be taken, thereby effectively preventing accidents and improving the system's safety and robustness. This process is a standard technology in the field of UAV autonomous navigation and will not be elaborated upon here.

[0080] Step 4: Judgment and Sub-target Extraction Process:

[0081] If the current instruction is the first instruction or an instruction change occurs at the current time step, the current instruction and the scene description of the current time step obtained in step 3, combined with the sub-target extraction prompts, are input into the pre-trained large model to obtain the sub-target list, and then proceed to step 5; otherwise, proceed directly to step 5. The pre-trained large model in this step is instantiated from a mature commercial large language model (LLM).

[0082] As described in step 2, each instruction requires multiple time steps to complete. In the first time step of each instruction, in addition to the attention step, a sub-target extraction process is also performed, as follows:

[0083] In complex three-dimensional environments, a single natural language command for navigation often encompasses multiple steps or composite objectives. The core task of the sub-target extraction step is to automatically extract a series of clear and executable sub-target sequences from the command sentence, ensuring that the UAV can complete the overall task efficiently and in stages. Therefore, through the command parsing process, complex navigation commands are broken down into ordered command sentences, and then each command sentence is further parsed and a series of fine-grained, clearly ordered sub-targets are extracted. Through this hierarchical and progressive task decomposition method, the system can effectively refine complex navigation tasks step by step, enabling the UAV agent to always focus on the current target and proceed in an orderly manner, ultimately completing the overall navigation task efficiently.

[0084] The prompts designed for the sub-target extraction process include, for example: Figure 9 As shown, the process includes cognitive roles, overall tasks, input domains, detailed tasks, precautions, output constraints, and expected outputs. The input domain includes the current instruction and a scene description at the current time step. Means-end analysis is performed on the current instruction using a Large Language Model (LLM) to identify all executable sub-goals or intermediate goals. Combined with scene-aware information, unachievable or irrelevant content is eliminated to ensure each sub-goal is executable in the current environment, avoiding the generation of false or unattainable goals. All sub-goals are arranged in their actual physical execution order, not their original syntactic order, ensuring the rationality and feasibility of task decomposition. All sub-goals are output in a standard JSON queue format for easy automatic reading and retrieval by downstream steps. Here, we take a specific moment in the navigation task as an example (assuming a new instruction has just been retrieved, i.e., the current instruction is the first instruction) to demonstrate the input and output of the sub-goal extraction process in detail. The input content is as follows: Figure 10 As shown, the output is a list of extracted sub-targets: ["Passing through the store", "Passing through the small elevator"].

[0085] Step 5: Judgment and Imagination Process:

[0086] If the current sub-target is the first sub-target or a sub-target switch occurs at the current time step, then select the current sub-target and the next sub-target, as well as the landmark corresponding to the current instruction sentence, and input them into the pre-trained large model in combination with the set imagination prompt words to obtain the possible visual state after the current sub-target is completed, and proceed to step 6; otherwise, proceed directly to step 6; the pre-trained large model in this step is instantiated by a mature commercial large language model (LLM).

[0087] Since the instruction contains multiple sub-goals, and each sub-goal requires multiple time steps to complete, for each time step, it is also necessary to determine whether the current time step is the first time step of the current sub-goal. If it is the first time step of the current sub-goal, then the "imagination" function in human cognition is simulated to obtain the possible visual state after the current sub-goal is completed, as follows:

[0088] In complex navigation tasks, although instructions are broken down into sub-objectives, the UAV agent still faces the challenge of determining "when can we determine that the current sub-objective has been completed" when executing each sub-objective. Inspired by the human ability to "mentally simulate," we designed an "imagination" step: When facing a goal, humans often actively simulate the environmental state after achieving the goal in their minds, using this as a basis for action and judgment. For example, "I want to wash the cup," so the completed state is "the cup is clean." Analogous to UAV navigation tasks, the "imagination" step can help the UAV proactively predict the possible visual state after achieving each sub-objective, including key landmarks and their spatial relationships from a first-person perspective.

[0089] prompts for the imagination process design, such as Figure 11 As shown, the process includes the cognitive role, task, input domain, output constraints, precautions, and expected output. The input domain contains the current sub-target and the next sub-target, as well as the landmark corresponding to the current command. The Large Language Model (LLM) actively "imagines" the key visual states of the environment visible in front of the drone after completing the current sub-target. The output is a concise, structured description of the expected visual state, focusing on the spatial location (e.g., left / right / center) and relative distance of landmarks, facilitating alignment with the perception results. This output provides an important reference for determining whether the sub-target has been achieved, effectively assisting in the design of subsequent achievement criteria and action decisions. All imagined results are output in standard JSON format for convenient processing in subsequent steps. Here, we take a sub-target in a navigation task as an example to demonstrate the input and output process of the imagined state in detail:

[0090] Fill the input field of the prompt with "Current subtarget: Takeoff. Next subtarget: Straight toward the neon lights. Landmark corresponding to the current command: the neon lights". The final output is ""state": "After I complete 'takeoff', I may see the neon lights in the center of my field of vision, about 100 meters away."

[0091] Step 6: Sub-goal completion judgment, decision-making, and multi-level memorization:

[0092] The current sub-target, the next sub-target, the possible visual state after the current sub-target is completed, the scene description at the current time step, and the original sub-target memory sequence of the current sub-target (the initial original sub-target memory sequence is empty) are combined with the set sub-target completion judgment prompt words and input into the pre-trained large model to determine whether the current sub-target is completed. If the current sub-target is the last sub-target in the sub-target list corresponding to the current instruction sentence, then the next sub-target is the next instruction sentence. The pre-trained large model used in the sub-target completion judgment process is instantiated from a mature commercial large language model (LLM).

[0093] In step 4, we divide the command statement into a series of fine-grained sub-targets. However, this hierarchical solution approach also brings new core challenges: how to accurately and automatically determine whether the current sub-target has been completed? If the determination is made too early, navigation may deviate; if the determination is delayed, redundant actions may occur, or even the optimal switching opportunity may be missed. Therefore, accurate determination of the completion status of sub-targets becomes a critical node in the navigation system process.

[0094] In the sub-goal completion judgment phase, using multi-source information as input, the system judges whether the current sub-goal has been achieved from multiple perspectives, including perception, memory, analogy, and reasoning, and provides a clear basis for whether the system should switch to the next sub-goal. The sub-goal completion judgment phase not only outputs a simple Yes / No conclusion but also explicitly provides the reasoning behind it to enhance the interpretability and reliability of the system.

[0095] Prompt words designed for sub-goal completion judgment, such as Figure 12 As shown, it includes cognitive roles, tasks, input domains, precautions, additional requirements, output constraints, and expected outputs.

[0096] During the sub-goal completion judgment phase, the input includes:

[0097] Current Subgoal Used to clarify what the objective is, and to ensure the completion of sub-objectives and the determination of tasks. Figure 1 To;

[0098] Next Subgoal Used to provide a switching context, enabling the subtarget to perceive whether it has reached the subtarget boundary after completing the judgment process, thus avoiding ambiguity or omission in the switching timing;

[0099] Scene description at the current time step (Current Observation) It is used to dynamically determine whether the current environmental state matches the sub-target situation and the expected outcome after the sub-target is completed, based on the real-time visual perception of the UAV, and serves as a realistic basis for judgment.

[0100] Possible visual states (Reference State) after the current sub-objective is completed. The output of the visualization process in step 5 is called to provide a mental simulation description of the "typical" visual state after the sub-goal is completed, as a reference for auxiliary comparison and reasoning.

[0101] The original subgoal memory sequence of the current subgoal. Based on all actions executed and observed scenarios under the current sub-goal, verify whether all key behaviors have been performed. The sub-goal memory sequence is initially empty.

[0102] Based on the above analysis, the reasoning process for determining the completion of a sub-goal can be formally represented as follows:

[0103]

[0104] Based on the above inputs, the large model outputs a clear "completed / incomplete" judgment result. It is worth emphasizing that although the basic output of the sub-target completion judgment process is only "True / False," to improve system transparency and traceability, this embodiment mandates that the large model agent explicitly output a "reason" field, that is, based on all input information, provide a detailed explanation of the reasons for making the "completed" or "incomplete" judgment. This not only facilitates system debugging and performance optimization but also helps with subsequent decision tracing and problem analysis. Here, we take a sub-target in the navigation task as an example to illustrate the input and output process of the sub-target completion judgment stage in detail. The input information is as follows: Figure 13 As shown, the output result is as follows Figure 14 As shown.

[0105] After the sub-goals are achieved and the judgment is completed, decisions are made based on the judgment results, and multi-level memorization is performed.

[0106] If the current sub-objective is not completed, a decision-making process is initiated: the instruction memory sequence (initially empty), the current time step scene description, collision warning, preset effective action list, and the current sub-objective are input into the pre-trained large model along with the set decision prompts to obtain the optimal action selection; the current time step scene description, the optimal action selection, and the landmark corresponding to the current instruction sentence are input into the pre-trained large model along with the set memory prompts to obtain single-step memory; the single-step memory is inserted into the original sub-objective memory sequence of the current sub-objective (initially empty), the UAV executes the optimal action selection, and the current time step ends; return to step 3 to proceed with the perception process of the next time step. The pre-trained large model used in the decision-making process and the single-step memory process is instantiated from a mature commercial Large Language Model (LLM).

[0107] When the current sub-goal is not yet achieved, the primary task is to decide "what action the agent should take next." This decision must not only accurately select the optimal action from the set of valid actions based on the current sub-goal and environmental state, but also possess high interpretability for subsequent verification and optimization. Unlike traditional black-box decision-making, the decision-making process in this embodiment not only outputs the specific action selection, but also the "thinking process" behind the decision and the probability distribution of each optional action, achieving transparency and traceability of the decision-making process.

[0108] Cue words designed for the decision-making process, such as Figure 15 As shown, it includes cognitive roles, tasks, input domains, precautions, additional requirements, output constraints, and expected outputs.

[0109] During the decision-making phase, the inputs include:

[0110] Current sub-target The sub-goals to be achieved at this stage indicate the intention of the navigation phase.

[0111] Scene description at the current time step The first-person perspective of the drone's perception of the current environment;

[0112] Collision warning information Warning: There may be a collision risk when certain actions are performed;

[0113] Instruction memory sequence Within the scope of the current instruction statement, the initial instruction memory sequence is empty for the history of executed actions and observed content. The instruction memory sequence consists of the compressed sub-target memory sequence of all previously completed sub-targets and the original sub-target memory sequence of the current sub-target.

[0114] Preset list of valid actions: A set of actions that the system can choose to execute.

[0115] Based on the above inputs, the output of the large model is designed to include three parts: the normalized candidate action probability distribution (probabilities), which represents the UAV decision-making system's preference for each action in a quantitative way; the final decision-selected action (selected_action), which is the action number with the highest probability; and the decision-making thought process (thought), which is the reason and reasoning link for this action selection, enhancing interpretability.

[0116] It can be seen that the decision-making stage does not directly output an action result, but rather comprehensively displays the degree of preference for all possible actions in the form of a quantified probability distribution, thus making the decision-making process more transparent. The final selected action is the option with the highest probability among all possible actions. Moreover, for each decision, the large model agent explicitly outputs the "thought process." This explanation needs to integrate information from multiple sources, such as the current sub-goal, environmental observations, instruction memory, and safety prompts, and elaborate on the reasoning logic of action selection in detail. For example: whether the selected action directly helps to achieve the current sub-goal, and the consistency analysis of goal achievement; whether there are obstacles in the current environment, and whether the action effectively avoids collision risks and ensures navigation safety; combining instruction-level memory, analyzing previously executed actions and observation results, such as actively considering early task termination (TaskFinish) when no landmarks are detected for several consecutive steps to avoid ineffective consumption; when there are multiple possible actions, explaining why the current action is preferred over other options, reflecting the rationality and scientific nature of the decision. This explainable and traceable decision-making mechanism is analogous to the human thought process before action: it not only relies on immediate sensory input but also reviews past experiences, assesses risks, and makes multi-dimensional trade-offs based on the target task. Through the output of this "thought process," the system can fully present the basis, judgment path, and selection logic of each decision, significantly improving the transparency, debuggability, and engineering verifiability of UAV intelligent decision-making. Here, we take the decision at a specific moment in a navigation task, 𝑡, as an example to demonstrate the input and output of the decision-making process in detail. The input information is as follows: Figure 16 As shown, the output result is as follows Figure 17 As shown.

[0117] In specific UAV navigation tasks, the construction of a memory system plays an irreplaceable and crucial role in the efficient decision-making and mission completion of the UAV. In daily cognitive activities, humans frequently access multi-level memory resources, combining past experiences with current perceptions to guide their actions. This ability is essential for avoiding repetitive exploration, improving path efficiency, and enabling cross-temporal reasoning and summarization. Similarly, in UAV vision-language navigation (VLN) tasks, a robust memory mechanism helps the UAV understand command context, track mission progress, avoid redundant paths, and enhance environmental adaptability.

[0118] This invention designs a three-level memory mechanism, namely step memory, subgoal memory, and instruction memory, to achieve multi-level and dynamically evolving task memory.

[0119] In the decision-making process, after obtaining the optimal action selection, single-step memorization is performed. This involves inputting the current time step scene description, the optimal action selection, the landmark corresponding to the current command, and the set prompt words into a pre-trained large model to obtain single-step memory. This single-step memory is then inserted into the original sub-target memory sequence of the current sub-target, which is initially empty. The pre-trained large model used in the single-step memorization process is instantiated from a mature commercial Large Language Model (LLM).

[0120] Tips designed for single-step memorization, such as Figure 18 As shown, this includes cognitive roles, tasks, input domains, precautions, output constraints, and expected outputs. Single-step memory records the observations (current time-step scene description) and actions (optimal action selection) at each time step, in the format "I see…, I do…", accurately capturing the perception-action pairs at each moment, which forms the foundation for all higher-level memories. During the execution of a sub-goal, the system continuously accumulates the corresponding single-step memories, forming the original sub-goal memory sequence for the current sub-goal. Taking a single-step memory at a certain moment as an example, the input information is as follows: Figure 19 As shown, the output result is as follows Figure 20 As shown.

[0121] If the current sub-target has been completed, the original sub-target memory sequence, combined with the set compressed prompt words, is input into the pre-trained large model to obtain the compressed sub-target memory sequence, and the original sub-target memory sequence is cleared. It is then determined whether the current sub-target is the last sub-target in the sub-target list. If it is, it is further determined whether the current command is the last command in the current navigation command. If it is, the task ends; otherwise, it returns to step 2 to switch command sentences. If it is not the last sub-target, it returns to step 5 to switch sub-targets according to the sub-target list. The pre-trained large model used in the process of summarizing and compressing the original sub-target memory sequence is instantiated from a mature commercial Large Language Model (LLM).

[0122] The prompts designed for the summarization and compression process of the original sub-target memory sequence are as follows: Figure 21 As shown, the original sub-target memory sequence composed of single-step memories is summarized and compressed using a Large Language Model (LLM) to generate structured compressed sub-target memories, simulating the solidification and abstraction process of human memory. Taking the compression process at a certain moment as an example, the input information is as follows: Figure 22 As shown, the output result is as follows Figure 23 As shown.

[0123] Finally, the compressed memory of all completed sub-targets, along with the original sub-target memory sequence of the current sub-target, are dynamically combined to form the current instruction memory sequence, such as... Figure 24As shown, this provides rich historical context for subsequent instruction-level reasoning and global decision-making.

[0124] This multi-level memory design provides the system with full-process, hierarchical historical information for decision-making, ensuring the continuity of navigation timing and the adaptive reasoning ability across task steps, effectively improving the agent's environmental understanding and adaptability.

[0125] The following is a specific experimental verification:

[0126] In our experiments, we used the well-known Qwen2.5-72B-Instruct, GPT 4o-mini, and internLM as large-scale language models (LLMs), and Qwen2.5-VL-32B and GPT 4o-mini as visual language models (VLMs). All experiments were conducted using the official APIs of these large-scale foundational models.

[0127] Given the limited number of zero-shot VLN methods tailored for UAVs, we selected four representative state-of-the-art (SOTA) baselines from the ground and UAV navigation domains: NavGPT, DiscussNav, InstructNav, and Aerial-STMR. To maintain consistency, we adapted all baselines to the UAV environment: replacing their original inputs with RGB and depth images obtained from the AerialVLN simulator, and standardizing the output format to generate executable UAV actions such as ascend, forward, and left turn. This adaptation allows methods originally designed for discrete VLN tasks to be migrated to continuous aerial VLN environments. To fairly evaluate the inference capabilities of the cueing architecture itself, we further removed the built-in mapping components from each baseline.

[0128] The evaluation adopted the standardized metrics commonly used in the VLN dataset, while also considering the characteristics of UAV trajectories in the AerialVLN dataset. In the official AerialVLN implementation, all metrics are based on the plane. The distance is calculated using two-dimensional Euclidean distance in a coordinate system. Considering that UAV navigation occurs in three-dimensional space, we further utilize the complete spatial... Coordinates are used to better reflect the accuracy of three-dimensional positioning. These include:

[0129] Navigation Error (NE): The average distance between the final position of the UAV and the target position;

[0130] Success Rate and ): The percentage of the trajectory in which the final position of the UAV is within 20 meters of the target position in two-dimensional and three-dimensional space;

[0131] Oracle Success Rate (OSR): The percentage of times during navigation that the drone is within 20 meters of the target location at least once.

[0132] In addition, we listed the average path length (PL) and the number of navigation steps to reflect the exploration tendency of each method; and used the number of lexical units used in each step as an indicator of model efficiency.

[0133] To differentiate the impact of the framework itself from the capabilities of the original large-scale foundational model (LFM), we evaluated various LLM / VLM combinations on the AerialVLN-S-Fine dataset. To balance performance and cost, we focused on medium-sized LFMs, testing four configurations: pairing Qwen2.5-72B-Instruct as the LLM with Qwen2.5-VL-32B and GPT 4o-mini as VLMs; and pairing Qwen2.5-VL-32B as the VLM with GPT 4o-mini and InternLM-20B as LLMs. Since STMR only uses VLM models, it is not included in comparisons between different LLMs. These foundational model combinations allow for a comprehensive comparison between baseline methods and our proposed method, resulting in more accurate and reliable evaluation results.

[0134] Figure 25 A comparison of the baseline method with different base models on AerialVLN-S-Fine is presented. Figure 26 The comparison of the baseline method with the AerialVLN-S Val-Seen under different base models is given. ★ indicates LLMs and † indicates VLMs. The best and second-best results are highlighted with bold and underline. It can be seen from the figure that: (1) FineCog-Nav consistently outperforms other zero-shot methods in terms of success rate (SR), ideal success rate (OSR), and navigation error (NE) under all configurations, proving its effectiveness in target-oriented navigation; (2) Although some baselines have limited exploration problems on smaller base models (such as InternLM-20B and GPT 4o-mini (about 8B)), our method can still maintain stable exploration behavior under different models and settings; (3) FineCog-Nav maintains strong performance while still maintaining efficient use of lexical units in each step, reflecting the effective use of language resources.

[0135] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention without departing from the principles and spirit of the present invention.

Claims

1. A zero-fine-tuning method for UAV visual language navigation based on fine-grained cognitive function module integration, characterized in that: Includes the following steps: Step 1: Input the navigation commands and the set command parsing prompts into the pre-trained large model to realize command parsing and obtain a list of several command sentences and the corresponding landmarks for each command sentence; Step 2: If the current instruction is the first instruction or the instruction has changed at the current time step, select the current instruction and the next instruction, as well as the landmark corresponding to the current instruction and the landmark corresponding to the next instruction at the current time step. Combine the set attention cue words and input them into the pre-trained large model to obtain the attention problem corresponding to the current instruction and proceed to step 3. Otherwise, proceed directly to step 3. Step 3: Input the attention problem corresponding to the current instruction and the image acquired at the current time step, along with the set perception prompts, into the pre-trained large model to obtain the scene description at the current time step; and perform collision warning based on the depth map at the current time step. Step 4: If the current instruction is the first instruction or the instruction changes at the current time step, input the current instruction and the scene description of the current time step obtained in Step 3, along with the sub-target extraction prompts, into the pre-trained large model to obtain the sub-target list, and proceed to Step 5; otherwise, proceed directly to Step 5. Step 5: If the current sub-target is the first sub-target or a sub-target switch occurs at the current time step, select the current sub-target and the next sub-target, as well as the landmark corresponding to the current command sentence, and input them into the pre-trained large model in combination with the set imagination prompt words to obtain the possible visual state after the current sub-target is completed, and then proceed to step 6. Otherwise, proceed directly to step 6; Step 6: Input the current sub-target and the next sub-target, the possible visual state after the current sub-target is completed, the scene description at the current time step, and the original sub-target memory sequence of the current sub-target, along with the set sub-target completion judgment prompt words, into the pre-trained large model to determine whether the current sub-target is completed; If the current sub-target is not completed, a decision-making process is performed: the instruction memory sequence, the current time step scene description, collision warning, the preset list of valid actions, and the current sub-target are input into the pre-trained large model along with the set decision prompts to obtain the optimal action selection; the current time step scene description, the optimal action selection, and the landmark corresponding to the current instruction sentence are input into the pre-trained large model along with the set memory prompts to obtain the single-step memory; the single-step memory is inserted into the original sub-target memory sequence of the current sub-target, and the UAV executes the optimal action selection, the current time step ends, and the process returns to step 3 to proceed with the perception process of the next time step; If the current sub-target has been completed, the original sub-target memory sequence, combined with the set compressed prompt words, is input into the pre-trained large model to obtain the compressed sub-target memory sequence, and the original sub-target memory sequence is cleared. Then, it is determined whether the current sub-target is the last sub-target in the sub-target list. If it is the last sub-target, it is further determined whether the current instruction sentence is the last instruction sentence of the current navigation instruction. If it is the last instruction sentence, the task ends. If it is not the last instruction sentence, it returns to step 2 to switch instruction sentences. If it is not the last sub-target, it returns to step 5 to switch sub-targets according to the sub-target list.

2. The method for zero-fine-tuning UAV visual language navigation based on fine-grained cognitive function module integration according to claim 1, characterized in that: In step 6, when determining whether the current sub-target has been completed, if the current sub-target is the last sub-target in the sub-target list corresponding to the current instruction sentence, then the next sub-target is the next instruction sentence.

3. The method for zero-fine-tuning UAV visual language navigation based on fine-grained cognitive function module integration according to claim 1, characterized in that: In step 1, the instruction parsing prompts include cognitive role, overall task, precautions, input domain, output format constraints, and expected output; The overall task includes two tasks: instruction segmentation and landmark extraction. Navigation instructions are filled in the input field, and standardized output format is required in the prompt words.

4. The method for zero-fine-tuning UAV visual language navigation based on fine-grained cognitive function module integration according to claim 1, characterized in that: In step 2, the attention prompts include cognitive roles, tasks, input domains, output constraints, and expected outputs; the input domain is filled with the current instruction sentence and its landmark, as well as the next instruction sentence and its landmark, and the prompts require a standardized output format.

5. The method for zero-fine-tuning UAV visual language navigation based on fine-grained cognitive function module integration according to claim 1, characterized in that: In step 3, the perceptual prompts include cognitive roles, tasks, input domains, output constraints, precautions, and expected outputs; The input domain includes a first-person RGB image and the attention problem corresponding to the current instruction. The output adopts a hierarchical structured output method of global information + detailed information. First, a concise summary description is used to quickly summarize the core features of the current scene. The scene was then divided into three areas—central, left, and right—based on spatial orientation, and the visual content of each area was described in detail. All descriptions were output in a standardized format.

6. The method for zero-fine-tuning UAV visual language navigation based on fine-grained cognitive function module integration according to claim 1, characterized in that: In step 4, the sub-target extraction prompts include cognitive role, overall task, input domain, detailed task, precautions, output constraints, and expected output; The input domain includes the current instruction sentence and the scene description of the current time step; the current instruction sentence is analyzed using a large language model to identify all executable sub-goals or intermediate goals contained therein, and combined with scene-aware information, unachievable or irrelevant content is eliminated to ensure that each sub-goal is executable in the current environment and to avoid generating false or unattainable goals; all sub-goals are output in a standardized format.

7. The method for zero-fine-tuning UAV visual language navigation based on fine-grained cognitive function module integration according to claim 1, characterized in that: In step 5, the imaginative prompts include cognitive roles, tasks, input domains, output constraints, precautions, and expected outputs; The input field contains the current subtarget and the next subtarget, as well as the landmark corresponding to the current command. The large language model generates the key visual states of the environment visible in front of the drone after the current subtarget is completed. The output visual expectation description focuses on the spatial location and relative distance of the landmark. The visual expectation description is output in a standardized format.

8. The method for zero-fine-tuning UAV visual language navigation based on fine-grained cognitive function module integration according to claim 1, characterized in that: In step 6, the sub-goal completion judgment prompts include cognitive role, task, input domain, precautions, additional requirements, output constraints, and expected output; the output includes the judgment result of whether the sub-goal has been completed or not, and the reason for making the judgment of completion or non-completion.

9. The method for zero-fine-tuning UAV visual language navigation based on fine-grained cognitive function module integration according to claim 1, characterized in that: In step 6, the decision prompts include cognitive role, task, input domain, precautions, additional requirements, output constraints, and expected output; The output consists of three parts: the normalized probability distribution of candidate actions, which represents the UAV decision-making system's preference for each action in a quantitative way; the final action selected, which is the action number with the highest probability; and the decision-making process, which is the reason and reasoning chain for this action selection.

10. The method for zero-fine-tuning UAV visual language navigation based on fine-grained cognitive function module integration according to claim 1, characterized in that: In step 6, the memory prompts include cognitive roles, tasks, input domains, precautions, output constraints, and expected outputs; observations and actions at each time step are recorded through single-step memory; during the execution of a sub-goal, the system continuously accumulates the corresponding single-step memories to form the original sub-goal memory sequence of the current sub-goal; the original sub-goal memory sequence composed of single-step memories is summarized and compressed using a large language model to generate structured compressed sub-goal memories; the compressed memories of all completed sub-goals, along with the original sub-goal memory sequence of the current sub-goal, are dynamically combined to form the current instruction memory sequence.

Images