Unmanned aerial vehicle visual language navigation method and system based on asynchronous collaborative architecture and target constraint decision

Abstract

The application discloses a UAV visual language navigation method and system based on an asynchronous collaborative architecture and target constraint decision, and the method comprises the following steps: a large language model is used to analyze natural language instructions into a structured subtask set; an asynchronous architecture is constructed, in which a main control thread is separated from a decision reasoning thread, and the reasoning thread extracts visual observations based on the latest frame; a multi-source space-time context tensor is constructed by fusing vision, semantics and historical actions; a visual language large model is used to generate a navigation strategy; for large-scale maneuvering, a macro-action output open-loop control sequence is defined; a termination judgment model based on geometric features is introduced, and the inequality constraint of the target center deviation degree and the field of view ratio is calculated to determine the subtask completion state. The application solves the control lag problem of the traditional serial navigation architecture under the large model reasoning delay, and solves the misjudgment problem of the probability model in a complex scene, and significantly improves the navigation real-time performance and robustness of the UAV in a real environment.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and UAV control, and in particular to a visual language navigation method and system for UAVs based on large model task decomposition and asynchronous control architecture for dynamic unstructured environments. Background Technology

[0002] Vision-Language Navigation (VLN) aims to enable intelligent agents to understand natural language instructions and navigate in unknown environments. Traditional methods rely on high-precision maps and real-time localization and mapping techniques, which struggle to respond to semantic instructions. Existing technologies include those that decompose tasks into large models, those that introduce target reference guidance mechanisms, and those that construct target hierarchy trees, but all have significant limitations.

[0003] Existing technologies generally employ a serial perception-decision-execution architecture, which suffers from a severe mismatch between the second-level inference latency of large models and the millisecond-level (20-100Hz) control requirements of UAVs. During high-speed flight, a 1-second delay can lead to a 5-meter positional error, increasing the risk of collision. During complex maneuvers such as a 180-degree turn, the camera experiences severe motion blur, and existing systems mostly rely on single-frame image processing, resulting in a sharp decline in feature extraction quality and causing navigation interruptions.

[0004] The task determination mechanism also has problems. Existing technologies mostly adopt soft determination based on statistical learning (such as function calculation of switching probabilities), which lacks physical hard constraints and is easily affected by the illusion of large models or environmental interference, leading to incorrect task completion. In terms of multimodal alignment, existing methods are limited to static scene understanding and lack temporal modeling of the dynamic changes in the appearance, scale and position of targets in high-speed motion, especially performing poorly in dense target environments.

[0005] Therefore, there is an urgent need for a navigation system that can balance the requirements of deep inference in large models and real-time control, and is robust under complex maneuvers. This invention proposes an innovative method that integrates asynchronous inference architecture, motion perception compensation mechanism, and physical constraint verification, effectively solving the above problems. Summary of the Invention

[0006] To address the aforementioned technical problems, the present invention aims to provide a visual language navigation method and system for unmanned aerial vehicles (UAVs) with asynchronous processing capabilities, high real-time performance, and high task completion rate.

[0007] The technical solution to achieve the purpose of this invention is: a UAV visual language navigation method based on asynchronous cooperative architecture and target constraint decision-making, comprising the following steps:

[0008] Step 1: Perform global task space mapping: Obtain natural language navigation instructions Using a large language model as a global planner, a mapping function is constructed. Natural language navigation instructions Mapped to a set of time-dependent discrete subtasks , where i is the subtask number;

[0009] Step 2: Perform asynchronous timing control: Construct an asynchronous parallel architecture that includes a main control thread and a decision inference thread; the main control thread maintains the image buffer in real time. The decision-making and reasoning thread at time Extract the image frame with the highest timestamp from the buffer. ,satisfy For the most recent moment, the timestamp in the discard buffer is lower than [previous time]. The old frame;

[0010] Step 3: Construct multi-source context tensors: convert image frames Encoding as visual cues Based on the current subtask description Next sub-task preview and historical action sequences Construct a multi-source contextual cue tensor ;

[0011] Step 4: Perform end-to-end decision reasoning and convert the multi-source context cue tensor into a single tensor. Input a large visual language model and output the navigation action command for the current moment. ;

[0012] Step 5: Generate macro action sequences and determine navigation action commands. Does it belong to the pre-defined complex maneuvering space? If so, then expand the function using macro actions. Generate includes Open-loop control sequence of basic action elements The control sequence Stored in a high-priority action queue for execution by the main control thread;

[0013] Step 6: Perform target constraint termination judgment and construct an anti-illusion judgment model based on spatial geometric features; calculate the center deviation of the target object in the current field of view. Pixel area ratio The current subtask is determined if and only if a preset set of geometric inequalities is satisfied. Completed, and the task index was updated.

[0014] Furthermore, the mapping function in step 1 Specifically, it is expressed as follows:

[0015] ;

[0016] in, System-level prompts that include thought chain guidance and JSON format constraints;

[0017] Generated subtasks For a structured quad: ,in Number the task. For action semantics, For the characteristics of the target object, This is the condition for terminating the task.

[0018] Furthermore, in step 3, the multi-source context cue tensor The construction formula is:

[0019] ;

[0020] in, This indicates a modal splicing operation. This indicates text embedding representation. A text-based serialization representation of historical actions;

[0021] The historical action sequence Includes the past Execution actions at each time step It is used to suppress oscillations.

[0022] Furthermore, the macro-action expansion function in step 5 Defined as:

[0023] When navigation action command is detected When the command is for large rotation, it is decomposed into Discrete micro-rotational motion :

[0024] ;

[0025] in , Rotate by the target angle;

[0026] Execute control sequence in the main control thread During the process, the decision-making and reasoning thread is temporarily suspended, and the system is in an open-loop control state.

[0027] Furthermore, the target constraint termination determination in step 6 specifically includes:

[0028] Define the bounding box of the target object as The image width is The height is , , , , ;

[0029] Central deviation formula:

[0030] ;

[0031] Pixel area ratio formula:

[0032] ;

[0033] The decision-making process is completed if and only if the following logic is satisfied:

[0034] ;

[0035] in, Features of the output results of the anti-hallucination detection model The center deviation threshold, The minimum area percentage threshold, Features of the target object.

[0036] This application also provides a UAV visual language navigation system based on asynchronous cooperative architecture and target constraint decision-making. The system is used to implement the above-mentioned method, including:

[0037] The global planning module is used to run the large language model client and is responsible for obtaining natural language navigation instructions. Using a large language model as a global planner, a mapping function is constructed. Natural language navigation instructions Mapped to a set of time-dependent discrete subtasks , where i is the subtask number;

[0038] The visual language navigation controller, as the core scheduling unit, integrates an image buffer and a macro action queue protected by a mutex lock, and runs independent communication main threads and AI inference threads.

[0039] The visual language navigation controller is used to construct an asynchronous parallel architecture that includes a main control thread and a decision reasoning thread; the main control thread maintains the image buffer in real time. The decision-making and reasoning thread at time Extract the image frame with the highest timestamp from the buffer. ,satisfy For the most recent moment, the timestamp in the discard buffer is lower than [previous time]. The old frame;

[0040] The visual language navigation controller is also used to determine navigation action commands. Does it belong to the pre-defined complex maneuvering space? If so, then expand the function using macro actions. Generate includes Open-loop control sequence of basic action elements The control sequence Stored in a high-priority action queue for execution by the main control thread;

[0041] A multimodal decision module is used to run a large visual language model, and the multimodal decision module is used to process image frames. Encoding as visual cues Based on the current subtask description Next sub-task preview and historical action sequences Construct a multi-source contextual cue tensor ;

[0042] The multimodal decision module is also used to process the multi-source context cue tensor. Input a large visual language model and output the navigation action command for the current moment. ;

[0043] The anti-illusion verification module is used to calculate the center deviation of the target object in the current field of view. Pixel area ratio The current subtask is determined if and only if a preset set of geometric inequalities is satisfied. Complete the task and update the task index, then verify the physical feasibility and target constraints of the judgment results.

[0044] This application also provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described UAV visual language navigation method based on asynchronous cooperative architecture and target constraint decision-making.

[0045] Beneficial effects:

[0046] 1. From the perspective of model performance, the core of this invention is the introduction of a deterministic anti-illusion judgment mechanism and macro-action generation strategy based on spatial geometric features. By calculating the joint constraint of target center deviation and field of view proportion, it replaces the traditional uninterpretable probability switching model. The navigation method of this invention solves the problems of illusions easily generated by generative large models in long-range navigation and decision interruption caused by loss of visual features during large-scale maneuvers. Compared with other visual language navigation technologies, it has greatly improved robustness and navigation success rate in complex dynamic environments.

[0047] 2. From a technical perspective, existing visual language navigation methods generally employ a serial processing mode of perception-decision-execution, which is difficult to adapt to the contradiction between large model inference latency and high-speed flight of UAVs. This invention innovatively introduces an asynchronous cooperative control architecture and a latest frame-first timing sampling mechanism, achieving decoupling between high-computing-power cognitive threads and high-frequency control threads, effectively solving the control oscillation problem caused by timing mismatch, and enhancing the system's real-time response capability during high-speed operations.

[0048] 3. From an application perspective, the global task mapping module based on the JSON standard and the open-loop control module based on macro-action queues adopted in this invention possess high modularity and versatility. This design allows the invention to be easily embedded into existing UAV flight control systems, and also realizes the navigation process through explicit thought chains and structured command outputs, thereby significantly reducing the debugging difficulty and technical integration cost of embodied intelligence algorithms in engineering deployment. Attached Figure Description

[0049] Figure 1 This is a diagram illustrating the overall architecture of the UAV visual language navigation system according to an embodiment of the present invention.

[0050] Figure 2 This is a flowchart illustrating the task planning and multimodal decision-making process according to an embodiment of the present invention. Detailed Implementation

[0051] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0052] Example 1:

[0053] like Figure 1 , Figure 2 As shown in the figure, an embodiment of the present invention provides a UAV visual language navigation method based on asynchronous cooperative architecture and geometric constraint decision-making, which includes the following steps:

[0054] Step 1: Perform global task space mapping and obtain natural language navigation instructions. Using a large language model as a global planner, a mapping function is constructed. ,Will Mapped to a set of time-dependent discrete subtasks ;

[0055] In this embodiment, the complexity of UAV visual language navigation tasks lies in the fact that natural language commands are typically unstructured and contain long-term temporal logic, and directly using them as input can lead to attentional divergence during navigation. To address this challenge, the global planning module utilizes a pre-trained large language model (such as GPT5 or Tongyi Qianwen) to parse high-entropy natural language into low-entropy structured data using navigation commands as input and guided by system-level prompts.

[0056] In this embodiment, step 1 first constructs system-level prompts that include thought chain guidance and JSON format constraints. This prompt word forces the model to output inference analysis fields before generating the final subtask, thus improving parsing accuracy. Then, the natural language instructions are... Input a large language model, through a mapping function Generate a set of subtasks. Each subtask generated... Defined as a structured quad: .in, Represents the task number. Represents action semantics (such as "flying along the route"), Represents the characteristics of the target object (such as a red truck). This represents the mission termination condition (e.g., "the target is in the center of the field of view"). This structured representation clarifies the execution boundaries of each stage, avoiding the accumulation of errors in long-range navigation.

[0057] Step 2 involves asynchronous timing control, constructing an asynchronous parallel architecture that includes a main control thread and a decision inference thread; the main control thread maintains the image buffer in real time. The decision-making and reasoning thread at time Extract the image frame with the highest timestamp from the buffer. ;

[0058] In this embodiment, existing navigation methods generally employ a serial architecture, while the inference time of high-performance visual language models is typically much longer than the cycle of the UAV's underlying control, resulting in a severe "perception-execution" timing mismatch. To address this issue, this invention adopts an asynchronous architecture.

[0059] In this embodiment, step 2 specifically includes:

[0060] Step 2.1: The main control thread operates based on a message notification architecture, handling the transmission and reception of IMU data and odometry information, as well as low-level attitude control at a frequency of 30Hz or higher, to ensure the UAV's flight stability. Simultaneously, the main thread maintains a circular image buffer. It stores the latest frames captured by the camera in real time.

[0061] Step 2.2: The decision-making reasoning thread runs as an independent process. When it completes the previous round of reasoning and is ready to make the next decision, it does not read the buffer sequentially, but instead retrieves the latest timestamp. Image frames And immediately discard the old frames in the buffer.

[0062] Step 3: Construct multi-source context tensors: convert image frames Encoding as visual cues Based on the current subtask description Next sub-task preview and historical action sequences Construct a multi-source contextual cue tensor ;

[0063] In this embodiment, to compensate for the limitations of single-frame images in spatial perception, it is necessary to provide the model with a spatiotemporal context. Specifically, S3 involves using CLIP or a similar visual encoder to process the image... Encode as a visual embedding, or directly convert to a Base64 stream as a visual cue. At the same time, the constraints of the current subtask will be... Information previewing future missions And the past Frames (e.g.) ) historical action text sequence Modal splicing is performed, and the formula is as follows: This multi-source fusion mechanism endows the system with the ability of "short-term memory" and "trend prediction".

[0064] Step 4: Perform end-to-end decision reasoning: Input a large visual language model and output the navigation action command for the current moment. ;

[0065] In this embodiment, a large visual language model (such as GPT5 or Tongyi Qianwen) deployed at the edge or in the cloud is used to process the tensor. End-to-end inference is performed. The model output includes not only specific action instructions (such as turn_left, move_forward), but also a brief semantic analysis of the current scene, enhancing the interpretability of the system.

[0066] Step 5: Generate macro action sequence: Determine Does it belong to the pre-defined complex maneuvering space? If so, then expand the function using macro actions. Generate includes Open-loop control sequence of basic action elements And store them in a high-priority action queue for execution by the main control thread;

[0067] In this embodiment, for complex maneuvers such as large rotations or U-turns, the UAV camera will experience severe motion blur or sudden changes in perspective, causing vision-based closed-loop control to fail. To address this issue, this invention introduces a macro-motion mechanism.

[0068] In this embodiment, step 5 specifically includes:

[0069] Step 5.1: The system presets a complex maneuvering space. It includes instructions such as "make a sharp right turn" and "make a 180-degree U-turn".

[0070] Step 5.2: When the deduced action... When calling the macro action expansion function. For example, for "90-degree right turn", the function decomposes it into... The basic sequence of motion elements for a "15-degree right yaw". .

[0071] Step 5.3: Sequence The actions are pushed into the high-priority macro action queue of the main control thread. At this point, the main thread pauses sending requests to the decision-making thread and rapidly executes the actions in the queue using an open-loop control approach until the queue is empty. This mechanism simulates the muscle memory of organisms, effectively mitigating the decision-making risks caused by visual blind spots.

[0072] Step 6: Perform geometric constraint termination judgment: Construct an anti-illusion judgment model based on spatial geometric features; calculate the center deviation of the target object in the current field of view. Pixel area ratio The current subtask is determined if and only if a preset set of geometric inequalities is satisfied. Completed, and the task index was updated.

[0073] In this embodiment, existing technologies often employ implicit probabilistic models to determine task completion, which can easily lead to premature task termination when the large model becomes misled (e.g., seeing the target in the distance and mistakenly believing it has been reached). This invention abandons probabilistic judgment and establishes a deterministic judgment model based on physical geometric features.

[0074] In this embodiment, step 6 specifically includes:

[0075] Step 6.1: Obtain the bounding box of the target object recognized by the model. .

[0076] Step 6.2: According to the formula Calculate the target center deviation, where the image width is... The height is ;

[0077] Step 6.3: According to the formula Calculate the area ratio of the target pixels.

[0078] Step 6.4: Execute the termination decision logic: if and only if If both are true, the subtask is considered complete, and the task index pointer is incremented by one. and This is a preset strict threshold (e.g., 100 pixels and 15% occupancy). If this threshold is not met, the system will forcibly override the model's completion signal and continue performing the approach action.

[0079] Example 2:

[0080] This application also provides a UAV visual language navigation system based on asynchronous cooperative architecture and target constraint decision-making. The system is used to implement the method described in Embodiment 1, including:

[0081] The global planning module is used to run the large language model client and is responsible for obtaining natural language navigation instructions. Using a large language model as a global planner, a mapping function is constructed. Natural language navigation instructions Mapped to a set of time-dependent discrete subtasks , where i is the subtask number;

[0082] The visual language navigation controller, as the core scheduling unit, integrates an image buffer and a macro action queue protected by a mutex lock, and runs independent communication main threads and AI inference threads.

[0083] The visual language navigation controller is used to construct an asynchronous parallel architecture that includes a main control thread and a decision reasoning thread; the main control thread maintains the image buffer in real time. The decision-making and reasoning thread at time Extract the image frame with the highest timestamp from the buffer. ,satisfy For the most recent moment, the timestamp in the discard buffer is lower than [previous time]. The old frame;

[0084] The visual language navigation controller is also used to determine navigation action commands. Does it belong to the pre-defined complex maneuvering space? If so, then expand the function using macro actions. Generate includes Open-loop control sequence of basic action elements The control sequence Stored in a high-priority action queue for execution by the main control thread;

[0085] A multimodal decision module is used to run a large visual language model, and the multimodal decision module is used to process image frames. Encoding as visual cues Based on the current subtask description Next sub-task preview and historical action sequences Construct a multi-source contextual cue tensor ;

[0086] The multimodal decision module is also used to process the multi-source context cue tensor. Input a large visual language model and output the navigation action command for the current moment. ;

[0087] The anti-illusion verification module is used to calculate the center deviation of the target object in the current field of view. Pixel area ratio The current subtask is determined if and only if a preset set of geometric inequalities is satisfied. Complete the task and update the task index, then verify the physical feasibility and target constraints of the judgment results.

[0088] Example 3:

[0089] This application also provides a mobile terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the above-described UAV visual language navigation method based on asynchronous cooperative architecture and target constraint decision-making.

[0090] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the invention. Therefore, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention should fall within the protection scope of the present invention.

Claims

1. A method for visual language navigation of UAV based on asynchronous collaborative architecture and target constraint decision-making, characterized in that, Includes the following steps: Step 1, global task space mapping is performed: natural language navigation instructions are obtained , a large language model is used as a global planner to construct a mapping function , natural language navigation instructions are mapped into a set of discrete sub-tasks with time sequence dependence , i is the sub-task number; Step 2: Perform asynchronous timing control: Construct an asynchronous parallel architecture that includes a main control thread and a decision inference thread; the main control thread maintains the image buffer in real time. The decision-making and reasoning thread at time Extract the image frame with the highest timestamp from the buffer. ,satisfy For the most recent moment, the timestamp in the discard buffer is lower than [previous time]. The old frame; Step 3: Construct multi-source context tensors: convert image frames Encoding as visual cues Based on the current subtask description Next sub-task preview and historical action sequences Construct a multi-source contextual cue tensor ; Step 4: Perform end-to-end decision reasoning and convert the multi-source context cue tensor into a single tensor. Input a large visual language model and output the navigation action command for the current moment. ; Step 5, macro action sequence generation is performed to determine whether the navigation action instruction belongs to a preset complex maneuver action space ;​ If so, then expand the macro action function generating an open loop control sequence comprising one base action primitive storing the control sequence in a high priority action queue for execution by a main control thread Step 6: Perform target constraint termination determination and construct an anti-illusion determination model based on spatial geometric features; Calculate the center deviation degree of the target object in the current field of view With the pixel area ratio Determine the current subtask, and update the task index, when and only when the preset geometric inequality group is satisfied is completed, and the task index is updated.

2. The method of claim 1, wherein, The mapping function in step 1 Specifically represented as: ； wherein, is a system-level prompt word comprising a thought chain induction and a JSON format constraint; Generated subtasks For a structured quad: ,in Number the task. For action semantics, For the characteristics of the target object, This is the condition for terminating the task.

3. The method of claim 1, wherein, the multi-source context cue word tensor in the step 3 The construction formula is: ； wherein, denotes a modal stitching operation, denotes a text embedding representation, denotes a text serialization representation of historical actions; The historical action sequence Includes the past Execution actions at each time step It is used to suppress oscillations.

4. The method of claim 1, wherein, The macro action expansion function in step 5 is defined as: When navigation action command is detected When the command is for large rotation, it is decomposed into Discrete micro-rotational motion : ； wherein , is the target rotation angle; While the main control thread executes the control sequence of the process, the decision reasoning thread is temporarily suspended and the system is in an open loop control state.

5. The method of claim 1, wherein, The target constraint termination determination in step 6 specifically includes: The bounding box of the target object is defined as , the image width is , the height is , , , , ; Central deviation formula: ； Pixel area ratio formula: ； The decision-making process is completed if and only if the following logic is satisfied: ； wherein, is a hallucination judgment model output result feature, is a center deviation threshold value, is a minimum area ratio threshold value, is a target object feature.

6. An unmanned aerial vehicle visual language navigation system based on asynchronous collaborative architecture and target constraint decision making, the system being configured to implement the method of any one of claims 1 to 5, characterized in that, include: The global planning module is used to run the large language model client and is responsible for obtaining natural language navigation instructions. Using a large language model as a global planner, a mapping function is constructed. Natural language navigation instructions Mapped to a set of time-dependent discrete subtasks , where i is the subtask number; The visual language navigation controller, as the core scheduling unit, integrates an image buffer and a macro action queue protected by a mutex lock, and runs independent communication main threads and AI inference threads. The visual language navigation controller is used to construct an asynchronous parallel architecture that includes a main control thread and a decision reasoning thread; the main control thread maintains the image buffer in real time. The decision-making and reasoning thread at time Extract the image frame with the highest timestamp from the buffer. ,satisfy For the most recent moment, the timestamp in the discard buffer is lower than [previous time]. The old frame; The visual language navigation controller is also used to judge whether the navigation action instruction belongs to a preset complex maneuver action space ;​ If so, then expand the function using macro actions. Generate includes Open-loop control sequence of basic action elements The control sequence Stored in a high-priority action queue for execution by the main control thread; A multimodal decision module is used to run a large visual language model, and the multimodal decision module is used to process image frames. Encoding as visual cues Based on the current subtask description Next sub-task preview and historical action sequences Construct a multi-source contextual cue tensor ; The multi-modal decision module is also configured to generate a multi-source context cue tensor An input visual language large model outputs a navigation action instruction at a current time ; The anti-illusion verification module is used to calculate the center deviation of the target object in the current field of view. Pixel area ratio The current subtask is determined if and only if a preset set of geometric inequalities is satisfied. Complete the task and update the task index, then verify the physical feasibility and target constraints of the judgment results.

7. A mobile terminal comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the UAV visual language navigation method based on asynchronous cooperative architecture and target constraint decision-making as described in any one of claims 1 to 5.

Images