A Multi-Dimensional Thinking Chain-Based Natural Language Multimodal Navigation Method and System for Unmanned Aerial Vehicles

Abstract

This invention discloses a natural language multimodal navigation method and system for unmanned aerial vehicles (UAVs) based on a multidimensional thought chain. The method primarily involves constructing a software-in-the-loop simulation environment deeply integrated with mission planning software to acquire spatiotemporally stamped real-world images. Next, natural language mission instructions are parsed, and key landmarks and their coordinates are identified by combining the real-world images. Then, the identification results and the structured mission sequence are input in parallel into a multidimensional thought chain inference engine for processing. Finally, the output of the multidimensional thought chain inference engine is integrated to generate a flight action sequence, which is then sent to the flight control system to complete the navigation mission. By embedding the explicit thought chain inference process into each step of the navigation decision-making process and deeply integrating it with the existing mission planning ecosystem, this application can significantly improve the autonomy, adaptability, and mission success rate of UAVs in unknown or dynamic environments.

Description

Technical Field

[0001] This invention relates to the field of autonomous navigation technology for unmanned aerial vehicles (UAVs), and more specifically to a UAV natural language multimodal navigation method and system based on a multidimensional thought chain. Background Technology

[0002] With the development of multimodal large language model technology, Vision-Language Navigation (VLN) for unmanned aerial vehicles (UAVs) has become an important research direction for improving UAV environmental perception and mission autonomy. This technology aims to enable UAVs to understand the operator's natural language commands, thereby navigating autonomously in complex three-dimensional environments.

[0003] Currently, traditional drone navigation systems heavily rely on the Global Positioning System (GPS) and pre-programmed waypoints. They are poorly adaptable to dynamic and unstructured environments and cannot respond to high-level, flexible natural language commands, which limits the application of drones in complex scenarios such as search and rescue and logistics delivery.

[0004] Secondly, existing VLN research is mostly focused on ground robots (such as indoor "room-to-room" tasks). However, UAV navigation involves obstacle avoidance in three-dimensional space, long-distance path planning, rapidly changing observation perspectives, and its own flight dynamics constraints, making it difficult to directly apply ground-based VLN solutions to aerial scenarios, resulting in decreased navigation accuracy and reliability.

[0005] Furthermore, advanced MLLM-based UAV VLN systems (such as OpenFly and UAV-VLN) typically treat the MLLM as a "black box," directly outputting navigation waypoints or maneuvers. This approach lacks an explicit, structured reasoning process, making it impossible to trace and verify the basis of decisions. Once the environmental scene deviates from the model's training data distribution or perceptual ambiguity arises, the model is prone to making subtle decision errors, leading to navigation deviations. These errors accumulate over long sequences of decisions, eventually causing the UAV to be unable to complete its mission or triggering passive safety mechanisms such as "crashing into a wall and returning to base," resulting in complete mission failure.

[0006] Furthermore, existing systems suffer from low integration with mature mission planning software, making it difficult to achieve closed-loop control. Currently, most research-based VLN systems fail to deeply integrate with widely used industrial mission planning software (such as Mission Planner). This prevents the system from fully utilizing the mature flight control, waypoint management, and data link communication capabilities already provided by these software programs. It also hinders the formation of an effective closed loop between the high-level semantic decisions of VLN and the underlying, real-time flight attitude control, battery management, and safety protocols, thus limiting its practicality and reliability in real-world missions.

[0007] Therefore, there is an urgent need in this field for a technical solution that can overcome the above-mentioned defects. Summary of the Invention

[0008] In view of the above problems, the present invention is proposed to provide a method and system for UAV natural language multimodal navigation based on multidimensional thinking chain to overcome or at least partially solve the above problems.

[0009] To achieve the above objectives, the present invention adopts the following technical solution:

[0010] In a first aspect, embodiments of the present invention provide a method for UAV natural language multimodal navigation based on a multidimensional thought chain, comprising the following steps:

[0011] S1: Construct a software-in-the-loop simulation environment that is deeply integrated with the mission planning software, communicate with the mission planning software through the flight control protocol, obtain real-time video streams in the simulation environment, and acquire real-scene images with time and space stamps.

[0012] S2: Parse the natural language task instructions to obtain a structured task sequence; at the same time, process the real-world image through a visual language model to identify at least one key landmark and the coordinates and relative position of the key landmark in the image;

[0013] S3: Input the recognition results and the structured task sequence into the multidimensional thinking chain reasoning engine in parallel for processing; the multidimensional thinking chain reasoning engine includes at least an action generation chain, a history storage chain, a natural language reasoning chain, and a tool construction coordinator;

[0014] S4: Integrate the output of the multi-dimensional thinking chain inference engine to generate the final executable flight action sequence, and convert the action sequence into low-level control commands that the mission planning software can recognize through the flight control protocol, and send them to the flight control system to control the UAV to complete the navigation task; at the same time, based on the results after the command execution, synchronously update the historical state and navigation context information in the multi-dimensional thinking chain inference engine.

[0015] Preferably, in S1, the software-in-the-loop simulation environment deeply integrated with the task planning software is constructed, including:

[0016] Start and interconnect the software-in-the-loop simulator, task planner, and 3D visualization environment, including:

[0017] The mission planner is connected to the flight controller in the software-in-the-loop simulator via TCP / IP protocol, and is used to send mission commands and receive flight status.

[0018] The flight controller sends control commands to the physics simulation engine via the first UDP protocol;

[0019] The physical simulation engine sends the dynamic model information containing the UAV's position and attitude to the three-dimensional visualization environment via the second UDP protocol;

[0020] The 3D visualization environment broadcasts its rendered video stream via a third UDP protocol;

[0021] By capturing the video stream to acquire real-world images, and by sending commands based on the MAVLink protocol format to the mission planner or the flight controller, the UAV can be controlled, forming a deeply integrated closed-loop system from perception and decision-making to control.

[0022] Preferably, in S2, parsing the natural language task instructions includes:

[0023] Using a pre-defined text parser, natural language instructions are structured and parsed into sequences containing landmark fields and standard verb fields based on single-sample prompts; wherein the verbs in the standard verb fields are taken from a predefined closed action set. The predefined closed action set is as follows:

[0024] {takeoff, land, goto, rtl, left, right, forward, back, up, down,hover}.

[0025] Preferably, in S2, processing the real-world image using a visual language model includes:

[0026] A multimodal large language model with 4-bit quantization is used to process images using inter-layer parallel computing techniques; by constructing specific query suggestions, the model is forced to output results in the format of "landmark name-pixel coordinates-relative position".

[0027] Preferably, in S3, the action generation chain includes:

[0028] Receive the structured task sequence and key landmark information;

[0029] The prompt word command forces the high-level navigation maneuver to be decomposed into one or more atomic actions, and outputs the target point coordinates and the sequence of atomic actions. The prompt word includes the following constraints:

[0030] Command constraints: Force the decomposition of high-level navigation commands into atomic actions within a closed action set;

[0031] Accuracy constraint: It stipulates that absolute geographic coordinates obtained from real-time observed landmark information must be used when performing waypoint flight maneuvers;

[0032] Energy efficiency constraints: The output action sequence association includes energy efficiency assessment information such as task duration estimation and power consumption estimation.

[0033] Preferably, in S3, the historical storage chain includes:

[0034] Navigation history states are recorded in a lightweight, structured format, which includes at least the following:

[0035] The task timeline is used to record key nodes and states in the sequence of actions.

[0036] Landmark visit statistics are used to aggregate observation data on landmarks;

[0037] Performance metrics, used to pre-calculate core indicators related to navigation energy efficiency;

[0038] When a path deviation is detected, the path replanning is initiated by backtracking to the previous successful landmark access node based on the historical state.

[0039] Preferably, in S3, the natural language reasoning chain includes:

[0040] Receive historical context information from the historical storage chain, perform quantitative risk assessment and confidence score for the current decision; when the risk level exceeds the threshold or the confidence level is lower than the threshold, generate one or more alternative action plans; each alternative plan includes a specific action, the reason for generation and its risk level.

[0041] Preferably, in S3, the tool construction coordinator is:

[0042] The system receives the outputs of the action generation chain, the historical storage chain, and the natural language inference chain, and performs decision consistency verification according to a preset set of integrated rules. The set of integrated rules includes at least: risk assessment alignment between the action sequence and the inference chain, historical context guiding the current decision, and compliance with security constraints.

[0043] When a conflict is detected during verification, arbitration is performed based on the priority of the rule set, and a final unified action sequence is output.

[0044] Secondly, embodiments of the present invention provide a UAV natural language multimodal navigation decision system based on a multidimensional thought chain. This system applies any of the above-described UAV natural language multimodal navigation methods based on a multidimensional thought chain, including:

[0045] The simulation and environment perception module is used to construct a simulation environment that is deeply integrated with the mission planning software, and communicates with the mission planning software through the flight control protocol to obtain real-time video streams in the simulation environment and acquire real-scene images with time and space stamps.

[0046] A multimodal information processing module is used to parse natural language commands and identify key landmarks in images;

[0047] A multidimensional thought chain reasoning engine, including a processor for executing the action generation chain, history storage chain, natural language reasoning chain, and tool construction coordinator;

[0048] The system integration and control module is used to generate final control commands and send the commands to the mission planning software through the flight control protocol, and then to the flight control system, so as to achieve deep integration and closed-loop interaction with the simulation system or the actual flight control system.

[0049] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the UAV natural language multimodal navigation method based on multidimensional thought chain as described in any of the preceding claims.

[0050] The technical solutions provided in the embodiments of the present invention aim to embed explicit thought chain reasoning processes into every step of navigation decision-making and deeply integrate them with the existing mission planning ecosystem, thereby significantly improving the autonomy, adaptability, and mission success rate of UAVs in unknown or dynamic environments.

[0051] The specific beneficial effects include at least the following:

[0052] 1. This invention achieves a high degree of integration of mission planning software and realizes closed-loop control; that is, it can generate standardized low-level flight control commands, which can be directly embedded into the existing UAV control link, realizing a complete closed loop from perception, decision-making to control;

[0053] 2. By combining text parsers and visual language models, semantic instructions can be transformed into specific perceptions and action sequences of the physical environment, avoiding reliance on precise GPS coordinates or pre-programmed waypoints.

[0054] 3. By constructing a multi-dimensional thinking chain reasoning engine consisting of an action generation chain, a history storage chain, a natural language reasoning chain, and a tool construction coordinator, the complex navigation decision-making task is decomposed into specialized sub-tasks such as safety constraints, state memory, risk assessment, and consistency verification. This eliminates the unpredictable risks and accumulated errors caused by the direct output of end-to-end "black box" models in existing technologies. At the same time, through the collaboration and checks and balances between the chains, a built-in decision safety redundancy is formed, ensuring the interpretability, verifiability, and interruptibility of the entire process from high-level semantic instructions to low-level control instructions. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0056] Figure 1 This is a framework diagram of the deeply integrated closed-loop system in this invention;

[0057] Figure 2 This is a diagram showing the format of the prompt words for this invention;

[0058] Figure 3 This is a flowchart illustrating the UAV natural language multimodal navigation method based on multidimensional thinking chain according to the present invention. Detailed Implementation

[0059] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0060] This invention discloses a method for UAV natural language multimodal navigation based on multidimensional thought chains, the steps of which include:

[0061] S1: Construct a software-in-the-loop simulation environment that is deeply integrated with the mission planning software, communicate with the mission planning software through the flight control protocol, obtain real-time video streams in the simulation environment, and acquire real-scene images with time and space stamps.

[0062] S2: Parse the natural language task instructions to obtain a structured task sequence; at the same time, process the real-world image through a visual language model to identify at least one key landmark and the coordinates and relative position of the key landmark in the image;

[0063] S3: Input the recognition results and the structured task sequence into the multidimensional thinking chain reasoning engine in parallel for processing; the multidimensional thinking chain reasoning engine includes at least an action generation chain, a history storage chain, a natural language reasoning chain, and a tool construction coordinator;

[0064] S4: Integrate the output of the multi-dimensional thinking chain inference engine to generate the final executable flight action sequence, and convert the action sequence into low-level control commands that the mission planning software can recognize through the flight control protocol, and send them to the flight control system to control the UAV to complete the navigation task; at the same time, based on the results after the command execution, synchronously update the historical state and navigation context information in the multi-dimensional thinking chain inference engine.

[0065] This application, upon inputting a UAV natural language task command, identifies key landmarks and action terms within the text command. Simultaneously, it activates a simulated image capture function, inputting a query into the visual language model to obtain the coordinates of key landmarks. Further, the task command and perceived information are fed into a multi-dimensional thought chain reasoning engine for decision-making, providing an action sequence and reasoning process. Finally, the action sequence is input into a command dispatcher to obtain SITL-level control commands, which are then transmitted to the underlying controller via TCP. Single-step simulation is implemented in a fully software-based simulation, and the next step of the single-step action is performed based on the command until the UAV reaches the target point, ending the navigation process.

[0066] In one embodiment, step S1 is used to construct a simulation loop and acquire real-time photos; the steps include:

[0067] Launch and interconnect the software-in-the-loop simulator, task planner, and 3D visualization environment;

[0068] The mission planner connects to the flight controller in the software-in-the-loop simulator via the TCP / IP protocol to send mission commands and receive flight status.

[0069] The flight controller sends control commands to the physics simulation engine via the first UDP protocol;

[0070] The physical simulation engine sends the dynamic model information containing the UAV's position and attitude to the three-dimensional visualization environment via the second UDP protocol;

[0071] The 3D visualization environment broadcasts its rendered video stream via a third UDP protocol;

[0072] By capturing the video stream to acquire real-world images, and by sending commands based on the MAVLink protocol format to the mission planner or the flight controller, the UAV can be controlled, forming a deeply integrated closed-loop system from perception and decision-making to control.

[0073] In some implementations, Python scripts send commands via MAVLink to acquire photos. The terminal launches the SITL control module, the Mission Planner planning module, and the FlightGear 3D navigation module processes, establishing a simulation loop via UDP and TCP connections. For example... Figure 1 As shown, FlightGear's Video Streaming encodes the video in real time and broadcasts each frame of MJPEG to the local loopback address and specified port via the UDP protocol. Python uses cv2.VideoCapture to pull the stream and simultaneously capture RGB real-world images with latitude, longitude, and timestamps into the library to achieve photo capture.

[0074] In one embodiment, step S2 includes:

[0075] Using a pre-defined text parser, natural language instructions are structured and parsed into sequences containing landmark fields and standard verb fields based on single-sample prompts;

[0076] A multimodal large language model with 4-bit quantization is used to process images using inter-layer parallel computing techniques; by constructing specific query suggestions, the model is forced to output results in the format of "landmark name-pixel coordinates-relative position".

[0077] In some specific implementations, a text parser is built to process task instructions to obtain key landmarks and action text segments, and a query is set. The molmo model is then used to obtain the coordinate values ​​of the key landmarks and their relative positions in the photos based on the query and the collected photos.

[0078] The text parser uses a one-shot structured prompt to solidify Roles, Goals, and Workflows, enabling the large model to break down any single natural language command into a "landmark + standard verb" JSON sequence in one go; the verbs are limited to {takeoff, land, goto, rtl, left, right, forward, back, up, down, hover}. The text parser upgrades traditional ground station waypoint navigation to natural language command navigation, using standard verb sequences to cover the entire set of ICAO phrase actions, ensuring that commonly used UAV navigation natural language commands can be parsed, and the landmark field corresponds to the "geographic reference point" in DO-178C.

[0079] Furthermore, 4-bit quantization of the Molmo-7B-O is used to automatically call GPU resources for computation, and HF-accelerate is used for inter-layer parallelism. The processor concatenates the image block tokens with the target word packets in the query to form a causal sequence, and generates coordinate text with fixed labels (max 2k tokens, stopping at <|endoftext|>) through the generate_from_batch autoregressive method, outputting an XML template in the format Landmark:xxx / Coordinates:[x,y] / Position:xxx. The edited query forces the model to report in three steps: "landmark-coordinates-relative position", corresponding to the "significant terrain + relative positioning" principle recommended by the Civil Aviation Administration of China (ICAO). The output format is uniformly set to the top left of the pixel origin, paired coordinates, and relative position, which can be input into the navigation agent for visual track backup, realizing low-altitude navigation perception for UAVs.

[0080] In one embodiment, step S3: Constructing the COT multidimensional derivation chain:

[0081] The COT derivation chain is established through prompt, and the results processed by the previous text parser module and molmo model are input. The output is the At sequence and the inference flow of the LLM sequence.

[0082] In this embodiment, after obtaining the pixel coordinates of the key landmarks identified by the molmo model, a coordinate transformation is required to convert the pixel coordinates into latitude, longitude, and altitude coordinates in the WGS-84 coordinate system. First, the pixel coordinates are normalized using pre-calibrated camera intrinsic parameters and distortion coefficients. Then, a rotation matrix is ​​multiplied by the normalized coordinates to transform the coordinate vector from the camera's own coordinate system to the UAV's body coordinate system. Next, real-time attitude data—roll, pitch, and yaw angles—provided by the UAV's IMU is used. A rotation matrix R_body_to_ned is constructed from the body coordinate system to the NED coordinate system using these Euler angles. The direction vector V_body in the body coordinate system is then transformed to V_ned in the NED coordinate system. Finally, the precise latitude, longitude, and altitude of the UAV in the WGS-84 coordinate system are obtained from the UAV's GNSS receiver. The NED coordinates of the target point are obtained through scaling, and the NED coordinates are converted back to the final WGS-84 latitude, longitude, and altitude coordinates using an Earth ellipsoid model.

[0083] Furthermore, upon receiving structured instructions and the latitude, longitude, and altitude coordinates of key landmarks, a command-observation landmark name mapping is performed. The `_run_action_chain` function first reads the instructions for the current step from `mission_instructions`, forming the "action request" for this mission. The landmark name is extracted using `step.get("landmark")`; if empty, the drone's current location is used by default; otherwise, the corresponding real WGS-84 coordinates are searched in the `observed_landmarks` dictionary using lowercase keys, thus mapping the command string to actual coordinates. A coordinate transformation function is then used to convert pixel coordinates to latitude, longitude, and altitude coordinates, ensuring that subsequent decisions are based on real geographic data. The mapped coordinates, current altitude, battery status, etc., are packaged into the `observed_real` dictionary and fed into the `action_chain` model along with the mission history. Based on these real observations and historical performance, the model determines whether takeoff, goto, left / right translation, or landing is required, and outputs a parameterized MAVLink-level action JSON.

[0084] For UAV navigation tasks, COT Mind Chain uses four LangChain sub-chains to connect "language-perception-state-history" into a closed loop, enabling hierarchical decision-making and collaborative reasoning, and realizing memory, decision-making, judgment and backtracking functions.

[0085] The action generation chain generates target points and their latitude, longitude, and altitude coordinates in real time using WGS-84 landmark-coordinate mapping, along with JSON action sequences (selected from {takeoff, land, goto, rtl, left, right, forward, back, up, down, hover}). The history storage chain records navigation history in Markdown-JSON format, enabling precise return to the previous key landmark and drone status when the drone deviates from its course, thus achieving historical backtracking. The LLM inference chain enforces a six-step interpretable output of "command-landmark-energy efficiency-history-rules-safety," providing confidence levels and risk alternatives. The tool constructs a coordinator for weighted disambiguation, outputting a unique At sequence.

[0086] In some implementations, the action generation chain includes:

[0087] Receive the structured task sequence and key landmark information;

[0088] The prompt word command forces the high-level navigation maneuver to be decomposed into one or more atomic actions, and outputs the target point coordinates and the sequence of atomic actions. The prompt word format is as follows: Figure 2 As shown, and includes the following constraints:

[0089] Command constraints: Force the decomposition of high-level navigation commands into atomic actions within a closed action set;

[0090] Accuracy constraint: It stipulates that absolute geographic coordinates obtained from real-time observed landmark information must be used when performing waypoint flight maneuvers;

[0091] Energy efficiency constraints: The output action sequence association includes energy efficiency assessment information such as task duration estimation and power consumption estimation.

[0092] In the field of UAV navigation, the selection of flight actions needs to consider UAV dynamics adaptation and safety issues. The core is to avoid power failure, communication interruption, positioning loss, and collisions with the environment. To this end, this invention, in the task description section of the prompt, encloses the action set with "You must only output actions from the following set:" and specifically prohibits the creation of new verbs. The constructed action set {takeoff, land, goto, rtl, left, right, forward, back, up, down, hover} conforms to the action decomposition of UAV flight characteristics, enabling dynamic adaptation for UAV vertical takeoff, 3D navigation, hovering stability, and lateral maneuvering. Simultaneously, the prompt "Any high-level maneuver must be decomposed into one or more of the above primitives." prompts the large model to use a safe decomposition mechanism to break down high-level commands (such as "reconnoiter a rooftop") into sub-operations in the action space. Furthermore, the prompt "Do not introduce new verbs like ascend, up, down, etc." restricts the large model from generating actions outside the action set, preventing the introduction of non-standard verbs. Each step can be monitored and terminated independently. Even if a single action fails, the system can still generate high-level RTL instructions through the thought chain to roll back to a safe state.

[0093] Furthermore, in UAV visual language navigation, ensuring navigation accuracy and real-time performance is a core challenge. This typically requires the system to execute actions based on high-confidence real-time environmental perception and strictly avoid cumulative errors caused by coordinate calculations or ambiguous commands. Therefore, the prompt's information input section uses `##CURRENT OBSERVED LANDMARKS(Name ->Lat / Lon / Alt):{observed_landmarks}` to continuously and synchronously observe the latitude, longitude, and altitude coordinates of landmarks in real time. The prompt's task description section uses a CRITICAL level requirement: "When using the 'goto' action, you MUST use the actual WGS-84 coordinates from OBSERVED_LANDMARKS." This restricts the large model to using the actual WGS-84 coordinates obtained from "OBSERVED_LANDMARKS" for the "goto" action, ensuring the real-time nature of the coordinates used for navigation. Simultaneously, directly using WGS-84 coordinates for navigation prevents errors caused by relative coordinate calculation algorithms.

[0094] In UAV navigation mission planning, navigation energy efficiency is a core performance indicator. It is typically evaluated using metrics such as total mission duration, estimated battery consumption, and path efficiency. The goal is to optimize flight time and energy consumption while ensuring mission success. The output format template for the prompts uses "expected_duration": 30.5, # estimated mission time; "battery_consumption": 2.5, # predicted battery consumption" to constrain the large model's reasoning under energy efficiency constraints. This guides the model to select more efficient path and action combinations during planning, enabling system-level energy efficiency monitoring and ensuring safety throughout the UAV navigation process.

[0095] In some implementations, the historical storage chain includes:

[0096] Navigation history states are recorded in a lightweight, structured format, which includes at least the following:

[0097] The task timeline is used to record key nodes and states in the sequence of actions.

[0098] Landmark visit statistics are used to aggregate observation data on landmarks;

[0099] Performance metrics, used to pre-calculate core indicators related to navigation energy efficiency;

[0100] When a path deviation is detected, the path replanning is initiated by backtracking to the previous successful landmark access node based on the historical state.

[0101] In this embodiment, the history_chain prompt can realize lightweight management of UAV navigation status data and achieve the traceability of UAV navigation.

[0102] Due to the flight dynamics limitations of drones, the onboard weight must be as light as possible, and the computing power and storage resources of the onboard computer are limited. This requires the system to strictly manage the historical navigation status in a lightweight manner, so as to retain key decision information while avoiding storage overflow caused by the accumulation of raw data.

[0103] Therefore, the output format template for prompt words is through

[0104] "historical_storage": {

[0105] "mission_timeline": [], / / Only retain structured key data

[0106] "landmark_visits": {}, / / Aggregate statistics to avoid piling up raw data

[0107] "performance_metrics": {} / / Pre-calculated metrics reduce real-time computing load

[0108] }

[0109] Historical information is compressed into a concise three-dimensional structure: `mission_timeline` retains only key nodes and states of the action sequence; `landmark_visits` processes raw observation data and aggregates statistical features of landmark visits; and `performance_metrics` pre-calculates core performance metrics to facilitate large models' navigation path planning inference based on energy efficiency constraints. This design ensures lightweight storage while covering the three complete dimensions of mission timeline, spatial behavior patterns, and system performance, maintaining the structural integrity of historical navigation state information.

[0110] During actual drone navigation, GPS signal fluctuations, visual positioning drift, or the appearance of sudden obstacles (such as birds or temporary structures) often lead to navigation path deviations or mission interruptions. In such cases, precise navigation status backtracking is essential to pinpoint the problem and initiate replanning. The prompts, through the `mission_timeline` module, precisely record the timestamp of each action, the drone's status (location, battery level), and execution results. When a path deviation or obstacle is detected, the system can immediately backtrack to the last successful landmark visit (via the visit records in `landmark_visits`) and quickly determine the root cause of the deviation based on pattern recognition in `historical_analysis` (e.g., "vulnerable to GPS interference in specific areas"). This structured historical storage not only provides a baseline state for replanning but also identifies high-success-rate path characteristics through `performance_metrics`, thereby generating alternative paths that bypass obstacle areas or have stronger anti-interference capabilities after backtracking.

[0111] In some implementations, the natural language inference chain includes:

[0112] Receive historical context information from the historical storage chain, perform quantitative risk assessment and confidence score for the current decision; when the risk level exceeds the threshold or the confidence level is lower than the threshold, generate one or more alternative action plans; each alternative plan includes a specific action, the reason for generation and its risk level.

[0113] In this application, the Natural Language Inference Chain (LLM_chain) prompts implement multi-dimensional risk assessment and security redundancy mechanisms, as well as deep integration of inference, decision-making, and history, enabling UAVs to make experience-based decisions and continuously evolve navigation.

[0114] In the aviation industry, redundancy is a core principle for ensuring flight safety. This principle requires critical systems and decision-making processes to be equipped with multiple backups to ensure that the failure of a single component or decision does not lead to systemic catastrophic consequences. The output format template for historical storage chain prompts is...

[0115] {

[0116] "confidence_score": 0.92, / / Confidence metric

[0117] "risk_assessment": "low", / / Risk level classification

[0118] "alternative_options": [] / / Alternative options

[0119] }

[0120] The large model is guided to perform confidence scoring and risk level assessment on the current navigation scheme, and alternative schemes are generated based on the confidence score and risk level. Specifically, this is done through...

[0121] "alternative_options": [

[0122] {

[0123] "action": "alternative_action",

[0124] "rationale": "why this alternative",

[0125] "risk_level": "medium"

[0126] } ]

[0128] The guiding model provides specific alternative actions in the alternative schemes, the reasoning process for generating alternative actions, and the risk level of alternative actions. This allows the alternative scheme to be switched in a timely manner when the drone navigation system reports a risk, thus ensuring the safety of drone flight.

[0129] Meanwhile, the prompts are tightly integrated with the historical storage chain through the `{historical_context}` parameter, forming a closed loop of reasoning-decision-history. This enables experience-based decision-making, giving the intelligent navigation system the ability to avoid risks, optimize energy efficiency, and continuously evolve. When faced with similar situations, the reasoning chain can invoke these experiences through `{historical_context}`. For example, if historical data shows that "Area A" has repeatedly deviated from its path due to weak GPS signals, then when the drone approaches "Area A" again, the reasoning chain will mark the risk level as "medium" or "high" in `risk_assessment` in advance and may proactively generate alternative options for detours, rather than blindly entering the area. The average energy consumption, task efficiency, and other data stored in `performance_metrics` provide direct evidence for the energy efficiency constraints of the reasoning chain. For example, when choosing between two alternative paths, the reasoning chain may find through historical data that path A, although shorter, is against the wind and has an average energy consumption 20% higher than path B. Therefore, it will prioritize path B and explain in `reasoning_chain` that this is based on energy efficiency optimization. The outcome of each inference decision (whether successful or not) is structured and recorded in the history chain, updating historical_analysis and performance_metrics to optimize the starting point for the next task's inference. For example, if a drone discovers a new shortcut to a target through exploration, this successful experience is recorded and shared with other drones in the cluster, thereby improving the overall mission efficiency of the fleet.

[0130] In some implementations, the tool constructs the coordinator as follows:

[0131] The system receives the outputs of the action generation chain, the historical storage chain, and the natural language inference chain, and performs decision consistency verification according to a preset set of integrated rules. The set of integrated rules includes at least: risk assessment alignment between the action sequence and the inference chain, historical context guiding the current decision, and compliance with security constraints.

[0132] When a conflict is detected during verification, arbitration is performed based on the priority of the rule set, and a final unified action sequence is output.

[0133] In this embodiment, the tool_chain prompt word of the tool coordinator can achieve consistency in UAV navigation decisions, ensuring that the system ultimately obtains a unified and reliable decision.

[0134] In practical applications of UAV navigation, the system faces multiple challenges, including asynchronous decision-making across multiple modules, sensor information conflicts, strictly limited resources, and compliance with aviation safety regulations. This necessitates that the navigation system possess decision consistency and real-time reliability, ensuring the generation of coordinated, unified, and safe final commands even under multi-source information input. The output format template within the prompt words guides the large model in verifying consistency across the entire decision-making chain.

[0135] {

[0136] "consistency_check": "passed",

[0137] "validation_results": [

[0138] "Validation point 1: passed",

[0139] "Validation point 2: passed" ]

[0141] }

[0142] Through consistency checks, the coordinator ensures the logical coherence of the entire decision-making chain from perception and reasoning to execution, avoiding system oscillations or dangerous actions caused by inconsistent outputs between modules.

[0143] Meanwhile, the task description section details the task rules:

[0144] "INTEGRATION RULES": {

[0145] "1": "The action sequence must be aligned with the inference chain",

[0146] "2": "Historical context should guide current decision-making",

[0147] "3": "Confidence scores must be consistent in the output".

[0148] 4: "Safety constraints must be followed".

[0149] 5: "Task objectives should be given priority."

[0150] }

[0151] This rule set establishes a decision-priority framework. When there are contradictions in the outputs of different chains (such as the action chain suggesting to move forward while the reasoning chain identifies risks), the system can analyze according to preset aviation safety criteria to ensure that the final output meets the "safety first" requirements of the UAV navigation industry.

[0152] In one embodiment, step S4: integrate the output of the multidimensional thinking chain inference engine to generate the final executable flight action sequence, and convert the action sequence into low-level control commands that the mission planning software can recognize through the flight control protocol, and send them to the flight control system to control the UAV to complete the navigation task; at the same time, based on the result after the command execution, synchronously update the historical state and navigation context information in the multidimensional thinking chain inference engine.

[0153] In this embodiment, high-level action sequences are transformed into low-level control instructions executable by the low-level controller ArduPilot through coordinate transformation and instruction mapping. The code constructs an instruction dispatcher using a conditional judgment structure to parse the type (ACTION) and associated parameters (LANDMARK_COORDINATE and ALTITUDE) of each action.

[0154] As shown in Table 1, the action parsing maps the high-level action sequences (such as "takeoff", "goto", "land" / "rtl", "left", "right", "forward", "back") output by the COT chain into specific control commands for the UAV API to call.

[0155] Table 1

[0156]

[0157] The control command corresponding to the goto action uses the WGS-84 latitude and longitude coordinate system. Based on the latitude and longitude of the target point specified in the action plan, it is converted into relative navigation coordinates using the LocationGlobalRelative function in the dronekit library, and then the drone is navigated to the target point using the simple_goto method.

[0158] `send_ned_velocity` is the UAV control function used to send a velocity command in the local NED (North-East-Down) coordinate system to the flight controller for a specified duration. This function uses the MAVLink `SET_POSITION_TARGET_LOCAL_NED` message to enable only the velocity fields (vx, vy, vz), controlling the UAV to fly at the set velocity vector without changing its attitude or acceleration. The function first constructs and sends this MAVLink message, then waits for the specified duration to maintain the UAV's velocity.

[0159] In some implementation schemes, the navigation method flow of the present invention is described with reference to... Figure 3 The steps include:

[0160] Simulation loop establishment and photo acquisition: A SITL+MP simulation loop is established, and a Python script sends commands via MAVLink to achieve photo acquisition;

[0161] Multimodal information processing: Construct a text parser to process task instructions, and use the molmo model to obtain the pixel coordinates and relative positions of key landmarks from the query and the collected photos;

[0162] COT Inference Chain: Write a prompt to establish the COT inference chain, input the landmark latitude, longitude and altitude and the processed instructions r, and LLM outputs the natural language inference chain Rt, action sequence At, and history storage Ht;

[0163] System interaction and integration: The output action sequence is further converted into instructions in MP to realize the integration of the intelligent navigation system, and the SITL+MP simulation loop is used for single-step interaction and simulation; step-by-step iteration is performed until the target point is reached.

[0164] Furthermore, in one embodiment, the present invention discloses a UAV natural language multimodal navigation decision system based on multidimensional thought chains. This system applies any of the above-described UAV natural language multimodal navigation methods based on multidimensional thought chains, including:

[0165] The simulation and environment perception module is used to construct a simulation environment that is deeply integrated with the mission planning software, and communicates with the mission planning software through the flight control protocol to obtain real-time video streams in the simulation environment and acquire real-scene images with time and space stamps.

[0166] A multimodal information processing module is used to parse natural language commands and identify key landmarks in images;

[0167] A multidimensional thought chain reasoning engine, including a processor for executing the action generation chain, history storage chain, natural language reasoning chain, and tool construction coordinator;

[0168] The system integration and control module is used to generate final control commands and send the commands to the mission planning software through the flight control protocol, and then to the flight control system, so as to achieve deep integration and closed-loop interaction with the simulation system or the actual flight control system.

[0169] The integration between different modules in this application requires data format restrictions based on the actual situation of UAV navigation. In some implementation schemes, the data format is shown in Table 2:

[0170] Table 2

[0171]

[0172] The action sequence and reasoning process are as follows:

[0173]

[0174] Based on the same inventive concept, embodiments of the present invention also provide a computer-readable storage medium storing a computer program. When executed by a processor, this program implements the UAV natural language multimodal navigation method based on multidimensional thought chains as described in any of the preceding claims. Therefore, the implementation of this program can refer to the implementation of the aforementioned methods, and repeated details will not be elaborated further.

[0175] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0176] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for unmanned aerial vehicle (UAV) natural language multimodal navigation based on multi-dimensional thought chain, characterized in that, Comprise: S1: build a software-in-the-loop simulation environment deeply integrated with task planning software, communicate with the task planning software through flight control protocol, obtain real-time video stream in the simulation environment, and collect real scene images with time and space stamps; S2: analyze the natural language task instruction to obtain a structured task sequence; at the same time, process the real scene image through a visual language model to identify at least one key landmark and the coordinates and relative position of the key landmark in the image; S3: input the identification result and the structured task sequence into a multi-dimensional thought chain reasoning engine for processing; the multi-dimensional thought chain reasoning engine at least includes an action generation chain, a history storage chain, a natural language reasoning chain, and a tool construction coordinator; S4: integrate the output of the multi-dimensional thought chain reasoning engine to generate a flight action sequence, convert the action sequence into a bottom-level control instruction recognizable by the task planning software, and issue the instruction to the flight control system to control the unmanned aerial vehicle to complete the navigation task; at the same time, based on the result after executing the instruction, the history state and navigation context information in the multi-dimensional thought chain reasoning engine are updated synchronously; In S3, the action generation chain comprises: Receiving the structured task sequence and key landmark information; Forcing to decompose high-level navigation maneuvers into one or more atomic actions through prompt word instructions, and outputting target point coordinates and atomic action sequences; wherein the prompt word contains the following constraints: Instruction constraint: force to decompose high-level navigation instructions into atomic actions in a closed action set; Precision constraint: when executing waypoint flight actions, absolute geographic coordinates obtained from real-time observation landmark information must be used; Energy efficiency constraint: the output action sequence is associated with energy efficiency evaluation information containing task time length estimation and power consumption estimation; The history storage chain comprises: Record the navigation history state in a lightweight structured format, which at least includes: Task timeline, used to record key nodes and states of action sequences; Landmark access statistics, used to aggregate observation data of landmarks; Performance indicators, used to pre-calculate core indicators related to navigation energy efficiency; When path deviation is detected, backtrack to the previous successful landmark access node based on the navigation history state, and start path re-planning; The natural language reasoning chain comprises: Receive the historical context information from the history storage chain, perform quantitative risk assessment and confidence scoring on the current decision; when the risk level exceeds the threshold or the confidence is lower than the threshold, generate one or more alternative action plans; each alternative plan includes specific actions, generation reasons, and risk levels; The tool construction coordinator is: Receive the output of the action generation chain, history storage chain, and natural language reasoning chain, and perform decision consistency verification according to the preset integration rule set; the integration rule set at least includes: alignment of action sequence and risk assessment of reasoning chain, historical context guiding current decision, and safety constraints being followed; When conflicts are found in the verification, arbitrate according to the priority of the integration rule set to output the final unified action sequence. 2.The unmanned aerial vehicle natural language multimodal navigation method of claim 1, wherein, In S1, the software-in-the-loop simulation environment deeply integrated with the task planning software comprises: The software-in-the-loop simulator, the task planner and the three-dimensional visualization environment are started and interconnected, wherein: The task planner is connected with the flight controller in the software-in-the-loop simulator through a TCP / IP protocol, for sending task instructions and receiving flight states; The flight controller sends control instructions to the physical simulation engine through a first UDP protocol; The physical simulation engine sends dynamic model information containing the position and attitude of the unmanned aerial vehicle to the three-dimensional visualization environment through a second UDP protocol; The three-dimensional visualization environment broadcasts the video stream rendered thereby through a third UDP protocol; By capturing the video stream to realize real scene image acquisition, and by sending instructions based on the MAVLink protocol format to the task planner or the flight controller, control of the unmanned aerial vehicle is realized, constituting a deep integration closed loop system from perception, decision-making to control. 3.The unmanned aerial vehicle natural language multimodal navigation method of claim 1, wherein, In S2, the natural language task instruction is parsed, including: Through a preset text parser, the natural language instruction is structured and parsed into a sequence containing a landmark field and a standard verb field based on a single sample prompt; wherein the verbs of the standard verb field are taken from a predefined closed action set. 4.The unmanned aerial vehicle natural language multimodal navigation method of claim 1, wherein, In S2, the real scene image is processed by a visual language model, including: A multi-modal large language model quantized by 4-bit is used to process the image using inter-layer parallel computing technology; by constructing specific query prompt words, the model is forced to output results in the format of "landmark name-pixel coordinates-relative position".

5. A multi-dimensional thought chain based unmanned aerial vehicle natural language multi-modal navigation decision system, characterized in that, The unmanned aerial vehicle natural language multi-modal navigation method based on the multi-dimensional thought chain according to any one of claims 1-4, the system comprising: A simulation and environment perception module for building a simulation environment deeply integrated with a task planning software, and communicating with the task planning software through a flight control protocol to obtain real-time video stream in the simulation environment and collect real scene images with time and space stamps; A multi-modal information processing module for parsing natural language instructions and identifying key landmarks in images; A multi-dimensional thought chain reasoning engine including a processor for executing the action generation chain, the history storage chain, the natural language reasoning chain and the tool construction coordinator; A system integration and control module for generating final control instructions and sending the instructions to the task planning software through the flight control protocol, and then to the flight control system, to realize deep integration and closed loop interaction with the simulation system or the actual flight control system.

6. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the unmanned aerial vehicle natural language multi-modal navigation method based on the multi-dimensional thought chain according to any one of claims 1-4.

Images