Unmanned aerial vehicle natural language multi-modal navigation method and system based on multi-dimensional thinking chain

Abstract

The invention discloses an unmanned aerial vehicle natural language multi-modal navigation method and system based on a multi-dimensional thinking chain. The method mainly comprises the following steps: constructing a software-in-the-loop simulation environment deeply integrated with task planning software, and collecting a live-action image with a time-space stamp; secondly, the natural language task instruction is analyzed, and key landmarks and position coordinates of the key landmarks are recognized in combination with a real scene image; then the recognition result and the structured task sequence are input into a multi-dimensional thinking chain reasoning engine in parallel to be processed; and finally, integrating the output of the multi-dimensional thinking chain inference engine, generating a flight action sequence, and issuing the flight action sequence to a flight control system to complete a navigation task. According to the method, the explicit thinking chain reasoning process is embedded into each step of navigation decision and is deeply integrated with the existing task planning ecology, so that the autonomy, adaptability and task success rate of the unmanned aerial vehicle in an unknown or dynamic environment can be remarkably improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of unmanned aerial vehicle autonomous navigation, more particularly to a multi-dimensional thought chain-based unmanned aerial vehicle natural language multi-modal navigation method and system. BACKGROUND

[0002] With the development of multi-modal large language model technology, unmanned aerial vehicle vision-language navigation (VLN) has become an important research direction to improve the environmental perception and task autonomy of unmanned aerial vehicles. This technology aims to enable unmanned aerial vehicles to understand natural language instructions from operators, thereby enabling autonomous navigation in complex three-dimensional environments.

[0003] Currently, traditional unmanned aerial vehicle navigation systems rely heavily on global positioning systems (GPS) and pre-programmed waypoints, which have poor adaptability in dynamic and unstructured environments, and cannot respond to high-level and flexible natural language instructions, limiting the application of unmanned aerial vehicles in complex scenarios such as search and rescue and logistics distribution.

[0004] Secondly, existing VLN researches are mostly focused on ground robots (such as indoor "room-to-room" tasks). However, unmanned aerial vehicle navigation involves three-dimensional space obstacle avoidance, long-distance path planning, rapidly changing observation angles, and self-flight dynamics constraints, making it difficult for ground VLN solutions to be directly applicable to aerial scenarios, resulting in decreased navigation accuracy and reliability.

[0005] In addition, advanced unmanned aerial vehicle VLN systems based on MLLM (such as OpenFly, UAV-VLN, etc.) usually treat MLLM as a "black box" and directly output navigation waypoints or actions from it. This approach lacks an explicit, structured reasoning process and cannot trace and verify the basis for decision-making. Once the environment deviates from the model training data distribution or there is a perception ambiguity, the model is prone to make undetectable decision errors, leading to navigation deviation. This error accumulates in long sequence decisions, ultimately causing the unmanned aerial vehicle to be trapped in a situation where it cannot complete the task, or triggering passive safety mechanisms such as "wall collision return", resulting in complete task failure.

[0006] Furthermore, existing systems have low integration with mature task planning software, making it difficult to achieve closed-loop control. Most current research VLN systems fail to deeply integrate with widely used task planning software in the industry (such as Mission Planner). This results in the system being unable to fully utilize the mature flight control, waypoint management, and data link communication capabilities already possessed by these software, and it is also difficult to form an effective closed loop between the high-level semantic decisions of VLN and the underlying, real-time flight attitude control, battery management, and safety protocols, thereby restricting its practicality and reliability in real tasks.

[0007] Therefore, there is an urgent need in the art for a technical solution capable of overcoming the above-mentioned defects. SUMMARY

[0008] In view of the above problems, the present application is proposed in order to provide a multi-dimensional thought chain-based unmanned aerial vehicle natural language multi-modal navigation method and system that overcomes the above problems or at least partially solves the above problems.

[0009] To achieve the above-mentioned purpose, the present application adopts the following technical solutions: In a first aspect, the present application embodiment provides a multi-dimensional thought chain-based unmanned aerial vehicle natural language multi-modal navigation method, the steps comprising: S1: constructing a software-in-the-loop simulation environment deeply integrated with a task planning software, communicating with the task planning software through a flight control protocol, obtaining real-time video streams in the simulation environment, and collecting real scene images with time and space stamps; S2: analyzing natural language task instructions to obtain a structured task sequence; at the same time, processing the real scene images through a visual language model to identify at least one key landmark and the coordinates and relative positions of the key landmark in the image; S3: inputting the identification results and the structured task sequence into a multi-dimensional thought chain reasoning engine for processing in parallel; the multi-dimensional thought chain reasoning engine at least includes an action generation chain, a history storage chain, a natural language reasoning chain, and a tool construction coordinator; S4: integrating the output of the multi-dimensional thought chain reasoning engine to generate a final executable flight action sequence, converting the action sequence into bottom-level control instructions recognizable by the task planning software through the flight control protocol, and issuing the instructions to the flight control system to control the unmanned aerial vehicle to complete the navigation task; at the same time, based on the results after the instructions are executed, synchronously updating the historical state and navigation context information in the multi-dimensional thought chain reasoning engine.

[0010] Preferably, in S1, the software-in-the-loop simulation environment deeply integrated with the task planning software comprises: starting and interconnecting a software-in-the-loop simulator, a task planner, and a three-dimensional visualization environment, wherein: the task planner is connected with a flight controller in the software-in-the-loop simulator through a TCP / IP protocol, for sending task instructions and receiving flight states; the flight controller sends control instructions to a physical simulation engine through a first UDP protocol; the physical simulation engine sends dynamic model information containing the position and attitude of the unmanned aerial vehicle to the three-dimensional visualization environment through a second UDP protocol; the three-dimensional visualization environment broadcasts the video stream rendered by it through a third UDP protocol; The video stream is captured to realize real scene image acquisition, and instructions based on the MAVLink protocol format are sent to the task planner or the flight controller to realize control of the UAV, thereby forming a deep integration closed loop system from perception, decision making to control.

[0011] Preferably, in S2, the natural language task instruction is parsed, including: The natural language instruction is parsed into a sequence containing a landmark field and a standard verb field based on a single-sample prompt through a preset text parser; the verbs in the standard verb field are taken from a predefined closed action set. The predefined closed action set is: {takeoff, land, goto, rtl, left, right, forward, back, up, down,hover}.

[0012] Preferably, in S2, the real scene image is processed through a visual language model, including: A multimodal large language model quantized by 4-bit is used to process the image through inter-layer parallel computing technology; a specific query prompt word is constructed to force the model to output results in the format of "landmark name-pixel coordinates-relative position".

[0013] Preferably, in S3, the action generation chain includes: The structured task sequence and key landmark information are received; High-level navigation maneuvers are forced to be decomposed into one or more atomic actions through prompt word instructions, and target point coordinates and atomic action sequences are output. The prompt word contains the following constraints: Instruction constraint: force high-level navigation instructions to be decomposed into atomic actions in the closed action set; Precision constraint: specifies that absolute geographic coordinates obtained from real-time observation of landmark information must be used when executing waypoint flight actions; Energy efficiency constraint: the output action sequence is associated with energy efficiency evaluation information containing task duration estimation and power consumption estimation.

[0014] Preferably, in S3, the history storage chain includes: The navigation history state is recorded in a lightweight structured format, which at least includes: Task timeline, used to record key nodes and states of the action sequence; Landmark access statistics, used to aggregate observation data of landmarks; Performance indicators, used to pre-calculate core indicators related to navigation energy efficiency; When a path deviation is detected, the history state is traced back to a previous successful landmark access node, and path re-planning is initiated.

[0015] Preferably, in S3, the natural language reasoning chain comprises: Receiving historical context information from the history storage chain, quantifying risk assessment and confidence score for the current decision; when the risk level exceeds a threshold or the confidence is lower than a threshold, generating one or more alternative action plans; each alternative plan includes specific actions, generated reasons and its risk level.

[0016] Preferably, in S3, the tool construction coordinator is: Receiving the outputs of the action generation chain, the history storage chain and the natural language reasoning chain, and performing decision consistency verification according to a preset integration rule set; the integration rule set at least includes: risk assessment alignment of action sequence and reasoning chain, historical context guiding current decision, and safety constraints being complied with; When a conflict is found in the verification, arbitration is performed according to the priority of the rule set, and a final unified action sequence is output.

[0017] In a second aspect, the embodiments of the present application provide a UAV natural language multi-modal navigation decision system based on multi-dimensional thinking chain, which applies the UAV natural language multi-modal navigation method based on multi-dimensional thinking chain as described above, and comprises: A simulation and environment perception module is configured to construct a simulation environment deeply integrated with a task planning software, communicate with the task planning software through a flight control protocol, obtain real-time video streams in the simulation environment, and collect real scene images with time and space stamps; A multi-modal information processing module is configured to analyze natural language instructions and identify key landmarks in images; A multi-dimensional thinking chain reasoning engine comprises a processor configured to execute the action generation chain, the history storage chain, the natural language reasoning chain and the tool construction coordinator; A system integration and control module is configured to generate a final control instruction, send the instruction to the task planning software through the flight control protocol, and then to a flight control system, so as to realize deep integration and closed-loop interaction with a simulation system or an actual flight control system.

[0018] In a third aspect, the embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, and the program is executed by a processor to realize the UAV natural language multi-modal navigation method based on multi-dimensional thinking chain as described above.

[0019] The above technical solution provided by the embodiment of the present application aims to embed an explicit thinking chain reasoning process into each step of navigation decision-making, and deeply integrate with the existing task planning, thereby significantly improving the autonomy, adaptability and task success rate of the unmanned aerial vehicle in unknown or dynamic environments.

[0020] Specific beneficial effects include at least: 1. The present application realizes high integration of task planning software and closed-loop control; that is, it can generate standardized bottom flight control instructions, directly embedded in the existing unmanned aerial vehicle control link, realizing a complete closed loop from perception, decision-making to control. 2. Through the cooperation of the text parser and the visual language model, semantic instructions can be converted into specific perception and action sequences for the physical environment, avoiding the dependence on accurate GPS coordinates or pre-programmed waypoints. 3. By constructing a multi-dimensional thinking chain reasoning engine composed of an action generation chain, a history storage chain, a natural language reasoning chain and a tool construction coordinator, complex navigation decision-making tasks are decomposed into professional sub-tasks such as safety constraints, state memory, risk assessment and consistency checking, eliminating the unpredictable risks and cumulative errors caused by the direct output of the end-to-end "black box" model in the prior art. At the same time, through the cooperation and balance between the chains, an internal decision-making safety redundancy is formed, ensuring the explainability, verifiability and interruptability of the whole process from high-level semantic instructions to bottom control instructions. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only a part of the present application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0022] Figure 1 It is a deep integration closed-loop system framework in the present application; Figure 2 It is a prompt word format diagram in the present application; Figure 3 It is an example flow diagram of the unmanned aerial vehicle natural language multi-modal navigation method based on multi-dimensional thinking chain in the present application. DETAILED DESCRIPTION

[0023] The technical solutions in the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0024] The embodiment of the application discloses a UAV natural language multi-modal navigation method based on a multi-dimensional thinking chain, and steps include: S1: a software-in-the-loop simulation environment deeply integrated with a task planning software is constructed, a flight control protocol is used to communicate with the task planning software, real-time video streams in the simulation environment are acquired, and real scene images with time and space stamps are collected; S2: a natural language task instruction is parsed to obtain a structured task sequence; at the same time, a visual language model is used to process the real scene images, at least one key landmark and coordinates and relative positions of the key landmark in the images are identified; S3: the identification results and the structured task sequence are input into a multi-dimensional thinking chain reasoning engine in parallel for processing; the multi-dimensional thinking chain reasoning engine at least includes an action generation chain, a history storage chain, a natural language reasoning chain and a tool construction coordinator; S4: outputs of the multi-dimensional thinking chain reasoning engine are integrated to generate a final executable flight action sequence, the action sequence is converted into bottom layer control instructions recognizable by the task planning software through the flight control protocol, and the control instructions are issued to a flight control system to control the UAV to complete a navigation task; at the same time, based on results after the instructions are executed, history states and navigation context information in the multi-dimensional thinking chain reasoning engine are synchronously updated.

[0025] After a UAV natural language task instruction is input, key landmarks and action items in the text instruction are identified, a simulation photographing function is started, a visual language model is input with a query to obtain coordinates of the key landmarks, further, the task instruction and perception information are input into a multi-dimensional thinking chain reasoning engine for decision making, an action sequence and a reasoning process are given, finally, the action sequence is input into an instruction distributor to obtain SITL level control instructions, and the control instructions are transmitted to a bottom layer controller through TCP, single step simulation is realized in a full software simulation, and single step actions of the next step are performed according to the instructions again until the UAV reaches a target point and the navigation is completed.

[0026] In an embodiment, step S1 is used to construct a simulation loop and collect real-time photos; steps include: a software-in-the-loop simulator, a task planner and a three-dimensional visualization environment are started and interconnected; the task planner is connected with a flight controller in the software-in-the-loop simulator through a TCP / IP protocol, and is used to send task instructions and receive flight states; the flight controller sends control instructions to a physical simulation engine through a first UDP protocol; The physical simulation engine sends the dynamic model information containing the position and attitude of the UAV to the three-dimensional visualization environment through a second UDP protocol; The three-dimensional visualization environment broadcasts its rendered video stream through a third UDP protocol; By capturing the video stream to realize real scene image acquisition, and sending instructions based on the MAVLink protocol format to the task planner or the flight controller to realize control of the UAV, a deep integration closed-loop system from perception, decision-making to control is formed.

[0027] In some embodiments, the python script sends instructions through MAVLink to realize photo acquisition. In the terminal, the SITL, the planning module Mission Planner, and the navigation module FlightGear 3D process are started to establish a simulation loop through UDP and TCP communication. As shown in Figure 1 The Video Streaming of FlightGear encodes the picture in real time and broadcasts each frame of MJPEG through the UDP protocol to the local loop address and the specified port. Python uses cv2.VideoCapture to pull the stream and synchronously capture the RGB real scene picture with latitude-longitude-time stamp into the database to realize photo acquisition.

[0028] In one embodiment, step S2 includes: Through a preset text parser, a natural language instruction is structured and parsed based on a single sample prompt into a sequence containing a landmark field and a standard verb field; A multi-modal large language model quantized by 4-bit is used to process images using inter-layer parallel computing technology; by constructing a specific query prompt word, the model is forced to output results in the format of "landmark name-pixel coordinates-relative position".

[0029] In some specific embodiments, the text parser is used to process the task instruction to obtain a key landmark and an action text segment, and a query is set. The molmo model is used to obtain the coordinate value of the key landmark and the relative position in the photo according to the query and the collected photo.

[0030] Wherein, the text parser solidifies Role / Goals / Workflow with one-shot structured prompt, so that the large model can decompose any single natural language instruction into a "landmark + standard verb" JSON sequence at one time; the verb only takes {takeoff, land, goto, rtl, left, right, forward, back, up, down, hover}. The text parser upgrades the traditional ground station waypoint navigation to natural language instruction navigation, covers the entire set of civil aviation ICAO phrase actions with a standard verb sequence, ensures that common unmanned aerial vehicle navigation natural language instructions can be parsed, and the landmark field corresponds to the "geographical reference point" in DO-178C.

[0031] Further, Molmo-7B-O is automatically invoked with 4-bit quantization to calculate GPU resources, and then HF-accelerate is used for inter-layer parallelism; the processor splices the image chunk Token and the target word bag in the query into a causal sequence, and generates a coordinate text with fixed labels (max 2k tokens, stop at <|endoftext|>) from batch autoregression, outputs an XML template in the format of Landmark∶xxx / Coordinates∶[x,y] / Position: xxx. The edited query forces the model to report in three steps of "landmark-coordinate-relative position", which corresponds to the "prominent terrain-relative positioning" principle recommended by civil aviation ICAO, and the output format is unified as the upper left pixel origin, coordinate pairs, and relative position, which can be input to the navigation agent for visual flight backup, realizing low-altitude navigation perception of unmanned aerial vehicles.

[0032] In one embodiment, step S3: build a COT multi-dimensional deduction chain: The COT deduction chain is established through the prompt, the results processed by the front text parser module and the molmo model are input, and the inference process of the At sequence and the sequence obtained by the LLM is output.

[0033] In this embodiment, after obtaining the pixel coordinates of the key landmarks recognized by the molmo model, coordinate transformation is needed to convert the pixel coordinates into longitude and latitude coordinates in the WGS-84 coordinate system. That is, first, the pixel coordinates are normalized using the pre-calibrated camera intrinsic parameters and distortion coefficients, and then the normalized coordinates are multiplied by a rotation matrix to convert the coordinate vector from the camera's own coordinate system to the UAV body coordinate system. Then, the real-time attitude data provided by the UAV IMU, including roll angle, pitch angle and yaw angle, are used. Through these Euler angles, a rotation matrix R_body_to_ned from the body coordinate system to the North-East-Down coordinate system is constructed. The direction vector V_body in the body coordinate system is converted to V_ned in the NED coordinate system. Finally, the accurate longitude and latitude of the UAV itself in the WGS-84 coordinate system are obtained from the GNSS receiver of the UAV, and the NED coordinates of the target point are obtained through the proportional relationship, and the NED coordinates are converted back to the final WGS-84 longitude and latitude coordinates through the Earth ellipsoid model.

[0034] Further, after receiving the structured instructions and the longitude and latitude coordinates of the key landmarks, the instruction-observation landmark name mapping is performed. The _run_action_chain function first reads the instructions of the current step in mission_instructions to form the "action request" of this task. The landmark name is extracted through step.get("landmark"), and if it is empty, the current position of the UAV is used by default; otherwise, the real WGS-84 coordinates are found in the observed_landmarks dictionary with the lowercase key name, realizing the mapping of the instruction string to the actual coordinates. Further, the coordinate conversion function is used to convert the pixel coordinates to longitude and latitude coordinates, ensuring that the subsequent decision is based on real geographic data. The mapped coordinates, current height, battery status, etc. are packaged into the observed_real dictionary, and are sent into the action_chain model together with the task history. The model determines whether to take off, goto, left-right translation or land according to the real observations and historical performance, and outputs the MAVLink level action JSON with parameters.

[0035] For UAV navigation tasks, the COT thought chain uses 4 LangChain sub-chains to string "language-perception-state-history" into a closed loop for hierarchical decision-making and collaborative reasoning, realizing the functions of memory, decision-making, judgment and backtracking.

[0036] The action generation chain generates target points and their longitude and latitude coordinates in real time with WGS-84 landmark-coordinate mapping, and JSON action sequences (selected from {takeoff, land, goto, rtl, left, right, forward, back, up, down, hover}); the historical storage chain records the navigation history in Markdown-JSON format, and the UAV can accurately return to the previous key landmark and the state of the UAV when it deviates, thereby realizing the historical backtracking function; the LLM inference chain enforces the "instruction-landmark-energy efficiency-history-rules-safety" six-step interpretable output, giving confidence and risk alternatives; the tool construction coordinator weights and disambiguates, and outputs a unique At sequence.

[0037] In some embodiments, the action generation chain includes: receiving the structured task sequence and key landmark information; The high-level navigation maneuver is forced to be decomposed into one or more atomic actions by the prompt word instruction, and the target point coordinates and the atomic action sequence are output. The prompt word format is as shown in the following formula: Figure 2 and contains the following constraints: instruction constraint: force the high-level navigation instruction to be decomposed into atomic actions in a closed action set; precision constraint: when executing the waypoint flying action, the absolute geographic coordinates obtained from real-time observation of landmark information must be used; energy efficiency constraint: the output action sequence is associated with energy efficiency evaluation information containing task duration estimation and power consumption estimation.

[0038] In the field of UAV navigation, the selection of flight actions needs to consider the dynamics adaptation and safety issues of UAV, the core of which is to avoid power failure, communication interruption, positioning loss and collision with the environment. Therefore, in the task description part of the prompt words, the action set is closed by "You must only output actions from the following set:" and the innovative verbs are specially pointed out, and the constructed action set {takeoff, land, goto, rtl, left, right, forward, back, up, down, hover} conforms to the action decomposition of the flight characteristics of the UAV, which can realize the dynamics adaptation of the vertical take-off, three-dimensional navigation, hovering stability and lateral maneuver of the UAV. At the same time, through "Any high-level maneuver must be decomposed into one or more of the above primitives.", the large model is prompted to use the safe decomposition mechanism to decompose high-level instructions (such as "reconnaissance on the roof of a certain building") into sub-operations in the action space, and through "Do not introduce new verbs like ascend, up, down, etc.", the large model is restricted from generating actions outside the action set to prevent the introduction of non-standard verbs. Each step can be independently monitored and terminated, and even if a single action fails, the system can still generate an RTL high-level instruction to back up to a safe state through the thought chain.

[0039] Further, in the UAV visual language navigation, guaranteeing navigation accuracy and real-time performance is a core challenge, which usually requires the system to perform actions based on high-confidence real-time environment perception and strictly avoid cumulative errors caused by coordinate calculation or instruction ambiguity. Therefore, in the information input part of the prompt words, input ##CURRENT OBSERVED LANDMARKS(Name ->Lat / Lon / Alt):{observed_landmarks} always synchronizes the real-time observation of the latitude, longitude and altitude coordinates of the landmarks, and in the task description part of the prompt words, the CRITICAL level requirement "When using 'goto' action, you MUST use the actual WGS-84 coordinates from OBSERVED_LANDMARKS." restricts the large model to use the actual WGS-84 coordinates from "OBSERVED_LANDMARKS" for the "goto" action, thereby ensuring that the coordinates used for navigation have real-time performance, and directly using WGS-84 coordinates for navigation can prevent errors caused by relative coordinate calculation algorithms.

[0040] In the navigation task planning of the UAV, the navigation energy efficiency is a core performance index. The total task duration, battery consumption estimation, path efficiency, and other indexes are usually used to evaluate the navigation energy efficiency. The goal is to optimize the flight time and energy consumption under the premise of ensuring task success. In the output format template of the prompt word, "expected_duration": 30.5, # task time estimation"battery_consumption": 2.5, # battery consumption prediction" are used to limit the large model to think and reason under the energy efficiency constraint, guide the model to select more efficient path and action combination when planning, and perform system-level energy efficiency monitoring to ensure the safety of the entire UAV navigation process.

[0041] In some embodiments, the history storage chain includes: The navigation history state is recorded in a lightweight structured format, which includes at least: A mission timeline for recording key nodes and states of the action sequence; Landmark visit statistics for aggregating observation data on landmarks; Performance metrics for pre-computing core indexes related to navigation energy efficiency; When a path deviation is detected, the history state is traced back to the previous successful landmark visit node, and path re-planning is started.

[0042] In this embodiment, the history storage chain history_chain prompt word can achieve lightweight management of UAV navigation state data and realize the traceability of UAV navigation.

[0043] Due to the flight dynamics limitations of the UAV, the on-board weight is required to be as light as possible, and the computing power and storage resources of the on-board computer are limited. Therefore, the system must strictly manage the history navigation state, retain key decision information, and avoid storage overflow caused by accumulation of raw data.

[0044] Therefore, in the output format template of the prompt word, "historical_storage": { "mission_timeline": [], / / only structured key data is retained "landmark_visits": {}, / / aggregated statistics to avoid accumulation of raw data "performance_metrics": {} / / pre-computed indexes to reduce real-time computation load } The historical information is compressed into three dimensions: the key nodes and states of the action sequence are retained in the mission_timeline, the original observation data is processed in the landmark_visits, and the statistical features of landmark visits are aggregated, and the core performance indicators are pre-calculated in the performance_metrics to facilitate the navigation path planning reasoning of the large model based on energy efficiency constraints. This design ensures the lightweight storage while covering the three complete dimensions of task timeline, spatial behavior pattern and system performance, maintaining the structural integrity of the historical navigation state information.

[0045] During actual navigation of the UAV, due to fluctuations in GPS signals, visual positioning drift, or the appearance of sudden obstacles such as birds or temporary buildings, the navigation path often deviates or the task is interrupted, at which time accurate navigation state backtracking must be relied on to locate the problem node and initiate re-planning. The prompt word accurately records the timestamp of each action, the state of the UAV (position, power) and the execution result through the mission_timeline module. When detecting path deviation or encountering obstacles, the system can immediately backtrack to the last successful landmark visit node (through the visit record in landmark_visits), and quickly determine the deviation source based on pattern recognition in historical_analysis (such as "vulnerable to GPS interference in a particular area"). This structured historical storage not only provides a benchmark state for re-planning, but also identifies high-success-rate path features through performance_metrics, thereby generating alternative paths that avoid obstacle areas or have stronger anti-interference capabilities after backtracking.

[0046] In some embodiments, the natural language reasoning chain includes: Receiving historical context information from the historical storage chain, quantifying risk assessment and confidence score for the current decision; when the risk level exceeds the threshold or the confidence is lower than the threshold, generating one or more alternative action plans; each alternative plan includes specific actions, generated reasons and risk levels.

[0047] The natural language reasoning chain LLM_chain prompt word in this application realizes multi-dimensional risk assessment and safety redundancy mechanism, as well as reasoning-decision-history deep integration, and can realize the experiential decision and continuous evolution navigation of the UAV.

[0048] In the field of aviation, the redundancy design principle is the core principle to ensure flight safety, which requires that key systems and decision-making processes must be equipped with multiple backups to ensure that the failure of a single component or decision does not result in systemic catastrophic consequences. The output format template part of the historical storage chain prompt word { "confidence_score": 0.92, / / Confidence quantification "risk_assessment": "low", / / Risk level classification "alternative_options": [] / / Alternative options } Guide the large model to give the confidence score of the current navigation scheme, risk level assessment, and generate alternative options based on confidence and risk level. Specifically, by "alternative_options": [ { "action": "alternative_action", "rationale": "why this alternative", "risk_level": "medium" } ] Guide the large model to give the specific alternative action in the alternative options, generate the reasoning process of the alternative action, and the risk level of the alternative action. This way, when the UAV navigation feedbacks the risk, the alternative options can be used in time to ensure the safety of the UAV flight.

[0049] Meanwhile, the hint word is closely integrated with the historical storage chain through the {historical_context} parameter, forming a closed loop of reasoning-decision-making-history, realizing experience-based decision-making, and enabling the intelligent navigation system to avoid risks, optimize energy efficiency, and continuously evolve. When facing similar situations, the reasoning chain can call on these experiences through {historical_context}. For example, if historical data shows that path deviation has occurred multiple times in "Region A" due to weak GPS signals, when the UAV approaches "Region A" again, the reasoning chain will mark the risk level as "medium" or "high" in risk_assessment in advance, and may automatically generate alternative_options for detour, rather than blindly rushing in. The average energy consumption, task efficiency, and other data stored in performance_metrics provide direct evidence for the energy efficiency constraints of the reasoning chain. For example, when choosing between two alternative paths, the reasoning chain discovers through historical data that although path A is shorter, it is against the wind, and the average energy consumption is 20% higher than path B, so it will preferentially recommend path B and explain in reasoning_chain that it is based on energy efficiency optimization. The results of each reasoning and decision-making (whether successful or not) are recorded in the historical chain in a structured manner, updating historical_analysis and performance_metrics, thereby optimizing the starting point of reasoning for the next task. For example, a UAV discovers a new shortcut to the target through exploration, and this successful experience is recorded and shared with other UAVs in the cluster, thereby improving the overall task efficiency of the fleet.

[0050] In some embodiments, the tool chain coordinator is: receiving the outputs of the action generation chain, the historical storage chain, and the natural language reasoning chain, and performing decision consistency verification according to a preset integration rule set; the integration rule set at least includes: alignment of action sequence and risk assessment of reasoning chain, historical context guiding current decision, and safety constraints being followed; when a conflict is found in the verification, arbitrating according to the priority of the rule set, and outputting a final unified action sequence.

[0051] In this embodiment, the tool chain hint word can realize UAV navigation decision consistency and ensure that the system ultimately obtains a unified and reliable decision.

[0052] In practical applications of UAV navigation, the system faces multiple challenges, including asynchronous decision-making across multiple modules, sensor information conflicts, strictly limited resources, and compliance with aviation safety regulations. This necessitates that the navigation system possess decision consistency and real-time reliability, ensuring the generation of coordinated, unified, and safe final commands even under multi-source information input. The output format template within the prompt words guides the large model in verifying consistency across the entire decision-making chain. { "consistency_check": "passed", "validation_results": [ "Validation point 1: passed", "Validation point 2: passed" ] } Through consistency checks, the coordinator ensures the logical coherence of the entire decision-making chain from perception and reasoning to execution, avoiding system oscillations or dangerous actions caused by inconsistent outputs between modules.

[0053] Meanwhile, the task description section details the task rules: "INTEGRATION RULES": { "1": "The action sequence must be aligned with the inference chain", "2": "Historical context should guide current decision-making", "3": "Confidence scores must be consistent in the output", 4: "Safety constraints must be followed". 5: "Task objectives should be given priority." } This rule set establishes a decision-priority framework. When there are contradictions in the outputs of different chains (such as the action chain suggesting to move forward while the reasoning chain identifies risks), the system can analyze according to preset aviation safety criteria to ensure that the final output meets the "safety first" requirements of the UAV navigation industry.

[0054] In one embodiment, step S4: integrate the output of the multidimensional thinking chain inference engine to generate the final executable flight action sequence, and convert the action sequence into low-level control commands that the mission planning software can recognize through the flight control protocol, and send them to the flight control system to control the UAV to complete the navigation task; at the same time, based on the result after the command execution, synchronously update the historical state and navigation context information in the multidimensional thinking chain inference engine.

[0055] In this embodiment, the high-level action sequence is converted into low-level control instructions executable by the underlying controller ArduPilot through coordinate conversion and instruction mapping. An instruction dispatcher is constructed by conditional judgment structure in the code to parse the type (ACTION) and associated parameters (LANDMARK_COORDINATE and ALTITUDE) of each action.

[0056] As shown in Table 1, action parsing maps the high-level action sequence (such as "takeoff", "goto", "land" / "rtl", "left", "right", "forward", "back") output by the COT chain into specific control instructions for API calls of the UAV; Table 1

[0057] Among them, the control instruction corresponding to the goto action adopts the WGS-84 longitude and latitude coordinate system, and according to the longitude and latitude of the target point specified in the action plan, the LocationGlobalRelative function in the dronekit library is used to convert it into relative navigation coordinates, and then the simple_goto method is used to navigate the UAV to the target point.

[0058] send_ned_velocity is a UAV control function used to send local NED (North-East-Down) coordinate system velocity instructions to the flight control for a specified duration. This function enables only the velocity field (vx, vy, vz) through the MAVLink SET_POSITION_TARGET_LOCAL_NED message, controls the UAV to fly at the set speed vector without changing its attitude or acceleration. The function first constructs and sends the MAVLink message, and then waits for the specified duration to make the UAV maintain the speed motion.

[0059] In some embodiments, the navigation method flow of the present application refers to Figure 3 , the steps include: Simulation loop establishment and photo collection: Establish a SITL+MP simulation loop, and a python script sends instructions through MAVLink to realize photo collection; Multi-modal information processing: Construct a text parser to process task instructions, and use the molmo model to obtain the pixel coordinates and relative positions of key landmarks from the query and the collected photos; COT inference chain: Write prompts to establish a COT inference chain, input landmark longitude and latitude, and processed r instructions, and the LLM outputs natural language reasoning chain Rt, action sequence At, and history storage Ht; System interaction and integration: further convert the output action sequence into instructions in MP, realize the integration of intelligent navigation system, and use the simulation loop of SITL+MP for single-step interaction and simulation; step iteration until the target point is reached.

[0060] Further, in an embodiment, the present disclosure provides a UAV natural language multi-modal navigation decision system based on a multi-dimensional thought chain, which applies the UAV natural language multi-modal navigation method based on a multi-dimensional thought chain as described above, and comprises: A simulation and environment perception module is configured to construct a simulation environment deeply integrated with a task planning software, communicate with the task planning software through a flight control protocol, obtain real-time video streams in the simulation environment, and collect real scene images with time and space stamps; A multi-modal information processing module is configured to analyze natural language instructions and identify key landmarks in images; A multi-dimensional thought chain reasoning engine comprises a processor configured to execute the action generation chain, the history storage chain, the natural language reasoning chain, and the tool construction coordinator; A system integration and control module is configured to generate final control instructions, send the instructions to the task planning software through the flight control protocol, and then to a flight control system, thereby realizing deep integration and closed-loop interaction with the simulation system or the actual flight control system.

[0061] The integration between different modules in the present application needs to be limited by data formats according to the actual situation of UAV navigation. In some embodiments, the data formats are as shown in Table 2: Table 2

[0062] The action sequence and the reasoning process are as follows:

[0063] Based on the same inventive concept, the present application further provides a computer readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to realize the UAV natural language multi-modal navigation method based on a multi-dimensional thought chain as described above. Therefore, the implementation of the program can refer to the implementation of the aforementioned method, and the repeated parts will not be described again.

[0064] In the present specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0065] The foregoing description of the disclosed embodiments enables a person skilled in the art to make or use the application. Modifications of these embodiments will occur to persons of skill in the art, and that the appended claims are intended to cover all such modifications that do not depart from the true spirit and scope of the application. Therefore, the application is not limited to the embodiments shown but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for unmanned aerial vehicle (UAV) natural language multimodal navigation based on multi-dimensional thought chain, characterized in that, Comprise: S1: build a software-in-the-loop simulation environment deeply integrated with task planning software, communicate with the task planning software through flight control protocol, obtain real-time video stream in the simulation environment, and collect real scene images with time and space stamps; S2: analyze natural language task instructions to obtain a structured task sequence; at the same time, process the real scene images through a visual language model to identify at least one key landmark and the coordinates and relative position of the key landmark in the image; S3: input the identification results and the structured task sequence into a multi-dimensional thought chain reasoning engine for processing; the multi-dimensional thought chain reasoning engine includes at least an action generation chain, a history storage chain, a natural language reasoning chain, and a tool construction coordinator; S4: integrate the output of the multi-dimensional thought chain reasoning engine, generate a flight action sequence, convert the action sequence into a bottom-level control instruction recognizable by the task planning software, and issue the instruction to the flight control system to control the UAV to complete the navigation task; at the same time, based on the results after the instruction is executed, the history state and navigation context information in the multi-dimensional thought chain reasoning engine are updated synchronously. 2.The unmanned aerial vehicle natural language multimodal navigation method of claim 1, wherein, In S1, a software-in-the-loop simulation environment deeply integrated with a task planning software is built, comprising: Start and interconnect the software-in-the-loop simulator, the task planner, and the three-dimensional visualization environment, wherein: The task planner is connected with the flight controller in the software-in-the-loop simulator through TCP / IP protocol, used for sending task instructions and receiving flight states; The flight controller sends control instructions to the physical simulation engine through the first UDP protocol; The physical simulation engine sends dynamic model information containing the position and attitude of the UAV to the three-dimensional visualization environment through the second UDP protocol; The three-dimensional visualization environment broadcasts the video stream rendered by it through the third UDP protocol; By capturing the video stream to realize real scene image acquisition, and by sending instructions based on MAVLink protocol format to the task planner or the flight controller, the UAV is controlled, forming a deep integration closed-loop system from perception, decision-making to control. 3.The unmanned aerial vehicle natural language multimodal navigation method of claim 1, wherein, In S2, the natural language task instructions are parsed, comprising: Through a pre-set text parser, the natural language instructions are structured and parsed into a sequence containing landmark fields and standard verb fields based on single-sample prompts; wherein the verbs of the standard verb field are taken from a pre-defined closed action set. 4.The unmanned aerial vehicle natural language multimodal navigation method of claim 1, wherein, In S2, the real scene images are processed through a visual language model, comprising: A multi-modal large language model quantized by 4-bit is used to process the images through inter-layer parallel computing technology; by constructing specific query prompt words, the model is forced to output results in the format of "landmark name-pixel coordinates-relative position". 5.The unmanned aerial vehicle natural language multimodal navigation method of claim 1, wherein, In S3, the action generation chain comprises: Receive the structured task sequence and key landmark information; Through prompt word instructions, high-level navigation maneuvers are forced to be decomposed into one or more atomic actions, and target point coordinates and atomic action sequences are output; wherein the prompt words contain the following constraints: Instruction constraint: force to decompose high-level navigation instructions into atomic actions in the closed action set; Precision constraint: specifies that absolute geographic coordinates obtained from real-time landmark observation information must be used when performing waypoint flight maneuvers; Energy efficiency constraint: the output action sequence is associated with energy efficiency evaluation information containing task duration estimation and power consumption estimation. 6.The unmanned aerial vehicle natural language multimodal navigation method of claim 1, wherein, In S3, the history storage chain includes: The navigation history state is recorded in a lightweight structured format, which at least includes: Task timeline, used to record key nodes and states of the action sequence; Landmark visit statistics, used to aggregate observation data on landmarks; Performance indicators, used to pre-calculate core indicators related to navigation energy efficiency; When path deviation is detected, the navigation history state is traced back to the previous successful landmark visit node, and path re-planning is initiated. 7.The unmanned aerial vehicle natural language multimodal navigation method of claim 1, wherein, In S3, the natural language reasoning chain includes: Receive historical context information from the history storage chain, and perform quantitative risk assessment and confidence scoring on the current decision; when the risk level exceeds the threshold or the confidence is lower than the threshold, one or more alternative action plans are generated; each alternative plan includes specific actions, generation reasons, and risk levels. 8.The unmanned aerial vehicle natural language multimodal navigation method of claim 1, wherein, In S3, the tool construction coordinator is: Receive the outputs of the action generation chain, history storage chain, and natural language reasoning chain, and perform decision consistency verification according to the preset integration rule set; The integration rule set at least includes: alignment of action sequence and risk assessment of reasoning chain, historical context guiding current decision, and safety constraints being followed; When conflicts are found during verification, arbitration is performed according to the priority of the integration rule set, and the final unified action sequence is output.

9. A multi-dimensional thought chain based unmanned aerial vehicle natural language multi-modal navigation decision system, characterized in that, The multi-dimensional thinking chain-based unmanned aerial vehicle natural language multi-modal navigation method according to any one of claims 1-8, the system comprises: A simulation and environment perception module for building a simulation environment deeply integrated with a task planning software, and communicating with the task planning software through a flight control protocol to obtain real-time video streams in the simulation environment and collect real scene images with time and space stamps; A multi-modal information processing module for analyzing natural language instructions and identifying key landmarks in images; A multi-dimensional thinking chain reasoning engine including a processor for executing the action generation chain, history storage chain, natural language reasoning chain, and tool construction coordinator; A system integration and control module for generating final control instructions and sending the instructions to the task planning software through the flight control protocol, and then to the flight control system, realizing deep integration and closed-loop interaction with the simulation system or actual flight control system.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the multi-dimensional thinking chain-based unmanned aerial vehicle natural language multi-modal navigation method according to any one of claims 1-8.

Images