Unmanned aerial vehicle intelligent task planning Agent system and method based on large language model

Abstract

The invention belongs to the technical field of unmanned aerial vehicles, and discloses an unmanned aerial vehicle intelligent task planning Agent system and method based on a large language model, and the system comprises a user interaction module, a large language model reasoning module, a tool calling module, a safety management module, a state sensing module and a memory module. The large language model reasoning module adopts a structured output parser to parse model output into structured data; the tool calling module comprises takeoff, landing, waypoint flight, state acquisition and other tool functions; the safety management module defines constraint parameters such as the minimum safe flight height, the maximum safe flight height and the single-time maximum moving distance. According to the method, intelligent task planning is realized through a closed-loop process of receiving a natural language instruction of a user, constructing a prompt template, calling a large language model for reasoning, performing safety verification, executing a tool function and feeding back observation information. The system and method improve the stability and reliability of the system and the success rate of complex task processing.

Description

Technical Field

[0001] This application belongs to the field of unmanned aerial vehicle (UAV) technology, specifically relating to an intelligent mission planning agent system and method for UAVs based on a large language model. Background Technology

[0002] In recent years, unmanned aerial vehicles (UAVs) have experienced rapid development in military, industrial, and civilian fields. UAV mission planning, as a core branch of UAV technology, has been widely applied in inspection, logistics, rescue, and agricultural monitoring. UAV mission planning involves multiple technologies, including environmental perception, path search, trajectory optimization, motion control, and human-machine interaction. Different technological approaches need to be selected for different mission scenarios and application requirements.

[0003] Traditional UAV mission planning systems primarily rely on pre-programmed rules and algorithms, requiring a high level of expertise from operators and lacking system flexibility. Existing UAV control methods mainly include remote controller operation, ground station software configuration, and pre-set flight path execution. These methods suffer from the following technical problems:

[0004] (1) The operation is complex and users need to be familiar with the professional control interface and parameter configuration.

[0005] (2) Poor task adaptability, making it difficult to make dynamic adjustments based on real-time environmental changes.

[0006] (3) The human-computer interaction method is limited and cannot understand task descriptions in natural language form.

[0007] (4) Lack of independent reasoning ability, and need to manually configure each step of complex multi-step tasks.

[0008] Therefore, there is an urgent need for a new method to combine large language models with UAV systems to improve the intelligence level of UAVs, enabling them to understand natural language commands, make autonomous reasoning decisions, and dynamically adjust their tasks. Summary of the Invention

[0009] To address the aforementioned technical problems in the existing technology, the purpose of this invention is to combine a large language model with an unmanned aerial vehicle (UAV) system to enhance the intelligence level of the UAV, enabling it to understand natural language commands, make autonomous reasoning decisions, and dynamically adjust its tasks. The technical solution is as follows:

[0010] A drone intelligent mission planning agent system based on a large language model includes the following modules:

[0011] (1) User interaction module: Receives natural language task instructions from users and feeds back the task execution results to users;

[0012] (2) Large Language Model Reasoning Module: Parses the user's natural language instructions, performs thought chain reasoning, and generates structured action decisions;

[0013] (3) Tool Invocation Module: Executes specific UAV control operations;

[0014] (4) Safety Management Module: Performs parameter verification and safety constraint checks on the action instructions generated by the large language model;

[0015] (5) Status perception module: Real-time acquisition of the UAV's position, attitude, and flight status information;

[0016] (6) Memory module: Stores the most recent K rounds of historical dialogue and task execution records, supporting context understanding and task continuity.

[0017] Furthermore, the user interaction module has a log information publishing function, which publishes the Agent's thought process, execution actions, and observation results in a structured JSON format, allowing users to monitor and debug.

[0018] Furthermore, the log information includes the following fields: role identifier, message type, and specific content.

[0019] Furthermore, the large language model inference module employs a structured outputter to parse the output of the large language model into structured data containing the following fields:

[0020] (1) The thought field records the reasoning process for analyzing the current state and task progress;

[0021] (2) Action name field action_name: Specifies the name of the utility function to be called;

[0022] (3) Action parameter field action_params: Specifies the input parameters of the utility function in key-value pairs;

[0023] (4) The final message field final_message: Returns the execution result to the user when the task is completed.

[0024] Furthermore, the tool invocation module includes the following tool functions:

[0025] (1) Take-off function: controls the UAV to take off to the specified altitude, the parameter is the target altitude (height);

[0026] (2) The landing tool function `land` controls the safe landing of the UAV;

[0027] (3) The waypoint flight tool function fly_to_waypoint controls the UAV to fly to the specified three-dimensional coordinate position. The parameters are the target coordinates x, y, z.

[0028] (4) The get_status utility function is used to obtain the current position, attitude and flight status information of the UAV.

[0029] Furthermore, the state awareness module obtains the UAV's location information, attitude information, and flight status through ROS topic subscription or direct API calls.

[0030] Furthermore, the memory module adopts a dialogue buffer window mechanism to store the dialogue history of the last 5 rounds; its data structure includes: user message queue, assistant message queue and system observation queue.

[0031] A drone intelligent task planning agent method based on a large language model includes the following steps:

[0032] Step 1: Receive user task instructions, add the instructions to the dialogue history, construct a prompt template, and assemble the current drone status, description of available tools, output format requirements, and dialogue history into a complete prompt;

[0033] Step 2: Use a large language model for reasoning. Parse the model output using a structured output parser to obtain the parsing result. Compare the parsing result with the preset error flag. If the parsing result equals the preset error flag, the parsing is considered to have failed, a format error message is added, and the process is returned to the large language model for re-reasoning. If the parsing result does not equal the preset error flag, the parsing is considered to have succeeded, and the thought process, action name, and action parameters are obtained. The thought process and action record are added to the dialogue history, and a log report is generated.

[0034] Step 3: Determine the action type, obtain the action name, and compare the action name with the preset terminator "Final". If the action name is equal to "Final", output a task completion signal, determine that the task is completed, and return the final message to the user; if the action name is not equal to "Final", continue to execute the subsequent process.

[0035] Step 4: Find the corresponding utility function and obtain the search results. Compare the search results with the preset null value flag. If the search result is null, it is determined that the corresponding utility function was not found, and an error observation message is generated; if the search result is not null, it is determined that the corresponding utility function was found, and the subsequent process continues.

[0036] Step 5: Perform security verification on the action parameters using the tool function to obtain the action parameter values. Compare the action parameter values ​​with a preset pass threshold. If the action parameter value is lower than the pass threshold, the verification fails, and security interception observation information is generated. If the action parameter value is not lower than the pass threshold, the verification passes, the tool function is executed, and the observation results are obtained. The observation results are added to the dialogue history and the observation log is published. Return to Step 2 to continue the loop reasoning until the task is completed or the maximum number of iterations is reached.

[0037] Furthermore, in step 1, the prompt template includes:

[0038] (1) System role setting: Define Agent as intelligent drone controller, and clarify that it performs logical reasoning based on user instructions and drone status and calls tools to complete tasks;

[0039] (2) Current state description: includes the three-dimensional coordinates of the UAV. Yaw angle and flight status;

[0040] (3) Tool list description: Describe in detail the function, parameter format and use cases of each available tool;

[0041] (4) Execution rule description: including spatial reasoning rules, self-correction rules and task termination rules;

[0042] (5) Output format instructions: Use the format specification generated by the structured output parser to require the model to be output in JSON format;

[0043] The drone's status is acquired by automatically switching positioning modes based on GPS signal strength. GPS positioning is used, and when the signal is weak, it seamlessly switches to visual positioning, synchronizing the positioning data to a large language model. The process of achieving accurate positioning through visual positioning is as follows:

[0044] (1) Calculate the binocular visual depth by calculating the relative depth of the target through the difference between the left and right views. The calculation formula is as follows:

[0045]

[0046] in, Indicates the target's depth relative to the drone. Indicates the focal length of a binocular camera. Indicates the baseline distance of the binocular cameras. The x-coordinate of the left view, Represents the x-coordinate of the right view;

[0047] (2) Calculate the absolute coordinates of the ORB feature points. Using the world coordinates and relative displacement of the feature points, calculate the absolute position of the UAV. The calculation expression is as follows:

[0048]

[0049] in, Represents the absolute three-dimensional coordinates of the drone. This indicates the world coordinates of ORB feature points used for environment matching. This represents the three-dimensional displacement vector of the feature point relative to the UAV.

[0050] (3) Positioning accuracy compensation, which corrects the error through the camera distortion coefficient, is calculated as follows:

[0051]

[0052] in, This indicates the final corrected coordinates. Indicates the initial calculated coordinates. and Indicates the camera distortion coefficient. This represents the radial distance of the feature point image.

[0053] Furthermore, in step 5, the security verification includes the following constraint parameters:

[0054] (1) Set the minimum flight altitude to 1.0 meter;

[0055] (2) Set the maximum flight altitude to 15.0 meters;

[0056] (3) Set the maximum single movement distance to 20.0 meters;

[0057] The specific security verification process is as follows:

[0058] (1) For takeoff command, extract target altitude parameters to obtain target altitude value, compare target altitude value with preset maximum and minimum safe altitude values. When target altitude value is lower than minimum safe altitude or greater than maximum safe altitude, it is determined that safety verification fails and safety interception information is returned; when target altitude value is higher than minimum safe altitude and lower than maximum safe altitude, it is determined that safety verification passes, the Boolean value is_safe is returned, and the verification result description is returned.

[0059] (2) For waypoint flight commands, first determine if the target altitude falls within the safe altitude constraint range, then calculate the horizontal distance between the current position and the target position. The calculation expression is as follows:

[0060]

[0061] in, and Indicates the direction of the target and Axis coordinates and Indicates the current and Axis coordinates;

[0062] The horizontal distance is compared with a preset maximum single movement distance. If the horizontal distance is greater than the preset maximum single movement distance, the security check fails, an interception message is returned, and the user is advised to perform the operation step by step. If the horizontal distance is not greater than the preset maximum single movement distance, the security check passes, a boolean value is_safe is returned, and a description of the check result is returned.

[0063] The loop reasoning process employs a task interruption mechanism. When a new instruction is input during task execution, the system sets an interruption flag, acquires the new instruction, and adds it to the dialogue history in the format of "user interrupted and input new instruction". In the next round of reasoning, the system adjusts the task execution strategy based on the updated context to understand the changes in user intent.

[0064] The maximum number of iterations is set to 15.

[0065] Beneficial effects: (1) Using a large language model as the decision-making core, it can understand and execute complex task instructions in natural language form, greatly reducing the professional threshold for UAV operation and improving the convenience of human-computer interaction. (2) Using a structured output parsing mechanism, the free text output of the large language model is converted into standardized action instructions, improving the stability and reliability of the system. (3) Through a multi-layered safety management mechanism, dangerous operations caused by model illusion or erroneous reasoning are effectively prevented. (4) Using a memory module and dialogue history management mechanism, the Agent has contextual understanding capabilities and can handle multi-turn dialogues and continuous tasks. (5) Supporting task interruption and dynamic adjustment, it can respond to real-time changes in user instructions and has good flexibility and adaptability. (6) Using a thought chain reasoning mechanism, the Agent can perform step-by-step reasoning and self-correction, improving the success rate of complex task processing. Attached Figure Description

[0066] Figure 1 This is a schematic diagram of the overall architecture of the UAV intelligent mission planning agent system based on a large language model according to the present invention.

[0067] Figure 2 This is a flowchart of the large language model reasoning module of the present invention;

[0068] Figure 3 This is a flowchart of the verification process for the security management module of the present invention;

[0069] Figure 4This is a flowchart of the UAV intelligent task planning agent method based on a large language model according to the present invention;

[0070] Figure 5 This is a schematic diagram of the structured output analysis of the present invention;

[0071] Figure 6 This is a schematic diagram of a specific embodiment 1 of the present invention;

[0072] Figure 7 This is a schematic diagram of a specific embodiment 2 of the present invention;

[0073] Figure 8 This is a schematic diagram of a specific embodiment 3 of the present invention. Detailed Implementation

[0074] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0075] like Figure 1 As shown, the overall architecture of the UAV intelligent mission planning agent system based on a large language model in this invention includes six core modules: a user interaction module, a large language model inference module, a tool invocation module, a security management module, a state awareness module, and a memory module. These modules exchange data through standardized interfaces, forming a complete mission planning closed loop.

[0076] (1) User interaction module

[0077] The user interaction module is responsible for receiving natural language task instructions from the user and providing feedback on the task execution process and results. This module supports two input methods: command-line terminal input and ROS topic subscription. User-input instructions are placed in a command queue to await processing.

[0078] The user interaction module is also responsible for log publishing, publishing information such as the agent's thought process, actions, and observation results in structured JSON format for easy monitoring and debugging. Log information includes the following fields: role identifier, message type, and content.

[0079] (2) Large Language Model Reasoning Module

[0080] like Figure 2As shown, the large language model inference module is the core decision-making unit of the system. This module uses a structured output parser and defines the data pattern for Agent actions: ① thought: thought process, analyzing the current state and task progress. ② action_name: tool name, must be a name in the tool list, filled with "Final" when the task ends. ③ action_params: tool parameters, a dictionary in key-value pair form. ④ final_message: final message, filled only when action_name is Final.

[0081] The prompt template for the large language model inference module includes the following core elements:

[0082] ① System role setting: Define Agent as intelligent drone controller, and clarify its task is to perform logical reasoning based on user instructions and drone status and call tools to complete the task.

[0083] ② Current state description: Real-time injection of the UAV's position coordinates (x, y, z), yaw angle, and flight state information, enabling the model to make decisions based on the real state.

[0084] ③ Tool List Description: Provides a detailed description of the function, parameter format, and use cases of each available tool, offering a reference for tool invocation in the model.

[0085] ④ Execution rule description: Includes three core rules:

[0086] Spatial reasoning rule: If the user command includes a relative direction (such as "fly forward 5 meters"), the target coordinates must be calculated based on the current coordinates;

[0087] Self-correction rule: If the last tool call resulted in an error, the cause of the error in the observation information should be analyzed and the parameters adjusted.

[0088] Task completion rules: When the task is considered complete, set action_name to "Final" and fill in final_message.

[0089] ⑤ Output format instructions: Use the format specification generated by the structured output parser to require the model to be output strictly in JSON format.

[0090] (3) Tool Call Module

[0091] The tool invocation module encapsulates the basic operational capabilities of the drone. Each tool function contains three elements: name, description, and execution function. The tool functions defined in this embodiment include:

[0092] ① Take-off equipment

[0093] Function: Controls the drone to take off and reach a specified altitude;

[0094] Parameter: height (target height, in meters);

[0095] Return: Description of the execution result.

[0096] ② Landing equipment

[0097] Function: Controls the safe landing of drones;

[0098] Parameters: None;

[0099] Return: Description of the execution result.

[0100] ③ Waypoint flight tool (fly_to_waypoint)

[0101] Function: Controls the drone to fly to a specified three-dimensional coordinate.

[0102] Parameters: x, y, z (target position coordinates, in meters);

[0103] Return: Description of the execution result.

[0104] ④ Status retrieval tool (get_status)

[0105] Function: Obtain the current status of the drone;

[0106] Parameters: None.

[0107] Returns a dictionary containing the position, orientation, and state.

[0108] The tool invocation module also provides a tool description generation function, which formats all tool information into text descriptions for reference during large language model inference.

[0109] (4) Security Management Module

[0110] like Figure 4 As shown, the security management module is responsible for performing security checks on the action instructions generated by the large language model. This module defines the following hard constraint parameters:

[0111] Minimum flight altitude (MIN_Z): Set to 1.0 meter to prevent the drone from flying too low and causing a collision;

[0112] Maximum flight altitude (MAX_Z): Set to 15.0 meters to prevent the drone from flying too high and exceeding the safe range;

[0113] Maximum single movement distance (SAFE_XY_RANGE): Set to 20.0 meters to prevent excessive movement in a single command;

[0114] The security verification process is as follows:

[0115] For takeoff and waypoint flight commands, first extract the target altitude parameter and verify whether it is within the range of [MIN_Z, MAX_Z]. If the altitude is lower than the minimum value, return a safety interception message; if the altitude is higher than the maximum value, also return an interception message.

[0116] For waypoint flight instructions, the horizontal distance between the current position and the target position also needs to be calculated:

[0117]

[0118] in, and Indicates the direction of the target and Axis coordinates and Indicates the current and Axis coordinates;

[0119] If the distance exceeds the maximum single movement distance limit, a security interception message will be returned, and the user will be advised to proceed step by step.

[0120] The security check function returns a tuple (is_safe, message), where is_safe is a boolean value indicating whether the check passed, and message is a description of the check result.

[0121] (5) State awareness module

[0122] The state awareness module is responsible for acquiring the drone's state information in real time. This module obtains the following drone states through ROS topic subscriptions or direct API calls:

[0123] ① Location information: three-dimensional coordinates The unit is meters;

[0124] ② Attitude information: Yaw angle, in degrees;

[0125] ③ Flight status: including status indicators such as "standby", "in flight", and "landing".

[0126] The state awareness module provides formatted state output functionality, converting state information into natural language descriptions for easy integration into prompts in large language models.

[0127] "coordinate (3.5, 2.0, 5.0); Yaw: 45°; Status: in flight

[0128] Visual positioning submodule: Includes a built-in binocular camera and ORB feature point tracking algorithm, automatically activated in GPS signal failure scenarios, achieving accurate positioning through the following core formula:

[0129] Binocular vision depth calculation: The relative depth of the target is calculated by solving the parallax of the left and right views. The formula is:

[0130]

[0131] in, Indicates the target's depth relative to the drone. Indicates the focal length of a binocular camera. Indicates the baseline distance of the binocular cameras. The x-coordinate of the left view, This represents the x-coordinate of the right view.

[0132] ORB Feature Point Absolute Coordinate Calculation: Combining the world coordinates and relative displacement of the feature points, the absolute position of the UAV is calculated using the following formula:

[0133]

[0134] in, Represents the absolute three-dimensional coordinates of the drone. This indicates the world coordinates of ORB feature points used for environment matching. This represents the three-dimensional displacement vector of the feature point relative to the UAV.

[0135] Positioning accuracy compensation: Errors are corrected using the camera distortion coefficient, using the following formula:

[0136]

[0137] in, This indicates the final corrected coordinates. Indicates the initial calculated coordinates. and Indicates the camera distortion coefficient. Represents the radial distance of feature points in the image;

[0138] The state awareness module can automatically switch positioning modes based on GPS signal strength, prioritizing GPS positioning and seamlessly switching to visual positioning when the signal is weak, and synchronizing the positioning data to the large language model inference module.

[0139] (6) Memory module

[0140] The memory module uses a dialogue buffer window mechanism to store the dialogue history of the most recent K rounds. In this embodiment, K is set to 5, meaning that the most recent 5 rounds of dialogue are retained.

[0141] The data structure of the memory module includes: ① User message queue: stores the task instructions input by the user; ② Assistant message queue: stores the agent's thought process and actions; ③ System observation queue: stores the feedback results of the tool's execution.

[0142] The memory module provides historical context during each round of inference, enabling the large language model to understand the continuity of the task and contextual relationships. When the user enters a new instruction, the system marks the new instruction as "the user interrupted and entered a new instruction," allowing the model to correctly understand changes in the user's intent.

[0143] like Figure 3 As shown, the process of the UAV intelligent task planning agent method based on a large language model in this invention is as follows:

[0144] Step 1: System initialization, start the ROS node, and initialize each module component.

[0145] Step 2: Enter the main loop and wait for the user to input task instructions.

[0146] Step 3: After receiving the user's instruction, set the loop flag and add the instruction to the memory module.

[0147] Step 4: Obtain the current drone status and format it as a status description string.

[0148] Step 5: Obtain the tool list description string.

[0149] Step 6: Build a prompt template, injecting the current status, tool description, format requirements, and dialogue history.

[0150] Step 7: Call the large language model for inference.

[0151] Step 8: Use the structured output parser to parse the model output. If parsing fails, add a format error message and return to step 7 to retry.

[0152] Step 9: Extract the thought process, action name, and action parameters.

[0153] Step 10: Add the thought process and action records to the memory module and publish the log.

[0154] Step 11: Determine if the action name is "Final". If so, output the final message, the task is complete, and return to step 2 to wait for a new task.

[0155] Step 12: Locate the corresponding utility function; if not found, generate error observation information.

[0156] Step 13: Call the security management module to perform parameter verification.

[0157] Step 14: If the verification passes, execute the utility function and obtain the execution result; if the verification fails, generate security interception observation information.

[0158] Step 15: Add the observation information to the memory module and publish the log.

[0159] Step 16: Check if the maximum number of iterations has been reached (15 rounds in this embodiment). If so, force the task to end; otherwise, return to step 4 to continue the loop.

[0160] Step 17: During the loop, detect the interruption flag. If the user inputs a new instruction, obtain the new instruction and add it to the memory module, and continue the current loop (the model will process the new instruction according to the context).

[0161] like Figure 5 As shown, the specific implementation of structured output parsing is as follows:

[0162] The raw output of the large language model is JSON-formatted text, for example:

[0163] json{

[0164] "thought": "The user requests to fly forward 5 meters. The current position is (0, 0, 5). Flying forward 5 meters means increasing the x-coordinate by 5, and the target position is (5, 0, 5)."

[0165] "action_name": "fly_to_waypoint",

[0166] "action_params": {"x": 5, "y": 0, "z": 5},

[0167] "final_message": null

[0168] }

[0169] The structured output parser parses this JSON text into an Agent Action Schema object, extracting the fields for subsequent processing. If the JSON format is incorrect or missing necessary fields, the parser throws an exception, triggering a retry mechanism.

[0170] Example 1:

[0171] like Figure 6 As shown, the user issues mission instructions to the drone agent in natural language: "Please take off to a height of 3 meters, then fly forward 5 meters, and finally land."

[0172] The system execution flow is as follows:

[0173] Round 1 Reasoning:

[0174] User commands are entered into the system and added to the memory module;

[0175] Current state: coordinates (0, 0, 0), status: standby;

[0176] Model thinking: The user requires a three-step task, the first step is to take off and reach a height of 3 meters;

[0177] Model output: action_name = "take off", action_params = {"height": 3};

[0178] Safety check: 3 meters within the range of [1, 15], passed;

[0179] Takeoff executed. Observation result: "Takeoff successful, current altitude 3 meters."

[0180] Round 2 Reasoning:

[0181] Current state: coordinates (0, 0, 3), status: in flight;

[0182] Model thinking: Takeoff completed, next step is to fly forward 5 meters, current x = 0, target x = 5;

[0183] Model output: action_name="fly_to_waypoint", action_params={"x": 5", "y": 0", "z": 3};

[0184] Safety verification: Height of 3 meters is legal, movement distance of 5 meters is less than the 20-meter limit, passed;

[0185] Flight execution, observation results: "Reached target point (5, 0, 3)".

[0186] Round 3 Reasoning:

[0187] Current status: Coordinates (5, 0, 3), Status: Flying;

[0188] Model thinking: Flight complete, final step is landing;

[0189] Model output: action_name = "land", action_params = {};

[0190] The landing was executed, and the observation result was: "Landing successful."

[0191] Round 4 Reasoning:

[0192] Current state: coordinates (5, 0, 0), status: standby;

[0193] Model thinking: All task steps have been completed;

[0194] Model output: action_name = “Final”, final_message = “Mission completed: Takeoff to 3 meters, flew forward 5 meters, and landed safely”;

[0195] The task has been completed, and the system is now waiting for the next user instruction.

[0196] Example 2:

[0197] like Figure 7 As shown, the user first enters: "Take off to 5 meters", and then enters a new command during the execution process: "Cancel, change to take off to 8 meters".

[0198] Execution process:

[0199] Round 1: Begin execution of takeoff to 5 meters;

[0200] Interrupt flag detected; new instructions read.

[0201] The new command is added to the memory and marked as "User interrupted and entered new command: Cancel, change to takeoff to 8 meters";

[0202] Model reasoning: Understand changes in user intent, cancel the original plan, and execute the new instruction;

[0203] Model output: action_name = "take off", action_params = {"height": 8};

[0204] Execute takeoff to 8 meters;

[0205] The system successfully responded to the user's real-time command changes, demonstrating good interactive flexibility.

[0206] Example 3:

[0207] like Figure 8 As shown, the user inputs a complex relative position command: "Fly forward 3 meters, then turn 90 degrees to the left, then fly forward 2 meters again." Assume the drone's initial position is (0, 0, 5) and its initial orientation is north (positive y-axis direction).

[0208] Execution process:

[0209] Round 1 Reasoning:

[0210] Current state: Coordinates (x, y, z): (0, 0, 5) | Yaw: 0° | Status: in flight;

[0211] Model thinking: User commands include relative direction. Currently, Yaw = 0° indicates north (positive y-axis direction). "Fly forward 3 meters" means moving along the positive y-axis, with a target position of (0, 3, 5).

[0212] Model output: action_name="fly_to_waypoint", action_params={"x": 0, "y": 3, "z": 5};

[0213] Execution successful. Observation: "Target point (0, 3, 5) reached".

[0214] Round 2 Reasoning:

[0215] Current state: Coordinates (x, y, z): (0, 3, 5) | Yaw: 0° | Status: in flight;

[0216] Model Thinking: Step 1 complete. The next step is to turn 90 degrees to the left. Currently, Yaw = 0°. After turning 90 degrees to the left, Yaw should be -90° or 270°, and the orientation will change to west (negative x-axis direction). This system's tools do not support direct turning operations, but this can be achieved by changing the direction of the flight target point. The user next wants to "fly forward 2 meters." After the turn, "ahead" is in the negative x-axis direction, so the target point should be (-2, 3, 5).

[0217] Model output: action_name = "fly_to_waypoint", action_params = {"x": -2", "y": 3", "z": 5};

[0218] Execution successful. Observation: "Target point (-2, 3, 5) reached".

[0219] Round 3 Reasoning:

[0220] Current state: Coordinates (x, y, z): (-2, 3, 5) | Yaw: -90° | Status: In flight;

[0221] Model thinking: All steps have been completed: fly forward 3 meters (0, 0, 5) → (0, 3, 5), turn left 90 degrees and fly forward 2 meters (0, 3, 5) → (-2, 3, 5);

[0222] Model output: action_name="Final", final_message="Task completed: Flyed forward 3 meters, turned 90 degrees to the left and then flew forward 2 meters. Current position (-2, 3, 5)".

[0223] The above embodiments are used to explain and illustrate the present invention, but not to limit the present invention. Any modifications and changes made to the present invention within the spirit and scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A UAV intelligent task planning agent system based on a large language model, characterized in that, Includes the following modules: (1) User interaction module: Receives natural language task instructions from users and feeds back the task execution results to users; (2) Large Language Model Reasoning Module: Parses the user's natural language instructions, performs thought chain reasoning, and generates structured action decisions; (3) Tool Invocation Module: Executes specific UAV control operations; (4) Safety Management Module: Performs parameter verification and safety constraint checks on the action instructions generated by the large language model; (5) Status perception module: Real-time acquisition of the UAV's position, attitude, and flight status information; (6) Memory module: Stores the most recent K rounds of historical dialogue and task execution records, supporting context understanding and task continuity.

2. The UAV intelligent task planning agent system based on a large language model according to claim 1, characterized in that, The user interaction module has a log information publishing function, which publishes the Agent's thought process, execution actions, and observation results in a structured JSON format, allowing users to monitor and debug.

3. The UAV intelligent task planning agent system based on a large language model according to claim 2, characterized in that, The log information includes the following fields: role identifier, message type, and specific content.

4. The UAV intelligent task planning agent system based on a large language model according to claim 1, characterized in that, The large language model inference module uses a structured outputter to parse the output of the large language model into structured data containing the following fields: (1) The thought field records the reasoning process for analyzing the current state and task progress; (2) Action name field action_name: Specifies the name of the utility function to be called; (3) Action parameter field action_params: Specifies the input parameters of the utility function in key-value pairs; (4) The final message field final_message: Returns the execution result to the user when the task is completed.

5. The UAV intelligent task planning agent system based on a large language model according to claim 1, characterized in that, The tool calling module includes the following tool functions: (1) Take-off function: controls the UAV to take off to the specified altitude, the parameter is the target altitude (height); (2) The landing tool function `land` controls the safe landing of the UAV; (3) The waypoint flight tool function fly_to_waypoint controls the UAV to fly to the specified three-dimensional coordinate position. The parameters are the target coordinates x, y, z. (4) The get_status utility function is used to obtain the current position, attitude and flight status information of the UAV.

6. The UAV intelligent task planning agent system based on a large language model according to claim 1, characterized in that, The state awareness module obtains the drone's location information, attitude information, and flight status through ROS topic subscription or direct API calls.

7. The UAV intelligent task planning agent system based on a large language model according to claim 1, characterized in that, The memory module uses a dialogue buffer window mechanism to store the dialogue history of the last 5 rounds. Its data structure includes: user message queue, assistant message queue, and system observation queue.

8. A method for intelligent task planning agents for unmanned aerial vehicles (UAVs) based on a large language model, characterized in that, Includes the following steps: Step 1: Receive user task instructions, add the instructions to the dialogue history, construct a prompt template, and assemble the current drone status, description of available tools, output format requirements, and dialogue history into a complete prompt; Step 2: Use a large language model for reasoning. Parse the model output through a structured output parser to obtain the parsing result. Compare the parsing result with the preset error flag. If the parsing result is equal to the preset error flag, the parsing is determined to have failed. Add a format error message and return to the large language model to reason again. When the parsing result is not equal to the preset error flag, the parsing is considered successful, and the thought process, action name and action parameters are obtained. The thought process and action record are added to the dialogue history and a log report is generated. Step 3: Determine the action type, obtain the action name, and compare the action name with the preset terminator "Final". If the action name is equal to "Final", output a task completion signal, determine that the task is completed, and return the final message to the user; if the action name is not equal to "Final", continue to execute the subsequent process. Step 4: Find the corresponding utility function and obtain the search results. Compare the search results with the preset null value marker. If the search result is null, it is determined that the corresponding utility function has not been found, and an error observation message is generated. If the search result is not empty, it is determined that the corresponding utility function has been found, and the subsequent process continues; Step 5: Perform security verification on the action parameters using the tool function to obtain the action parameter values. Compare the action parameter values ​​with a preset pass threshold. If the action parameter value is lower than the pass threshold, the verification fails, and security interception observation information is generated. If the action parameter value is not lower than the pass threshold, the verification passes, the tool function is executed, and the observation results are obtained. The observation results are added to the dialogue history and the observation log is published. Return to Step 2 to continue the loop reasoning until the task is completed or the maximum number of iterations is reached.

9. The UAV intelligent task planning agent method based on a large language model according to claim 8, characterized in that, In step 1, the prompt template includes: (1) System role setting: Define Agent as intelligent drone controller, and clarify that it performs logical reasoning based on user instructions and drone status and calls tools to complete tasks; (2) Current state description: includes the three-dimensional coordinates of the UAV. Yaw angle and flight status; (3) Tool list description: Describe in detail the function, parameter format and use cases of each available tool; (4) Execution rule description: including spatial reasoning rules, self-correction rules and task termination rules; (5) Output format instructions: Use the format specification generated by the structured output parser to require the model to be output in JSON format; The drone's status is acquired by automatically switching positioning modes based on GPS signal strength. GPS positioning is used, and when the signal is weak, it seamlessly switches to visual positioning, synchronizing the positioning data to a large language model. The process of achieving accurate positioning through visual positioning is as follows: (1) Calculate the binocular visual depth by calculating the relative depth of the target through the difference between the left and right views. The calculation formula is as follows: in, Indicates the target's depth relative to the drone. Indicates the focal length of a binocular camera. Indicates the baseline distance of the binocular cameras. The x-coordinate of the left view, Represents the x-coordinate of the right view; (2) Calculate the absolute coordinates of the ORB feature points. Using the world coordinates and relative displacement of the feature points, calculate the absolute position of the UAV. The calculation expression is as follows: in, Represents the absolute three-dimensional coordinates of the drone. This indicates the world coordinates of ORB feature points used for environment matching. This represents the three-dimensional displacement vector of the feature point relative to the UAV. (3) Positioning accuracy compensation, which corrects the error through the camera distortion coefficient, is calculated as follows: in, This indicates the final corrected coordinates. Indicates the initial calculated coordinates. and Indicates the camera distortion coefficient. This represents the radial distance of the feature point image.

10. The UAV intelligent task planning agent method based on a large language model according to claim 8, characterized in that, In step 5, the security verification includes the following constraint parameters: (1) Set the minimum flight altitude to 1.0 meter; (2) Set the maximum flight altitude to 15.0 meters; (3) Set the maximum single movement distance to 20.0 meters; The specific security verification process is as follows: (1) For takeoff command, extract target altitude parameters to obtain target altitude value, compare target altitude value with preset maximum and minimum safe altitude values. When target altitude value is lower than minimum safe altitude or greater than maximum safe altitude, it is determined that safety verification fails and safety interception information is returned; when target altitude value is higher than minimum safe altitude and lower than maximum safe altitude, it is determined that safety verification passes, the Boolean value is_safe is returned, and the verification result description is returned. (2) For waypoint flight commands, first determine if the target altitude falls within the safe altitude constraint range, then calculate the horizontal distance between the current position and the target position. The calculation expression is as follows: in, and Indicates the direction of the target and Axis coordinates and Indicates the current and Axis coordinates; The horizontal distance is compared with a preset maximum single movement distance. If the horizontal distance is greater than the preset maximum single movement distance, the security check fails, an interception message is returned, and the user is advised to perform the operation step by step. If the horizontal distance is not greater than the preset maximum single movement distance, the security check passes, a boolean value is_safe is returned, and a description of the check result is returned. The loop reasoning process employs a task interruption mechanism. When a new instruction is input during task execution, the system sets an interruption flag, acquires the new instruction, and adds it to the dialogue history in the format of "user interrupted and input new instruction". In the next round of reasoning, the system adjusts the task execution strategy based on the updated context to understand the changes in user intent. The maximum number of iterations is set to 15.

Images