Autonomous drone system and method based on large model agent

Abstract

This application provides an autonomous unmanned aerial vehicle (UAV) system and method based on a large model intelligent agent, relating to the field of autonomous UAVs. The system includes: a large model intelligent agent module, a tool invocation interface layer, and a real-time control module. The large model intelligent agent module operates at a first frequency, used for task understanding and planning based on natural language task instructions and visual observation information, obtaining task planning results. The task planning results include at least a task sequence and the task's control objective, and may also include tool invocation requests. The tool invocation interface layer responds to tool invocation requests, invoking corresponding functional tools to perform task-related operations, and feeding back the execution results to the large model intelligent agent module and / or the real-time control module. The real-time control module operates at a second frequency greater than the first frequency, used for acquiring control objectives or acquiring control objectives and execution results, and combining real-time visual observation information and real-time UAV status information to generate control commands for UAV flight.

Description

Technical Field

[0001] This application relates to the field of autonomous unmanned aerial vehicle (UAV) technology, and in particular to an autonomous UAV system and method based on a large model intelligent agent. Background Technology

[0002] The application of drones in complex scenarios such as industrial inspection and emergency rescue is becoming increasingly widespread, and the requirements for autonomous capabilities are constantly increasing. Traditional autonomous flight systems based on preset programs or rules have poor adaptability in dynamic and unstructured environments and have difficulty handling complex natural language task commands.

[0003] Related technologies attempt to introduce artificial intelligence models, such as large language models, into UAV systems to improve their intelligence level, but they have significant drawbacks: On the one hand, the reasoning process of large models is time-consuming, making it difficult to meet the millisecond-level high-frequency closed-loop requirements of UAV flight control, resulting in insufficient real-time assurance. On the other hand, in practical UAV systems, positioning, navigation, and reconstruction modules built based on mature algorithms such as visual inertial odometry, simultaneous localization, and mapping are the engineering foundation for reliable flight; however, related solutions lack a unified interface and intelligent scheduling mechanism to toolize these core functions. The large model is regarded as a closed, all-powerful brain, either attempting to integrate all functions internally and becoming bloated and inefficient, or remaining isolated from specialized modules, unable to flexibly and accurately call upon these proven engineering assets, resulting in limited overall system capabilities and poor resource utilization.

[0004] Therefore, the relevant technologies are difficult to reliably execute complex natural language instructions in open and dynamic scenarios. Summary of the Invention

[0005] The purpose of this application is to provide an autonomous unmanned aerial vehicle (UAV) system and method based on a large model intelligent agent. This system architecture can effectively integrate the high-level task understanding capabilities of the large model with the professional functions of mature engineering modules while ensuring real-time flight control performance, so as to support the reliable execution of complex natural language command tasks in open and dynamic scenarios.

[0006] A first aspect of this application discloses an autonomous unmanned aerial vehicle (UAV) system based on a large-scale intelligent agent, comprising: The large model intelligent agent module runs at the first frequency and is used to perform task understanding and planning based on natural language task instructions and visual observation information to obtain task planning results. The task planning results include at least a task sequence and the control objectives of each task in the task sequence, or may also include tool invocation requests. The control objectives characterize the executable state requirements of the task in the current environment. The tool call interface layer encapsulates multiple functional tools. In response to the tool call request, it calls the corresponding functional tool to perform task-related operations and feeds back the execution results to the large model intelligent agent module and / or the real-time control module. The real-time control module operates at a second frequency greater than the first frequency, and is used to acquire the control target or acquire the control target and the execution result, and generate control commands to control the flight of the UAV by combining real-time visual observation information and real-time UAV status information.

[0007] Optionally, the system further includes an accelerated inference module for providing hardware-accelerated computation for the multimodal large model in the large model agent module; The accelerated inference module includes at least one neural network processor and performs at least one of the following optimization operations for the multimodal large model: Before deploying the multimodal large model to the neural network processor, the model parameters are subjected to numerical precision transformation to reduce storage and computational load. During the inference process of the multimodal large model, the amount of data involved in the model calculation is reduced based on the correlation between the input data and the current task. When the multimodal large model generates sequences, it uses a retrieval or prediction mechanism to provide candidate sequences, and the multimodal large model verifies the candidate sequences.

[0008] Optionally, the numerical precision transformation of the model parameters includes: Perform a linear transformation on at least some of the weight parameter matrices of the model, such that the error of the transformed parameters when converted to a low-precision numerical representation is less than the error of direct conversion; The amount of input data used in the calculation is reduced according to preset rules, including: When the multimodal large model processes the visual observation information, it evaluates the importance of each region in the visual observation information to the current task and retains data regions with importance higher than the threshold for subsequent calculations.

[0009] Optionally, the accelerated inference module further includes: The backend management module is used to select the first computing unit for the first execution of model inference based on the type and status of the currently available computing resources when the system starts up, and to continuously monitor the inference status and performance indicators of the first computing unit during system operation. If the first computing unit detects an inference error or its real-time inference performance does not meet the requirements of the current task, the backend management module will switch the ongoing or subsequent inference task to the alternative second computing unit for execution.

[0010] Optionally, the tool calling the functional tools encapsulated in the interface layer includes at least one of the following: Airborne perception and modeling tools, including localization and modeling tools for acquiring the position and attitude of the UAV itself, and detection and tracking tools for identifying and tracking specific targets from visual observation information; Airborne environmental understanding tools include 3D reconstruction tools for generating 3D structural information of the environment, and forced landing point assessment tools for evaluating safe landing areas; An airborne payload control tool is used to control the actuators carried by the UAV, the actuators including a gimbal control tool, a lighting tool, and a cargo release tool; Cloud-based collaboration tools are used to invoke computing tools or data services deployed on cloud servers when the drone's communication module is connected to the network.

[0011] Optionally, the large model agent module further includes: The memory storage module is used to store historical information during task execution and the execution results; The large model intelligent agent module is also used to perform task understanding and planning based on the historical information, natural language task instructions, visual observation information and laser information to obtain a task sequence, as well as the control objectives and tool call requests of each task in the task sequence. The laser information is collected by a laser device or predicted by the large model intelligent agent module; and / or, it is used to update the control objectives based on the execution results.

[0012] Optionally, the system further includes: The safety monitoring module is used to monitor the control commands and / or the flight status of the UAV according to preset safety rules, and to execute emergency control strategies in case of violation of safety constraints.

[0013] Optionally, the system further includes: A multi-path visual perception module, including a binocular vision unit and / or a depth vision unit, is used to provide visual observation information; The status awareness module is used to provide drone status information; The flight control module is used to control the flight of the UAV according to the control commands.

[0014] Optionally, the large model agent module is further configured to: upon receiving example information similar to the current task, incorporate the example information into the task understanding and planning process through a few-shot transfer unit to adjust the control objective or the tool invocation request; The small sample migration unit is configured to perform at least one of the following operations: The example information is converted into a structured prompt template to guide target identification and task execution strategies; The example information is indexed into the memory storage module, and during subsequent task execution, if an appearance or scene similar to the example information is detected, the detection threshold is adjusted. Based on the example information, the model parameters related to the task of the large model agent module are updated. The update includes at least one of the following: low-rank parameter update and full fine-tuning of some modules.

[0015] A second aspect of this application discloses an autonomous unmanned aerial vehicle (UAV) method based on a large model intelligent agent, comprising: Obtain natural language task instructions; The large model agent module, running at the first frequency, performs task understanding and planning based on natural language task instructions and visual observation information to obtain task planning results. The task planning results include at least a task sequence and control objectives for each task in the task sequence, or may also include tool invocation requests. The control objectives characterize the executable state requirements of the task in the current environment. If the task planning result also includes the tool invocation request, the functional tool encapsulated in the tool invocation interface layer is invoked to perform task-related operations and obtain the execution result; The real-time control module, operating at a second frequency, acquires the control target or acquires the control target and the execution result, and generates control commands for controlling the drone's flight by combining real-time visual observation information and real-time drone status information; the second frequency is greater than the first frequency. Control the drone to fly according to the control commands.

[0016] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the autonomous unmanned aerial vehicle method based on a large model intelligent agent as described in the second aspect of this application.

[0017] A fourth aspect of this application discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the autonomous unmanned aerial vehicle method based on a large model intelligent agent as described in the second aspect of this application.

[0018] A fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the autonomous unmanned aerial vehicle method based on a large model intelligent agent as described in the second aspect of this application.

[0019] The technical solution provided in this application has the following beneficial effects: First, a large-model intelligent agent module is introduced at a lower first frequency to handle task understanding and planning, transforming complex natural language commands and environmental observations into abstract decisions containing task sequences, control objectives, and tool call requests. Simultaneously, a real-time control module is introduced at a higher second frequency, continuously processing control objectives from the large-model intelligent agent module and / or execution results from the tool call interface layer, and combining this with real-time visual observation information and real-time UAV status information to generate precise flight control commands. This dual-module, multi-frequency collaborative architecture achieves a slow-thinking, fast-execution intelligent control paradigm, resolving the contradiction between the inference latency of the large model and the real-time requirements of flight control. It retains the cognitive advantages of the large model in task understanding and planning while ensuring the stability, agility, and safety of the UAV in dynamic environments.

[0020] Furthermore, the tool call interface layer encapsulates various specialized functional tools (such as localization, object detection, and 3D reconstruction), eliminating the need for large-scale intelligent agent modules to implement all functions internally or be directly coupled with complex algorithms. These functional tools can be invoked through tool call requests to perform task-related operations. This design reduces the complexity of the intelligent agent model and allows the system to easily integrate new functional tools to expand its capabilities, thereby improving the system's engineering practicality and scenario adaptability.

[0021] Finally, the real-time control module has independent closed-loop control capabilities based on the latest sensor data (real-time visual observation information and real-time UAV status information). This ensures that even if the large model agent module fails to update the control target in time due to inference delay, the system can still rely on the local rapid response of the real-time control module to make safe and reasonable adjustments according to environmental changes. This significantly improves the overall system's mission success rate and operational safety redundancy in complex, dynamic, and even partially unknown environments. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the structure of an autonomous unmanned aerial vehicle system based on a large model intelligent agent, provided in an embodiment of this application; Figure 2 This application provides a schematic diagram of a backend management module and a model deployment pipeline. Figure 3 This is a schematic diagram of the software architecture of an autonomous unmanned aerial vehicle system based on a large model intelligent agent, provided in an embodiment of this application; Figure 4 This is a schematic diagram of the hardware architecture of an autonomous unmanned aerial vehicle system based on a large model intelligent agent, provided in an embodiment of this application. Figure 5 This is a flowchart illustrating the steps of an autonomous unmanned aerial vehicle (UAV) method based on a large model intelligent agent, as provided in an embodiment of this application. Figure 6 This is a flowchart illustrating the implementation of an autonomous shooting and camera movement and dynamic target tracking task provided in an embodiment of this application. Figure 7 This is a flowchart illustrating the implementation of an automated industrial inspection method provided in an embodiment of this application. Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Reference Figure 1 As shown, Figure 1 This is a schematic diagram of the structure of an autonomous unmanned aerial vehicle system based on a large-scale intelligent agent, provided in an embodiment of this application. Figure 1 As shown, the system includes: The large model intelligent agent module runs at the first frequency and is used to perform task understanding and planning based on natural language task instructions and visual observation information to obtain task planning results. The task planning results include at least a task sequence and the control objectives of each task in the task sequence, or may also include tool invocation requests. The control objectives characterize the executable state requirements of the task in the current environment. The tool call interface layer encapsulates multiple functional tools. In response to the tool call request, it calls the corresponding functional tool to perform task-related operations and feeds back the execution results to the large model intelligent agent module and / or the real-time control module. The real-time control module operates at a second frequency greater than the first frequency, and is used to acquire the control target or acquire the control target and the execution result, and generate control commands to control the flight of the UAV by combining real-time visual observation information and real-time UAV status information.

[0026] In this embodiment, the large model intelligent agent module, the tool call interface layer, and the real-time control module are typically integrated into the onboard computing unit (OCU) of the UAV and work in conjunction with hardware such as the flight controller and sensors.

[0027] The large-scale intelligent agent module, acting as the system's brain, operates at a relatively low initial frequency (e.g., 1Hz to 5Hz, configurable according to task complexity). It is responsible for receiving and understanding natural language task instructions, and combining this with visual observation information from sensors to perform task understanding, decomposition, and planning. Natural language task instructions can be commands issued by the user via voice, text, or a ground station interface, such as, "Inspect the cooling tower on the southeast side of the plant, focusing on photographing potentially corroded pipe welds." Visual observation information can be collected from forward-looking, downward-looking, and pan-tilt-zoom cameras, providing environmental context for task planning, including real-time images, video streams, or processed features.

[0028] Specifically, the large-scale intelligent agent module typically includes a visual language model, a large language model, or a multimodal large model fusion thereof. Its workflow includes: (a) Instruction parsing and scene parsing: understanding the user's intent and the current environmental state based on natural language task instructions and visual observation information. (b) Task decomposition and serialization: decomposing the overall task into a logically coherent sequence of tasks. For example, decomposing it into sub-tasks such as "fly to region A", "search for target B", and "closely observe C". (c) Generating executable outputs: generating two types of key outputs for each task in the task sequence, namely: control target and tool call request.

[0029] Among them, the control objective is a high-level, abstract state requirement used to characterize the executable state requirements of the task in the current environment. For example, "navigate the UAV to an airspace centered at geographic coordinates (X,Y) with a radius of R meters." A tool invocation request is a structured command used to request the invocation of specific external functional tools to assist in completing the task, such as requesting the "global path planning tool" along with the target coordinates.

[0030] It is understandable that the generation of tool invocation requests during task planning is conditional and optional. The large model agent module autonomously determines whether external tools are needed to assist in completing sub-objectives based on the complexity of the task decomposition and the sufficiency of current environmental information. Specifically, when a task requires specific external functions (such as precise path planning, precise target positioning, or professional analysis), the large model agent module will generate corresponding tool invocation requests. For example, for a sub-task of "navigating to a specific coordinate point," a request to invoke a path planning tool is typically generated to obtain a safe and feasible flight trajectory. For relatively simple sub-tasks (e.g., "flying forward a predetermined distance" or "yawing to the left by a specific angle" based directly on the current visual image), the large model agent module may directly generate high-level control objectives (such as velocity vector or heading angle commands) without generating additional tool invocation requests. In this case, the real-time control module will directly generate control commands based on the control objectives, real-time visual observation information, and real-time UAV status information.

[0031] The tool call interface layer serves as a bridge connecting the cognitive planning layer (large model agent) and the underlying specialized functional modules. It encapsulates a series of independent, mature, and reliable functional tools using standardized interfaces (such as Application Programming Interfaces, APIs). Its workflow is as follows: it receives tool call requests from the large model agent module; based on the type and parameters of the request, it asynchronously or synchronously calls the corresponding functional tool to perform specific operations (e.g., calling a target detection tool to identify "pipe welds"). Subsequently, it formats the tool's execution results (such as pose data, bounding box coordinates, planned path points, etc.) and feeds them back to the large model agent module, or, as needed, to the real-time control module.

[0032] The real-time control module, acting as the cerebellum of the system, operates independently at a second frequency (e.g., 20Hz, 50Hz, 100Hz) higher than the first frequency. Its core responsibility is to ensure the safe, smooth, and real-time response of the UAV's flight control. The module receives diverse inputs, including: control objectives from the large model agent module, execution results from the tool call interface layer, real-time visual observation information (such as acquired instantaneous images and depth data), and real-time UAV status information (such as position, velocity, attitude, and angular velocity).

[0033] The real-time control module can contain Model Predictive Control (MPC), lightweight neural network strategies (such as diffusion strategies), or traditional geometric motion planners. Within each control cycle, it integrates some or all of the above inputs and generates control commands (such as attitude angle commands and motor throttle commands) through its internal controller or policy network, which are then output to the flight control system for execution.

[0034] Understandably, the high-frequency operation and decision generation of the real-time control module do not depend on the successive inference results of the large model intelligent agent module, thereby decoupling the uncertain time delay of complex cognitive planning from the safety-critical control loop.

[0035] The collaborative workflow of the three modules is as follows: After the user issues a natural language task command, the large-scale intelligent agent combines visual observation information to perform task understanding and planning, obtaining the task planning result. Next, if the task planning result includes a tool invocation request, the tool invocation interface layer invokes a specific functional tool to execute task-related operations and feeds back the execution result to the large-scale intelligent agent and / or the real-time control module. Then, the real-time control module generates control commands for the drone's flight based on the control objective, execution result, real-time visual observation information, and real-time drone status information. During operation, the large-scale intelligent agent module can update subsequent control objectives based on new visual observation information or execution results; the tool invocation interface layer may also directly provide detected obstacle information to the real-time control module, enabling it to adjust commands in real time for obstacle avoidance. This cycle repeats, forming a collaborative autonomous closed loop of slow planning, fast execution, and tool empowerment.

[0036] The technical solution adopted in this embodiment introduces a large model intelligent agent module responsible for task understanding and planning at a lower first frequency, transforming complex natural language commands and environmental observations into abstract decisions containing task sequences, control objectives, and tool call requests. Simultaneously, a real-time control module is introduced at a higher second frequency, continuously receiving control objectives from the large model intelligent agent module and / or execution results from the tool call interface layer, and combining this with real-time visual observation information and real-time UAV status information to generate precise flight control commands. This dual-module, multi-frequency collaborative architecture achieves a slow-thinking, fast-execution intelligent control paradigm, resolving the contradiction between the inference latency of the large model and the real-time requirements of flight control. It retains the cognitive advantages of the large model in task understanding and planning while ensuring the stability, agility, and safety of the UAV in dynamic environments. Furthermore, the tool call interface layer encapsulates various dedicated functional tools (such as localization, target detection, and 3D reconstruction), allowing the large model intelligent agent module to execute task-related operations by calling these functional tools through tool call requests without needing to implement all functions internally or be directly coupled with complex specific algorithms. This design reduces the complexity of the agent model and allows the system to easily integrate new functional tools to expand its capabilities, improving its engineering practicality and scenario adaptability. Finally, the real-time control module possesses independent closed-loop control capabilities based on the latest sensor data (real-time visual observation information and real-time UAV status information). This ensures that even if the large-model agent module fails to update the control target in time due to inference delays, the system can still rely on the rapid local response of the real-time control module to make safe and reasonable adjustments according to environmental changes. This significantly improves the overall system's mission success rate and operational safety redundancy in complex, dynamic, and even partially unknown environments.

[0037] In some embodiments, the system further includes: A multi-path visual perception module, including a binocular vision unit and / or a depth vision unit, is used to provide visual observation information; The status awareness module is used to provide drone status information; The flight control module is used to control the flight of the UAV according to the control commands.

[0038] In this embodiment, the multi-path vision perception module acts as the system's eyes, providing rich visual data with 3D spatial understanding capabilities to the large-scale intelligent agent module and the real-time control module, thereby achieving low-cost 3D perception without relying on LiDAR. The binocular vision unit typically consists of two cameras passing a target, generating a depth map in real time by calculating parallax; the depth vision unit refers to a sensor that can directly output pixel-level depth information, such as a structured light depth camera or a time-of-flight (ToF) camera.

[0039] In practical deployments, the systems can be flexibly combined. For example, forward-looking binoculars can be used for navigation and obstacle avoidance, downward-looking ToF cameras can be used for precise landing, and the gimbal main camera can be used for mission photography. All images and depth data are timestamped and then aligned to form a unified visual observation information stream.

[0040] The state perception module, acting as the system's proprioceptor, is responsible for providing the UAV's state information. It integrates data from various sensors, including the Internal Measurement Unit (IMU), Global Navigation Satellite System (GNSS), magnetometer, barometer, and ultrasonic altimeter, and uses filtering algorithms (such as Kalman filtering) to calculate the UAV's precise position, velocity, attitude, angular velocity, and other state information in real time. This information forms the basis for high-precision control and navigation decisions.

[0041] The flight control module is the final execution unit. It receives control commands from the real-time control module and converts them into specific control signals (such as PWM waves) to drive the various power units (such as motors) of the UAV through internally high-frequency proportional-integral-derivative (PID) controllers and nonlinear controllers. In this way, the abstract control commands are transformed into precise physical motion.

[0042] The technical solution adopted in this embodiment uses a multi-path visual perception module as the primary means of environmental perception. While ensuring sufficient spatial understanding capabilities (for obstacle avoidance and localization), it reduces hardware costs and system complexity, making drones with advanced autonomous capabilities more competitive in the market and easier to deploy. The state perception module ensures that the drone can obtain its own state information in various environments (indoor / outdoor, open / obstructed), which is the cornerstone of stable control and autonomous navigation. The flight control module, as the execution end, ensures that the smooth trajectory or target state planned by the upper layer can be executed accurately and quickly, guaranteeing the effective implementation of the entire intelligent system's intent.

[0043] In some embodiments, the tool invoking the functional tools encapsulated in the interface layer includes at least one of the following items A-1 to A-4: Item A-1: ​​Airborne perception and modeling tools, including localization and modeling tools for acquiring the position and attitude of the UAV itself, and detection and tracking tools for identifying and tracking specific targets from visual observation information; Item A-2: Airborne environmental understanding tools, including 3D reconstruction tools for generating 3D structural information of the environment, and forced landing point assessment tools for evaluating safe landing areas; Item A-3: Airborne payload control tools, used to control the execution devices carried by the UAV, the execution devices including gimbal control tools, lighting tools, and cargo release tools; Item A-4: Cloud-based collaboration tools, used to invoke computing tools or data services deployed on cloud servers when the drone's communication module is connected to the network.

[0044] In this embodiment, the tool in item A-1 provides the UAV with the ability to recognize its own state and model its environment. The localization and modeling tool typically encapsulates algorithms such as Visual Inertial Odometry (VIO), Visual Inertial Navigation System (VINS), or Simultaneous Localization and Mapping (SLAM). It continuously outputs the UAV's high-precision pose (position and attitude) and can build / maintain an environmental map. This tool provides a precise spatiotemporal reference and spatial context for navigation decisions. Specifically, when the large model agent module needs to obtain the UAV's own position and attitude for navigation decisions, or needs to build an environmental map to support subsequent tasks, it will call this localization and modeling tool. The localization and modeling tool continuously receives UAV state information and visual observation information, and through fusion and optimization calculations, outputs the UAV's own position and attitude, and builds and maintains a local or global environmental map (which may be a sparse feature point map, a dense point cloud map, or a semantic raster map, etc.). Finally, the encapsulated pose data and / or map information (such as passable areas and key landmarks) are used as the execution result and fed back to the large model agent module and / or real-time control module.

[0045] The detection and tracking tool encapsulates deep learning-based algorithms for object detection, instance segmentation, and multi-object tracking. It can identify and continuously track specific objects (such as vehicles, pipes, and defects) from a visual stream, outputting their category, location, and motion state. Specifically, when the task involves specific objects (such as "cooling towers" or "pedestrians"), the large model agent module requests this module. The detection and tracking tool receives the specified visual observation information, identifies the target object based on a pre-trained or online-loaded model, and outputs its category, bounding box, pixel mask, and other information. Furthermore, this module can associate the same target across consecutive frames, outputting the target's trajectory in the image coordinate system, and can combine depth or pose information to estimate the target's relative position and velocity in 3D space.

[0046] The tools in section A-2 provide in-depth analysis and advanced representation capabilities of the environment. Among them, the 3D reconstruction tool encapsulates scene representation algorithms such as 3D Gaussian splashing and neural radiation fields, generating textured, dense 3D models based on multi-view images for advanced tasks such as fine measurement and digital twins. Specifically, when performing tasks requiring fine spatial understanding or offline analysis (e.g., omnidirectional 3D modeling of complex industrial equipment to detect deformation), the large model agent module invokes this module; the 3D reconstruction tool receives image sequences from multiple perspectives, potentially containing depth information, and uses its internal algorithms to generate a 3D scene model (containing rich geometric details and visual appearance information) as the execution result. The large model agent module can then use this model for more precise measurements (e.g., calculating crack length), planning fine observation paths (e.g., fly-around scanning), or providing visual evidence for generating detailed task reports.

[0047] The forced landing point assessment tool comprehensively analyzes visual and map information to assess the flatness, slope, and obstacle conditions of the ground area. In emergency situations (such as low battery or communication interruption), it outputs the optimal safe landing point coordinates, thereby improving the survivability of drones in complex or unknown environments.

[0048] The tools in section A-3 directly translate the agent's decisions into control of the physical actuators onboard the UAV, achieving a closed loop from perception planning to physical actions. Specifically, the gimbal control tool precisely controls the gimbal's pitch, roll, yaw angles, and speed to enable automatic mapping, target tracking and shooting, or scanning observation. The lighting control tool controls the on / off state, brightness, and angle of the onboard lights in low-light or nighttime conditions to supplement visual perception or perform specific lighting tasks. The cargo release tool controls the robotic arm, grappling hook, and dispenser to perform physical interaction tasks such as material delivery, sample collection, and equipment operation. By invoking these tools, large-scale intelligent agents can directly schedule hardware actions, much like scheduling software functions, expanding the UAV's mission execution dimensions.

[0049] The tools in section A-4 embody the system's cloud-edge-device collaborative architecture. When the drone's communication modules (such as 4G / 5G / Wi-Fi) are in a good connectivity state, the tool's interface layer can call powerful computing resources or data services deployed on cloud servers across network boundaries. For example, it can call high-precision 3D reconstruction services that require massive computing power, or access the latest high-precision map databases, meteorological data services, or professional defect knowledge bases in the cloud. The processing results of the cloud tools are returned to the onboard system via the network and integrated into the local decision-making and planning processes.

[0050] The technical solution adopted in this embodiment encapsulates complex professional algorithms such as localization, detection, reconstruction, and security assessment into standardized tools through a tool call interface layer. This results in a clear system architecture, allowing each module to be developed, optimized, and updated independently, thus improving the maintainability and technological evolution capabilities of the entire system. Integration through the interface layer, rather than requiring a single end-to-end model to replace everything, reduces technical risks and ensures the system's reliability, interpretability, and engineering feasibility in real-world scenarios, facilitating rapid product development and deployment.

[0051] In some embodiments, the large model agent module further includes: The memory storage module is used to store historical information during task execution and the execution results; The large model intelligent agent module is also used to perform task understanding and planning based on the historical information, natural language task instructions, visual observation information and laser information to obtain a task sequence, as well as the control objectives and tool call requests of each task in the task sequence. The laser information is collected by a laser device or predicted by the large model intelligent agent module; and / or, it is used to update the control objectives based on the execution results.

[0052] In this embodiment, the memory storage module is a structured data storage and management system within or closely associated with the large model intelligent agent module. The historical information and execution results it stores typically include, but are not limited to, the following categories: (a) Task context history, including received natural language task instructions, decomposed task sequences, control targets generated by each task, and tool call requests. (b) Environmental observation snapshots: representative visual observation information (such as images and feature vectors) stored in keyframe form, along with their associated timestamps and UAV poses. (c) Tool call records and feedback: detailed records of each request content executed through the tool call interface layer, the specific tool called, the execution results returned by the tool (such as detected target location, planned path, reconstructed model pointers, etc.), and the execution status (success / failure / timeout). (d) Spatial-semantic map index: through interaction with tools such as the localization modeling module and 3D reconstruction tools, the memory storage module can establish or index a lightweight spatial memory. For example, semantic information such as "Area A has been inspected" can be associated with spatial coordinates to form a cognitive map. (e) Task status and progress: Dynamic status information such as which stage the current task has been executed, which sub-goals have been achieved, which have encountered obstacles, and whether the user has interrupted or modified the instructions.

[0053] The large model agent module can perform task understanding and planning based on historical information. Specifically, when receiving new natural language task instructions or needing to replan, the large model agent module's reasoning process actively retrieves historical information from its memory storage module and uses it as additional contextual input. For example, when faced with the task "continue inspecting the factory roof," the large model agent module retrieves historical information from the previous inspection task, recalling incomplete inspection points, discovered defect locations, and keyframes of the scene at that time. This allows it to accurately understand the meaning of continuing and plan the subsequent task sequence starting from the last interruption point. For example, when re-entering a previously explored environment, the large model agent module can call upon the spatial-semantic map in its memory to quickly identify known landmarks (such as wind turbine No. 3), avoiding redundant mapping, and can directly plan a path to a specific target (such as the wind turbine's gearbox), significantly improving task initiation efficiency and localization robustness. For example, when decomposing complex tasks, large model agents can refer to the execution records of similar tasks in the past (tool call records and feedback), select those task decomposition strategies and tool combinations that have been verified to be efficient and reliable, and avoid operation sequences that have led to failure in the past.

[0054] The large model agent module can update control objectives based on execution results. Specifically, when the tool calls the interface layer and returns execution results, these results are written to the memory storage module in real time, updating the current task's state perception. Furthermore, the large model agent module periodically, or upon receiving critical feedback, queries the latest state in memory (especially execution results related to the current task) for reassessment. Based on the new assessment, the large model agent module can dynamically update the originally defined control objectives. For example: Original objective: "Fly to the preset coordinate point (X,Y,Z) to take pictures," Execution result: The landing point assessment tool reports that the ground below this coordinate point is uneven; Updated objective: "Find a flat location with good visibility near coordinate point (X,Y,Z) to take pictures."

[0055] By employing the technical solution of this embodiment, the UAV, through its memory storage module, can transcend the limitations of a single mission session, understanding and executing complex instructions with time dimensions and context dependencies, thus expanding the depth of human-computer interaction and the coherence of task management. Simultaneously, the memory storage module, acting as an experience knowledge base, enables the planning of large-scale intelligent agent modules to reuse successful strategies, avoid known risks, reduce ineffective exploration and redundant computation, thereby making more informed and efficient decisions and accelerating task execution. Furthermore, by dynamically updating control objectives based on execution results, the system can autonomously adjust its strategies when environmental changes occur or unexpected events occur during tool execution, improving the success rate and autonomy of tasks in open and dynamic environments.

[0056] In some embodiments, the large model agent module is further configured to: upon receiving example information similar to the current task, incorporate the example information into the task understanding and planning process via a few-shot transfer unit to adjust the control objective or the tool invocation request; wherein the few-shot transfer unit is configured to perform at least one of the following operations: Item B-1: Convert the example information into a structured prompt template to guide target identification and task execution strategies; Item B-2: Index the example information into the memory storage module, and adjust the detection threshold when a similar appearance or scene is detected during subsequent task execution; Item B-3: Based on the example information, update the model parameters related to the task of the large model agent module, and the update includes at least one of the following: low-rank parameter update, full fine-tuning of some modules.

[0057] In this embodiment, in order to enable the system to quickly adapt to new scenarios and tasks, the large model agent module supports few-shot transfer functionality, that is, allowing users to provide a small amount of example information, and through the built-in few-shot transfer unit, this example information is efficiently integrated into its existing task understanding and planning framework, so as to achieve rapid adaptation to unknown or novel tasks without the need for time-consuming, resource-intensive, and potentially system-stability-affecting complete model retraining.

[0058] For item B-1, this operation analyzes and abstracts the example information provided by the user (e.g., an image marked "insulator damaged" and a descriptive text), extracts key features and task intent, and converts it into a structured prompt template (e.g., "Defect type: rust; Appearance: reddish-brown patches; Shooting requirements: close-up multi-angle"). This is used to guide the large model agent module during task planning, enabling it to generate more precise control objectives and tool call requests.

[0059] For item B-2, this operation extracts features from example information (especially visual examples) to form a feature vector, which is then indexed into the system's memory storage module as a new record and associated with a specific task label. When performing related tasks subsequently, the system can dynamically adjust the sensitivity or threshold of relevant sensing tools when similar appearances or scenes are perceived, improving the recognition rate of new target categories. For example, in routine inspections, the rust detection threshold is set to 0.8 to avoid false alarms; however, when the memory bank contains user-provided examples of "minor initial rust," the threshold can be temporarily lowered to 0.6 in similar component areas, thereby improving the detection rate of similar minor defects.

[0060] Regarding item B-3, this operation, based on a small amount of example information provided by the user, performs targeted updates to the model parameters in the large model agent module that are relevant to the current task. Depending on the task requirements, data volume, and computational resources, different strategies can be employed for this update. Specifically, low-rank parameter updates only insert or adapt a small number of low-rank trainable new parameter layers (e.g., using LoRA technology) into the original model network. Only these minimally sized new parameters are fine-tuned. After completion, only this portion of parameters (typically only a few MB) needs to be deployed and updated, resulting in minimal interference with the fundamental capabilities of the original large model, yet significantly improving its performance for the new task. Full fine-tuning of certain modules involves fine-tuning all parameters of specific downstream functions within the large model agent module (such as the last layer of the visual encoder, the task planning head, the defect classification layer, etc.) based on task characteristics. Finally, the updated parameter files (whether low-rank adaptation layers or fine-tuned modules) can be quickly deployed to the edge, enabling the system to acquire specialized capabilities for the new task without requiring full model retraining.

[0061] By adopting the technical solution of this embodiment, users do not need to prepare massive amounts of data for every new scenario and new target. They only need to provide a few typical examples to complete the initial configuration and adaptation of the task in a few hours or even minutes, which greatly reduces the threshold and cycle of customized task deployment.

[0062] In some embodiments, the system further includes an accelerated inference module for providing hardware-accelerated computation for the multimodal large model in the large model agent module. The accelerated inference module includes at least one neural network processor and performs at least one of the following optimization operations for the multimodal large model: Item D-1: Before deploying the multimodal large model to the neural network processor, perform numerical precision transformation on the model parameters to reduce storage and computational load; Item D-2: During the inference process of the multimodal large model, reduce the amount of data involved in the model calculation based on the correlation between the input data and the current task; Item D-3: When the multimodal large model generates sequences, it uses a retrieval or prediction mechanism to provide candidate sequences, and the multimodal large model verifies the candidate sequences.

[0063] In this embodiment, to efficiently run complex large models on a resource-constrained (computing power, power consumption, storage) airborne platform, the system includes an accelerated inference module. The hardware core of this accelerated inference module is at least one neural network processor integrated into the airborne computing unit; for example, Huawei's Ascend series, Horizon Robotics' Journey series, Rockchip's built-in network processor, or similar domestically produced AI (Artificial Intelligence) acceleration chips. Compared to general-purpose processors and even graphics processors, neural network processors offer orders of magnitude advantages in energy efficiency and computational density when performing typical AI operations such as matrix multiplication and addition.

[0064] To further improve inference performance on neural network processors, the accelerated inference module also performs at least one of the above-mentioned optimization operations, from D-1 to D-3, on multimodal large models.

[0065] For term D-1, since the original parameters of large multimodal models are typically 32-bit floating-point numbers (FP32), they occupy a large amount of storage space and are computationally slow. In order to take advantage of the low-precision computing power of neural network processors, the model parameters are transformed for numerical precision before being deployed to the neural network processor (e.g., the model parameters are converted to 8-bit integers INT8) to reduce storage and computational costs.

[0066] Optionally, the numerical precision transformation of the model parameters includes: performing a linear transformation on at least a portion of the model's weight parameter matrix, such that the error of the transformed parameters when converted to a low-precision numerical representation is less than the error of direct conversion.

[0067] In this model, at least some of the weight parameter matrices can be the query Q, key K, and value V projection matrices in the attention mechanism, or large linear layers in the feedforward network. By applying a linear transformation (usually an orthogonal or approximately orthogonal rotation matrix), the transformed parameter distribution is made more favorable for low-precision representation, that is, the parameters are aligned to more advantageous positions on the quantization grid. This makes the overall error (such as mean square error) caused by the quantization process (mapping continuous values ​​to discrete integer points) smaller than the error caused by directly quantizing the original parameters. Thus, the accuracy and stability of low-precision inference are significantly improved without increasing runtime overhead.

[0068] Regarding term D-2, the massive amount of visual tokens generated by multimodal large models (such as visual encoders) during image processing significantly increases the computational complexity of subsequent attention, resulting in inference latency. Therefore, during the inference process of multimodal large models, the amount of data involved in model computation is reduced based on the relevance of the input data to the current task, thereby lowering inference latency.

[0069] Optionally, the amount of input data involved in the calculation can be reduced according to preset rules, including: when the multimodal large model processes the visual observation information, based on the importance assessment of each region in the visual observation information to the current task, retaining data regions with importance higher than a threshold for subsequent calculation.

[0070] For example, when performing the task of "tracking vehicles ahead," areas far from the road and background areas such as the sky are of lower importance; while the center of the road and areas where vehicles may appear are of higher importance. By calculating an importance score for each image region or visual token, only data regions with importance scores higher than a certain preset threshold (i.e., visual information highly relevant to the current task) are retained. This is equivalent to reducing the amount of data involved in the core model calculation without changing the model architecture and output interface, thereby reducing the computational complexity and latency of a single inference.

[0071] Regarding item D-3, when the large model generates text (such as control targets or tool call requests) in an autoregressive manner, it needs to decode each token sequentially, resulting in slow generation speed and difficulty in meeting real-time interaction requirements. Therefore, when the multimodal large model generates sequences, a retrieval or prediction mechanism is used to provide candidate sequences, which are then verified by the multimodal large model.

[0072] Specifically, from the memory storage module or a pre-built knowledge base, instruction templates, tool call formats, or descriptive phrases used in similar historical tasks are retrieved based on the current context to serve as candidate sequences for retrieval. Simultaneously, a lightweight draft model is used to quickly generate a predicted candidate sequence based on the same context. Both the retrieved and predicted candidate sequences are submitted to a multimodal large-scale model for validation. The multimodal large-scale model quickly determines which parts of the candidate information are reasonable and accepted (i.e., whether they conform to their own distribution). This allows a single inference process to skip the generation of multiple tokens, transforming the serial generation process into a parallel validation process, thereby significantly reducing the overall text generation latency.

[0073] The technical solution adopted in this embodiment achieves efficient and stable deployment of large models on resource-constrained edge platforms. By leveraging the basic computing power provided by a neural network processor and incorporating D-1 optimization operations, the feasibility and accuracy of complex multimodal large model inference are ensured while strictly controlling power consumption and cost. Furthermore, D-2 optimization operations dynamically focus on task-related visual information, eliminating redundant computations; and D-3 optimization operations shorten the language generation waiting time. Thus, the overall response speed of the large model agent module is improved by an order of magnitude, better meeting the real-time decision-making requirements of UAVs in dynamic environments.

[0074] In some embodiments, the accelerated inference module further includes: The backend management module is used to select the first computing unit for the first execution of model inference based on the type and status of the currently available computing resources when the system starts up, and to continuously monitor the inference status and performance indicators of the first computing unit during system operation. If the first computing unit detects an inference error or its real-time inference performance does not meet the requirements of the current task, the backend management module will switch the ongoing or subsequent inference task to the alternative second computing unit for execution.

[0075] In this embodiment, the backend management module acts as a scheduling center, responsible for dynamically allocating computing resources in a complex operating environment to ensure that the inference service of the large model agent can run continuously, stably, and efficiently, thereby ensuring the continuity and security of the UAV's autonomous mission execution.

[0076] The types of currently available computing resources can include the presence of Huawei NPUs (Neural Processing Units), Horizon Robotics NPUs, and CPUs (Central Processing Units). Based on the types of available computing resources and their initial health status, an optimal first computing unit is dynamically selected and designated for the initial execution of the model inference task. The selection strategy can comprehensively consider factors such as peak computing power, energy efficiency ratio, and support for specific precision. For example, if the lowest latency is required, a dedicated NPU with the highest peak computing power might be prioritized; if higher precision is required, a backend supporting higher precision (such as FP16) or with better optimization might be selected; if performance requirements are not high, to save power, a lightweight backend with better energy efficiency or an energy-saving CPU core might be switched to.

[0077] During inference based on the selected first computing unit, the inference status (whether it is running normally) and key performance indicators (such as single inference latency, throughput, processor temperature and load) of the first computing unit can be monitored. When any of the following situations are detected, it is determined that the first computing unit has an inference error or the real-time inference performance does not meet the performance requirements of the current task: (a) Inference error, the first computing unit returns a hardware or driver error code, experiences a kernel crash, or outputs obviously invalid (such as all zeros or NaN values); (b) Performance failure, the actual inference performance of the first computing unit (such as latency duration) exceeds the maximum threshold allowed by the task; (c) The processor triggers frequency reduction due to overheating, resulting in continuous performance degradation.

[0078] Once it is determined that an inference error has occurred in the first computing unit or that the real-time inference performance does not meet the performance requirements of the current task, the backend management module initiates a switching process: interrupting the inference task currently running on the first computing unit and switching the unfinished inference task to a pre-configured alternative second computing unit, which can be another NPU or GPU; resuming and continuing inference on the second computing unit to ensure that the decision-making process of the large model agent module is not interrupted.

[0079] The technical solution adopted in this embodiment enables the backend management module to achieve fault tolerance from static deployment to dynamic runtime. Through dynamic backend selection and failover mechanisms, it adapts to heterogeneous hardware platforms during initialization and continuously monitors performance and dynamically migrates tasks throughout the system's lifecycle. Therefore, the system can effectively cope with transient hardware failures, avoiding the risk of the entire large-scale intelligent agent module becoming paralyzed due to the failure of a single computing unit. This allows the UAV to operate reliably in more demanding and variable physical environments. The backend management module enables the system to dynamically allocate computing units based on real-time task requirements and system health status, achieving optimal utilization of computing resources.

[0080] like Figure 2 As shown, Figure 2 This application provides a schematic diagram of a backend management module and a model deployment pipeline. To efficiently deploy and run multimodal large models on a UAV-borne neural network processor, this application proposes a multi-level optimization strategy combination covering pre-deployment, inference, and runtime scheduling. The aim is to systematically reduce inference latency and power consumption, improve computational throughput, and effectively control accuracy loss. Specifically, the optimization strategy includes at least: Pre-deployment rotation matrix quantization (strategy 1): Before deploying a large multimodal model to a neural network processor, the model parameters are transformed to reduce the amount of storage and computation.

[0081] Inference Process Optimization (Strategies 2 & 3): Task-aware dynamic computation simplification and sequence generation acceleration. Specifically, token pruning configuration (Strategy 2): During the inference process of the multimodal large model, the amount of data involved in model computation is reduced based on the relevance of input data to the current task. Retrieval Speculative Sampling Configuration (Strategy 3): When the multimodal large model generates sequences, candidate sequences are provided using retrieval or prediction mechanisms, and the multimodal large model verifies these candidate sequences. This transforms some serial computation into parallel verification, effectively accelerating the serialization output process for language, planning, etc.

[0082] Runtime selection and rollback mechanism (strategy 4): When the system starts up, the backend management module selects the first computing unit for the first execution of model inference based on the type and status of the currently available computing resources. During system operation, the inference status and performance indicators of the first computing unit are continuously monitored. If an inference error is detected in the first computing unit or the real-time inference performance does not meet the performance requirements of the current task, the backend management module will switch the ongoing or subsequent inference task to the alternative second computing unit for execution.

[0083] In this way, a coordinated optimization of accuracy, speed, and power consumption is achieved. Strategy 1 ensures the baseline accuracy of low-precision inference at the parameter representation level; Strategies 2 and 3 dynamically simplify the computation process in the spatial (visual) and temporal (language) dimensions, respectively. The combination of these three strategies, under strictly limited edge resources, achieves a systematic improvement in the performance of large-model inference, meeting the stringent real-time requirements of UAV missions. Furthermore, the backend management mechanism of Strategy 4 transforms the single, static model deployment into a dynamic, elastic computing service, which can optimize resource allocation based on context. This allows the system to maintain the continuous operation of core intelligent agent functions even when facing uncertainties in real-world scenarios such as occasional processor failures, overheating and frequency throttling, or instantaneous overload, greatly enhancing the robustness and reliability of the entire system.

[0084] Understandably, the optimization strategies described above can also be applied to the inference computation of the neural network model in the real-time control module.

[0085] In some embodiments, the system further includes: The safety monitoring module is used to monitor the control commands and / or the flight status of the UAV according to preset safety rules, and to execute emergency control strategies in case of violation of safety constraints.

[0086] In this embodiment, a safety monitoring module is included in the system to ensure safety. This module operates with the highest priority, continuously monitoring control commands from the real-time control module and the UAV's flight status (position, speed, attitude, battery level, communication link, etc.).

[0087] The safety monitoring module contains preset safety rules, which are typically loaded during system initialization. These safety rules constitute multi-dimensional, quantifiable safety constraints, including: (a) spatial constraints, including geofencing, such as no-fly zones where drones are absolutely prohibited; (b) altitude restrictions, i.e., maximum flight altitude and minimum takeoff altitude; (c) boundary restrictions, i.e., the maximum horizontal flight range allowed by the mission; (d) dynamic constraints, including speed / acceleration restrictions; (e) attitude angle restrictions, including maximum pitch and roll angles to prevent stall or loss of control; (f) system state constraints, including battery safety thresholds, triggering a return-to-home when the battery level falls below a certain value, and immediately making an emergency landing when it falls below an even lower value; (g) communication link timeout, i.e. communication interruption with the ground station exceeding a specified time, which is considered a loss of contact; (h) sensor failure, i.e. abnormal or lost data from critical sensors; (i) obstacle avoidance constraints, including minimum obstacle avoidance distance, such as the distance to any obstacle must not be less than this value; and (j) emergency approach judgment, i. the speed and acceleration of an approaching obstacle exceeding a safety threshold.

[0088] The safety monitoring module runs a monitoring and decision-making cycle at a very high frequency (usually no less than that of the real-time control module): In each cycle, it synchronously reads the current control commands and flight status data, and compares and verifies the read data with all preset safety rules; if all data meet the safety constraints, the control commands are issued normally. If any safety constraint is detected to be violated or about to be violated, the normal issuance of control commands is interrupted; at this time, the module will enforce the preset highest-priority emergency control strategy according to the type and severity of the violation, and directly take over or override the current control commands. Among them, the emergency control strategy may include: (a) emergency hovering, that is, immediately stop all translational movements and maintain the current altitude and attitude; (b) automatic return to home, that is, trigger the return to the preset takeoff point or safety point and climb to a safe altitude; (c) active obstacle avoidance, that is, generate an instantaneous speed command opposite to the direction of the obstacle.

[0089] The technical solution adopted in this embodiment uses a safety supervision module to manage functional safety (avoiding dangers caused by system failures) and operational safety (avoiding dangers caused by agent decision-making errors) under a unified framework, which conforms to the design principles of high-reliability systems and provides the necessary compliance foundation for the application of fully autonomous UAVs in complex airspace.

[0090] like Figure 3 As shown, Figure 3This is a schematic diagram of the software architecture of an autonomous unmanned aerial vehicle (UAV) system based on a large-scale intelligent agent, provided in an embodiment of this application. This software system, through modular and layered decoupling design, organically integrates advanced cognition, real-time control, safety supervision, and support services. It can run collaboratively on the same processor or distributed computing nodes, forming a complete closed loop for autonomous task execution. Specifically, the software architecture includes the following core modules: 1. Large Model Intelligent Agent Module (Planning and Cognition Layer): This module serves as the core of the system's intelligent decision-making, operating at a relatively low planning frequency (first frequency), responsible for task understanding, decomposition, and high-level planning. It integrates: A multimodal encoder is used to perform feature fusion and unified encoding on heterogeneous data such as input natural language task instructions and visual observation information.

[0091] Multimodal large models (such as visual language models or large language models) serve as cognitive engines, receiving encoded multimodal contexts, performing instruction semantic parsing, logical reasoning, and task decomposition to generate macro-level task plans.

[0092] The spatial awareness and grounding component is used to spatially associate and "land" the abstract semantic description (such as "target object") output by the multimodal large model with the real-time visual image or environment map, and output a specific and operable spatial target representation, such as the target's region of interest in the image, its coordinates or heading in three-dimensional space.

[0093] The memory storage module is used to persistently store task execution history, key environmental observation frames, map semantic indexes, and tool call results, and supports historical information retrieval based on retrieval enhancement generation technology to provide experiential context for current decision-making.

[0094] The tool scheduler, based on the task planning results, is responsible for converting task intents into standardized tool call requests and coordinating the scheduling of external functional modules.

[0095] 2. Real-time Control Module (Execution Layer): This module operates independently of the agent module, running periodically at a high control frequency (second frequency, such as 20Hz-100Hz). It receives control objectives from the large model agent module, as well as real-time visual observation information and real-time UAV status information from sensors. This module employs generative or optimization methods such as diffusion strategies and model predictive control to directly output high-frequency, continuous, and smooth control commands (such as attitude and throttle commands). Its operation does not depend on the successive inference delay of the large model, ensuring the real-time performance and determinism of the control loop.

[0096] 3. Safety Monitoring Module (Safety Layer): This module continuously monitors the control commands output by the real-time control module and the actual flight status of the UAV. It has a built-in set of verifiable preset safety rules, including dynamic limitations (speed, attitude), spatial constraints (geofencing, no-fly zones), system state thresholds (battery level, communication link), and obstacle avoidance rules. Once any violation of safety constraints or impending violation is detected, this module has the authority to immediately cut or reject control commands, or directly trigger and execute preset emergency control strategies (such as emergency hovering, automatic return to home, or forced landing), thereby ensuring absolute flight safety.

[0097] 4. Tool Invocation Interface Layer (Capability Abstraction Layer): This layer encapsulates a series of professional UAV-related functional tools using standardized application programming interfaces (APIs), providing unified and transparent capability invocation services to the upper-layer intelligent agent modules. The encapsulated functional tools include, but are not limited to: environmental perception and modeling tools, navigation and motion control tools, target interaction tools, safety and mission-specific tools, and interaction and reporting tools.

[0098] 5. Accelerated Inference Module (Computation Acceleration Layer): This module is specifically designed to address the computational challenges of deploying large models on the edge. Its core functions include: Model Deployment Optimization, which is responsible for fixed-point quantization (such as INT8 conversion), operator adaptation, and accuracy optimization configuration of the policy network in the multimodal large model and real-time control module; Runtime Management, which provides a unified inference interface and has runtime backend selection and fault rollback mechanisms, enabling dynamic scheduling or switching of inference tasks among available neural network processors or general-purpose computing units based on system status, ensuring continuous, efficient, and stable inference tasks.

[0099] 6. Map / Positioning Module: Continuously provides high-precision real-time pose estimation for UAVs, and can build and maintain environmental maps online for navigation and planning.

[0100] 7. Task Memory and Retrieval Module: This module provides structured storage of end-to-end task data (plans, observations, operation records, and spatial semantic annotations) and supports rapid retrieval based on multimodal conditions. It is crucial for achieving task continuity, experience reuse, and context awareness.

[0101] 8. Log and Replay Module: Synchronously records raw sensor data, internal status of each module, decision flow and key events throughout the entire system lifecycle, supports accurate scene playback and data analysis, and is used for system debugging, performance optimization, security auditing and algorithm iteration.

[0102] Through the aforementioned software architecture, this application resolves the fundamental contradiction between the long latency of complex cognitive reasoning and the hard real-time requirement of safe control on a resource-constrained UAV edge platform. The modules collaborate through clearly defined interfaces, achieving a complete autonomous capability closed loop from receiving natural language commands to generating safe, reliable, and adaptive physical actions, thus combining intelligence, security, and engineering practicality.

[0103] like Figure 4 As shown, Figure 4 This is a schematic diagram of the hardware architecture of an autonomous unmanned aerial vehicle (UAV) system based on a large-scale intelligent agent, provided in an embodiment of this application. This hardware platform provides a specialized physical foundation for edge-side intelligence, and its core design aims to achieve the organic integration of high-performance AI computing, multimodal accurate perception, and reliable flight control. The hardware architecture specifically includes the following components: 1. Flight platform: Usually a multi-rotor drone, providing the flight carrier and power system.

[0104] 2. Flight Control Module: An independent, microcontroller-based, highly reliable control unit used to receive high-level control commands and perform low-level closed-loop control and fault protection for the UAV's attitude, speed, and position. It can use an STM32H7 series MCU or an equivalent one, running proven flight control firmware (such as PX4). 3. Onboard Computing Unit: The core of edge intelligence, it is an embedded high-performance computing module integrating a main processor and large-capacity memory. It can be optionally equipped with one or more neural network processors (such as Huawei Ascend NPU, Horizon Robotics Journey NPU, Rockchip RK series NPU), specifically used to accelerate the inference calculation of neural network models in large-model intelligent agent modules and real-time control modules, solving the edge computing power bottleneck.

[0105] 4. Multi-channel visual perception module: Includes a binocular vision unit and / or a depth vision unit, supporting 2-6 camera inputs (such as forward-looking, downward-looking, and gimbal cameras). This design can obtain 3D information of the environment through stereo vision or active depth sensing without relying on LiDAR, providing crucial visual observation information for obstacle avoidance, navigation, and scene understanding.

[0106] 5. Status Awareness Module: Integrates multiple sensors such as inertial measurement unit, global navigation satellite system receiver (optional RTK), magnetometer, barometer, and altimeter to provide real-time and accurate status information (position, speed, attitude, etc.) of the UAV.

[0107] 6. Communication module and human-machine interaction module: Supports multiple communication methods such as Wi-Fi, data transmission, 4G / 5G, etc., to realize the reception of user natural language commands, flight monitoring and real-time transmission of mission data.

[0108] 7. Storage and Data Recording Module: Used to store system software, mission data, flight logs, constructed environment maps, memory, and automatically generated reports, providing a data foundation for mission review, system optimization, and security auditing.

[0109] Through the aforementioned hardware architecture, this application achieves deep integration of dedicated AI computing (NPU), low-cost 3D visual perception, and highly reliable flight control. In particular, the optional configuration of a domestically produced NPU and lidar-free depth perception directly addresses the core challenges of productization in terms of computing power cost, power consumption, supply chain security, and overall system cost. This hardware architecture provides a stable, efficient, and scalable physical operating environment for the aforementioned software systems, together forming a complete and feasible UAV system solution capable of "driving complex autonomous tasks with natural language."

[0110] This application also provides an autonomous unmanned aerial vehicle (UAV) method based on a large-model intelligent agent, referring to... Figure 5 As shown, Figure 5 This is a flowchart illustrating the steps of an autonomous unmanned aerial vehicle (UAV) method based on a large-scale intelligent agent, as provided in an embodiment of this application. The method includes steps S510 to S550: Step S510: Obtain natural language task instructions.

[0111] The natural language task instructions can be voice instructions (converted to text by an onboard or cloud-based speech recognition module) or directly input text instructions. For example, "Inspect the cooling tower on the southeast side of the factory building, focusing on photographing the welded seams of pipes that may be corroded."

[0112] Step S520: The large model agent module, running at a first frequency, performs task understanding and planning based on natural language task instructions and visual observation information to obtain task planning results; wherein, the task planning results include at least a task sequence and control objectives for each task in the task sequence, or may also include tool invocation requests; the control objectives characterize the executable state requirements of the task in the current environment.

[0113] Specifically, the multimodal large model (such as VLM) within the large model agent module performs semantic parsing of natural language instructions and combines current visual observation information to understand the environmental state. Subsequently, the complex overall task logic is decomposed into an ordered, executable task sequence; at the same time, control objectives are generated for each task in the task sequence; or tool call requests can be generated for each task.

[0114] In this context, a control objective is a high-level, abstract state requirement that characterizes the executable state requirements of a task in the current environment. For example, "navigate the drone to an airspace with a radius of R meters centered on the geographic coordinates." A tool invocation request, on the other hand, is a structured command used to request the invocation of specific external functional tools to assist in completing a task. For instance, a request to the "global path planning tool" along with the target coordinates might be generated to achieve the "navigation" objective; similarly, a request to the "target detection tool" might be generated to "identify a cooling tower."

[0115] Step S530: If the task planning result also includes the tool call request, call the functional tool encapsulated in the tool call interface layer to perform task-related operations and obtain the execution result.

[0116] Specifically, the tool call interface layer receives tool call requests from step S520, and calls the corresponding encapsulated specialized function tools at the underlying level to perform specific calculations or control operations based on the type and parameters of the request. For example, it calls a localization and modeling tool to obtain its own precise pose, and calls a target detection / tracking tool to identify and lock onto a "cooling tower" in an image.

[0117] After the invoked functional tool completes its execution, it returns the results (such as detected target coordinates, planned waypoint sequences, etc.) through the tool call interface layer. These results are specific, quantitative data used to update the system's understanding of the environment and guide the next action. Specifically, these execution results are fed back to the large model agent module (for subsequent iterative planning in step S520) and also directly provided to the subsequent real-time control module.

[0118] Step S540: The control target or the control target and the execution result are obtained by the real-time control module running at the second frequency, and control commands for controlling the flight of the UAV are generated by combining real-time visual observation information and real-time UAV status information; the second frequency is greater than the first frequency.

[0119] Specifically, the real-time control module, acting as the cerebellum of the system, operates independently at a second frequency (e.g., 20Hz, 50Hz, 100Hz) higher than the first frequency. In some embodiments, if no tool invocation is required, the control target, real-time visual observation information, and real-time UAV status information are fused within each control cycle, and control commands for UAV flight are generated through an internal controller or policy network. In other embodiments, if tool invocation is required, the control target, execution result, real-time visual observation information, and real-time UAV status information are fused, and control commands for UAV flight (such as attitude angle commands and motor throttle commands) are generated through an internal controller or policy network.

[0120] Step S550: Control the drone to fly according to the control command.

[0121] Specifically, the flight control module executes the control commands generated in step S540, which drive the motors, control surfaces and other actuators to cause the UAV to produce actual attitude and position changes, thereby gradually completing the task sequence planned in step S520.

[0122] In some embodiments, prior to step S550, the method further includes: monitoring the control commands and / or the flight status of the UAV according to preset safety rules, and executing emergency control strategies in the event of a violation of safety constraints, so as to ensure absolute flight safety.

[0123] In some embodiments, after step S550, the method further includes: determining task completion and outputting results. When the task completion conditions are met (e.g., completion of the shooting list, completion of inspection points, target loss exceeding a threshold, user termination, etc.), the task result is output. Specifically, this includes: generating a structured report (inspection defect list, location, evidence images / videos, timestamps); and outputting video clip / camera movement metadata.

[0124] It is important to emphasize that steps S520 to S550 constitute a dynamic, closed-loop iterative process. As the UAV executes flight commands (step S550), the environmental state changes, and new visual observation information and tool execution results (such as "target lost" or "mission completed") are continuously generated. This feedback information is input into the next round of step S520, enabling the large model agent module to reassess and adjust its mission planning, control objectives, and tool invocation requests based on the latest environmental state and mission progress, thereby achieving autonomous adaptation and dynamic correction during mission execution.

[0125] The technical solution of this embodiment clearly decouples the complex cognitive reasoning process with uncertain time delays from the flight control process requiring high-frequency deterministic time delays by setting large-model intelligent agent modules and real-time control modules that operate at different frequencies. Through interaction with the control target, the powerful cognitive capabilities of the large model can be utilized while fundamentally avoiding the stability and safety risks caused by its inference delays directly entering the control closed loop, achieving an organic unity of slow thinking and fast action. The tool call interface layer encapsulates various dedicated functional tools (such as localization, target detection, 3D reconstruction, etc.), enabling the large-model intelligent agent module to invoke these functional tools to perform task-related operations through tool call requests, thereby reducing the complexity of the intelligent agent model.

[0126] The following sections will further elaborate on the implementation process of the autonomous UAV method based on large model intelligent agents proposed in this application, using two specific application scenarios to fully demonstrate the technical details and broad application potential of this application.

[0127] Example 1: Autonomous shooting with camera movement and dynamic target tracking.

[0128] This embodiment demonstrates the system's application in scenarios such as film and television shooting and security surveillance. The drone hardware configuration includes: a Pixhawk-type flight control module (e.g., based on an STM32H743 MCU, running PX4 firmware); an integrated edge computing module as the onboard computing unit, which has a CPU and can optionally be equipped with a domestic NPU (e.g., Rockchip RK3588 with built-in NPU, Horizon Robotics Journey series, or Huawei edge NPU module); multiple cameras (multi-channel visual perception modules), including at least a gimbal main camera for shooting, a forward-looking binocular camera for depth perception and obstacle avoidance, and an optional downward-looking auxiliary camera; a sensor suite including an IMU, barometer, GNSS (optional RTK), and altimeter; and a communication module supporting data transmission, Wi-Fi, or 4G / 5G. This hardware platform runs the complete software architecture described in this application, including a large-scale intelligent agent module, a real-time control module, a tool interface layer, and a security monitoring module.

[0129] The system operating parameters can be configured as follows: the real-time control module output frequency is 50Hz (range 20-100Hz); the multi-channel visual perception module is a forward-looking binocular camera at 640×480@30fps; the main camera recording is 1080p or higher resolution; the obstacle avoidance safety distance is 1.5 meters; the maximum tracking speed is 6m / s; the target tracking distance range is 5-15 meters; large model inference runs on the NPU, and token pruning and speculative sampling are enabled to optimize latency.

[0130] like Figure 6 The method flowchart shown in this embodiment includes the following steps in its execution process: Step E1-1: Task Input: The user inputs natural language task instructions via voice or text, such as: "Follow the target vehicle ahead, maintaining a relative distance of 10 meters and a height of 8 meters; automatically detour around pedestrians or obstacles; circle the target clockwise once and then return to the takeoff point and land."

[0131] Step E1-2: Understanding and Task Decomposition: Parsing natural language task instructions into structured subtasks: Subtask A: Target acquisition and locking (calling target detection / tracking tools); Subtask B: Relative pose preservation (output relative distance / orientation / height constraints); Subtask C: Orbital shooting (generating orbital direction, angular velocity, duration / number of orbits); Subtask D: Return to Home and Landing (call map navigation tools or flight control return to home tools).

[0132] It also outputs the tool call parameters and success criteria (target loss threshold, orbit completion conditions, etc.).

[0133] Step E1-3: Target grounding and tracking tool invocation: The large model agent module invokes the target detection / segmentation tool to locate the target region on the main view or forward-looking camera; invokes the multi-target tracking tool to output the target ID and its motion state in consecutive frames; if the target is briefly occluded, the large model agent module can invoke the memory module to retrieve the target appearance features and assist in re-identification.

[0134] Step E1-4: Depth estimation and local spatial understanding: If a depth camera is present, use the depth map directly; if a stereo camera is used, generate depth / parallax through a stereo depth estimation model; form a local obstacle representation (which can be a point cloud, depth raster, or equivalent form) without using LiDAR, as the obstacle avoidance input for the real-time control module.

[0135] Steps E1-5: Real-time trajectory generation and obstacle avoidance: The real-time control module reads at 50Hz the following: the control target, UAV status information (position / velocity / attitude / heading), local obstacle representations (visual observation information), and the execution results of the tool. Based on these inputs, the module (e.g., using a diffusion strategy) calculates smooth, proactive obstacle avoidance control commands in real time.

[0136] Steps E1-6: Gimbal and Composition Control (Tool Call): The large model agent module calls the gimbal control tool to output the gimbal angle / angular velocity according to composition rules (such as the rule of thirds, keeping the target slightly above the center of the image, keeping the horizon, etc.); and can call the camera parameter tool to adjust zoom and exposure.

[0137] Steps E1-7: Safety monitoring and emergency response: If an obstacle is detected to be approaching or the location is abnormal, the safety monitoring module will prioritize obstacle avoidance or hovering; if the battery level is below the threshold or there is an unrecoverable anomaly, the module will trigger a return to home or an emergency landing (the landing point will be provided by the tool evaluation module as a candidate area).

[0138] Steps E1-8: Mission End and Result Output: After completing the orbit and return landing, output the video file and the camera movement trajectory metadata (timestamp, pose, gimbal angle, etc.); and write the effective cue template, target features, and tool parameters of this mission into the memory bank to provide small sample transfer capability for subsequent similar shooting missions.

[0139] Example 2: Automated Industrial Inspection. This example focuses on the automated inspection of industrial facilities such as substations, pipelines, and fans. The drone utilizes an industrial-grade platform equipped with windproof and redundancy systems. Its onboard computing unit supports domestically produced NPUs (Huawei / Horizon Robotics / Rockchip) and employs edge-side optimization technologies such as rotation matrix quantization, token pruning, and speculative sampling. For sensors, it is equipped with a forward-looking binocular / multi-view camera for obstacle avoidance and positioning, a gimbal zoom camera for detailed image capture, and optional RTK GNSS and thermal imagers. The software integrates dedicated tools such as SLAM / VINS, map navigation, defect identification, and report generation.

[0140] like Figure 7 The method flowchart shown in this embodiment includes the following steps in its execution process: Step E2-1: Small Sample Task Transfer: To reduce reliance on large amounts of labeled data, the system employs a small sample transfer mechanism. Users provide a small number of examples (e.g., 2–10 defect example images or text descriptions of defects and threshold requirements) as the basis for small sample transfer. For example, the few defect examples provided by the user are converted into structured prompt templates to directly guide the agent's attention and judgment logic; or the example features are stored in a memory bank, and the sensitivity to similar defects is improved in actual detection through retrieval enhancement methods. For specific high-value defect types, parameter efficient adaptation technology (such as LoRA) can also be used to perform lightweight fine-tuning of the model at the ground station before being deployed to the UAV, achieving rapid upgrades to accurate model capabilities.

[0141] Step E2-2: Generate inspection plan and route skeleton: Before the inspection, the user gives the task in natural language, such as: "Inspect the roof of Plant No. 3, focusing on taking pictures of the cooling tower fan and pipe valves; if leakage, rust or foreign object accumulation is found, mark the location and take 3 close-up pictures; return and generate a report after completion."

[0142] The large model intelligent agent module performs task understanding and planning based on task description, historical map / last inspection memory, on-site no-fly zone and safe altitude, and obtains a set of inspection points (which can be semantic points: fans, valves, pipe interfaces, etc.), flight path skeleton (which can be a segmented target point sequence or area coverage strategy), shooting list and parameters (zoom magnification range, shooting angle, exposure strategy), and defect triggering rules (such as confidence threshold, number of repeated verifications, etc.).

[0143] Step E2-3: Tool Invocation for Localization and Navigation: The large model agent module invokes localization and modeling tools to build / update local maps and poses, and invokes the map navigation module to navigate between semantic points.

[0144] Step E2-4: On-site shooting and defect identification: After arriving at each preset point, the intelligent agent schedules the gimbal and camera tools to automatically execute a multi-angle shooting sequence, and simultaneously calls the defect identification tool to analyze the real-time image to identify anomalies such as cracks, corrosion, and leakage.

[0145] Step E2-5: Safety Supervision and Emergency Response Strategy: The entire inspection process is protected by a safety supervision module to deal with anomalies such as strong winds and positioning drift, ensuring operational safety.

[0146] Step E2-6: Report Generation (Product Output): After the task is completed, the system automatically calls the report generation tool to summarize the inspection track, the completion status of all inspection points, and the list of defects found (with type, confidence level, precise location coordinates and evidence images / videos), forming a structured inspection report that can be delivered directly, and uploading it to the back-end management system through the communication link.

[0147] The two embodiments above specifically demonstrate how the proposed solution seamlessly integrates the advanced cognitive capabilities of large-scale intelligent agents into actual UAV operations through tool call interfaces and a fast-slow collaborative architecture. Embodiment 1 demonstrates the system's ability to complete complex and aesthetically pleasing interactive tasks in dynamic, unstructured environments; Embodiment 2 demonstrates the system's ability to achieve automated and intelligent operations and data product delivery under professional, process-oriented conditions. Both embodiments collectively demonstrate the high versatility, strong adaptability, and significant productization value of the proposed technical solution, solving a series of autonomous challenges in real-world tasks ranging from creative shooting in open scenes to rigorous industrial inspections.

[0148] This application also provides an electronic device, see embodiments thereof. Figure 8 , Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 8 As shown, the electronic device 800 includes a memory 810 and a processor 820. The memory 810 and the processor 820 are connected via a bus for communication. The memory 810 stores a computer program that can run on the processor 820 to implement the steps of the autonomous unmanned aerial vehicle method based on a large model intelligent agent described in the embodiments of this application.

[0149] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the autonomous unmanned aerial vehicle method based on a large model intelligent agent described in this application.

[0150] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the autonomous unmanned aerial vehicle method based on a large model intelligent agent as described in this application.

[0151] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0152] This application describes embodiments of methods and apparatus according to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0156] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0157] The above provides a detailed description of an autonomous unmanned aerial vehicle system and method based on a large model intelligent agent provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An autonomous unmanned aerial vehicle (UAV) system based on a large-scale intelligent agent model, characterized in that, include: The large model intelligent agent module runs at the first frequency and is used to perform task understanding and planning based on natural language task instructions and visual observation information to obtain task planning results. The task planning results include at least a task sequence and the control objectives of each task in the task sequence, or may also include tool invocation requests. The control objectives characterize the executable state requirements of the task in the current environment. The tool call interface layer encapsulates multiple functional tools. In response to the tool call request, it calls the corresponding functional tool to perform task-related operations and feeds back the execution results to the large model intelligent agent module and / or the real-time control module. The real-time control module operates at a second frequency greater than the first frequency, and is used to acquire the control target or acquire the control target and the execution result, and generate control commands to control the flight of the UAV by combining real-time visual observation information and real-time UAV status information.

2. The system according to claim 1, characterized in that, The system also includes an accelerated inference module, which provides hardware-accelerated computing for the multimodal large model in the large model agent module; The accelerated inference module includes at least one neural network processor and performs at least one of the following optimization operations for the multimodal large model: Before deploying the multimodal large model to the neural network processor, the model parameters are subjected to numerical precision transformation to reduce storage and computational load. During the inference process of the multimodal large model, the amount of data involved in the model calculation is reduced based on the correlation between the input data and the current task. When the multimodal large model generates sequences, it uses a retrieval or prediction mechanism to provide candidate sequences, and the multimodal large model verifies the candidate sequences.

3. The system according to claim 2, characterized in that, The numerical precision transformation of the model parameters includes: Perform a linear transformation on at least some of the weight parameter matrices of the model, such that the error of the transformed parameters when converted to a low-precision numerical representation is less than the error of direct conversion; The amount of input data used in the calculation is reduced according to preset rules, including: When the multimodal large model processes the visual observation information, it evaluates the importance of each region in the visual observation information to the current task and retains data regions with importance higher than the threshold for subsequent calculations.

4. The system according to claim 2, characterized in that, The accelerated inference module also includes: The backend management module is used to select the first computing unit for the first execution of model inference based on the type and status of the currently available computing resources when the system starts up, and to continuously monitor the inference status and performance indicators of the first computing unit during system operation. If the first computing unit detects an inference error or its real-time inference performance does not meet the requirements of the current task, the backend management module will switch the ongoing or subsequent inference task to the alternative second computing unit for execution.

5. The system according to any one of claims 1-4, characterized in that, The tool that calls the function tools encapsulated in the interface layer includes at least one of the following: Airborne perception and modeling tools, including localization and modeling tools for acquiring the position and attitude of the UAV itself, and detection and tracking tools for identifying and tracking specific targets from visual observation information; Airborne environmental understanding tools include 3D reconstruction tools for generating 3D structural information of the environment, and forced landing point assessment tools for evaluating safe landing areas; An airborne payload control tool is used to control the actuators carried by the UAV, the actuators including a gimbal control tool, a lighting tool, and a cargo release tool; Cloud-based collaboration tools are used to invoke computing tools or data services deployed on cloud servers when the drone's communication module is connected to the network.

6. The system according to any one of claims 1-4, characterized in that, The large model intelligent agent module also includes: The memory storage module is used to store historical information during task execution and the execution results; The large model intelligent agent module is also used to perform task understanding and planning based on the historical information, natural language task instructions, visual observation information and laser information to obtain a task sequence, as well as the control objectives and tool call requests of each task in the task sequence. The laser information is collected by a laser device or predicted by the large model intelligent agent module; and / or, it is used to update the control objectives based on the execution results.

7. The system according to any one of claims 1-4, characterized in that, The system also includes: The safety monitoring module is used to monitor the control commands and / or the flight status of the UAV according to preset safety rules, and to execute emergency control strategies in case of violation of safety constraints.

8. The system according to any one of claims 1-4, characterized in that, The system also includes: A multi-path visual perception module, including a binocular vision unit and / or a depth vision unit, is used to provide visual observation information; The status awareness module is used to provide drone status information; The flight control module is used to control the flight of the UAV according to the control commands.

9. The system according to claim 6, characterized in that, The large model agent module is further configured to: upon receiving example information similar to the current task, incorporate the example information into the task understanding and planning process via a few-shot transfer unit to adjust the control objective or the tool invocation request; wherein the few-shot transfer unit is configured to perform at least one of the following operations: The example information is converted into a structured prompt template to guide target identification and task execution strategies; The example information is indexed into the memory storage module, and during subsequent task execution, if an appearance or scene similar to the example information is detected, the detection threshold is adjusted. Based on the example information, the model parameters related to the task of the large model agent module are updated. The update includes at least one of the following: low-rank parameter update and full fine-tuning of some modules.

10. An autonomous unmanned aerial vehicle (UAV) method based on a large model intelligent agent, characterized in that, include: Obtain natural language task instructions; The large model agent module, running at the first frequency, performs task understanding and planning based on natural language task instructions and visual observation information to obtain task planning results. The task planning results include at least a task sequence and control objectives for each task in the task sequence, or may also include tool invocation requests. The control objectives characterize the executable state requirements of the task in the current environment. If the task planning result also includes the tool invocation request, the functional tool encapsulated in the tool invocation interface layer is invoked to perform task-related operations and obtain the execution result; The real-time control module, operating at a second frequency, acquires the control target or acquires the control target and the execution result, and generates control commands for controlling the drone's flight by combining real-time visual observation information and real-time drone status information; the second frequency is greater than the first frequency. Control the drone to fly according to the control commands.

Images