Unmanned aerial vehicle control method and system based on visual language action model

Abstract

This application provides a method and system for controlling unmanned aerial vehicles (UAVs) based on a visual language action model, relating to the field of UAV control technology. The method includes: acquiring task instructions described in natural language; generating a semantic guidance vector by reasoning based on current environmental visual observation information and the task instructions using a semantic understanding system running at a first frequency, and writing the semantic guidance vector into shared memory; wherein the semantic guidance vector is used to represent task intent and environmental semantics, and the semantic understanding system is constructed based on a visual language action model; reading the most recently written semantic guidance vector from the shared memory using an action generation system running at a second frequency, and generating flight control quantities based on the latest semantic guidance vector, real-time environmental visual observation information, and real-time UAV state information; wherein the second frequency is higher than the first frequency; and controlling the UAV to perform a flight mission according to the flight control quantities.

Description

Technical Field

[0001] This application relates to the field of unmanned aerial vehicle (UAV) control technology, and in particular to a UAV control method and system based on a visual language action model. Background Technology

[0002] Traditional autonomous flight control systems for unmanned aerial vehicles (UAVs) typically employ a modular, sequential architecture, executing environmental perception, state estimation, path planning, and flight control in turn. While such methods are stable in specific scenarios, they heavily rely on pre-defined rules and lack the ability to understand the semantics of high-level natural language commands, making it difficult to perform open-scenario tasks such as "following a pedestrian in red" or "searching the area near a park bench."

[0003] With the development of Vision Language Action (VLA) models, it has become possible to directly control drones using natural language commands. Related technologies typically employ a single, parameter-intensive VLA model to synchronously process visual observation, language understanding, and action generation in an end-to-end manner. However, drone flight control demands extremely high real-time performance in closed-loop responses. On airborne embedded platforms with severely limited computing power and power consumption, this tightly coupled, single-model architecture has a fundamental flaw: the computational overhead and time delay of a single model inference operation severely limit the frequency of action generation, making it impossible to match the control frequency required for the drone's rapid dynamic changes. This creates a direct conflict between high-level semantic understanding and low-level high-frequency control, making it difficult for the system to accurately understand complex commands while ensuring real-time performance and stability during flight. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a drone control method and system based on a visual language action model, so as to overcome the above problems or at least partially solve the above problems.

[0005] A first aspect of this application discloses a drone control method based on a visual language action model, the method comprising: Task instructions for obtaining natural language descriptions; The semantic understanding system, operating at a first frequency, infers based on current environmental visual observation information and the task instructions, generates a semantic guidance vector, and writes the semantic guidance vector into shared memory; wherein, the semantic guidance vector is used to represent the task intent and environmental semantics, and the semantic understanding system is constructed based on a visual language action model; The action generation system, operating at a second frequency, reads the latest semantic guidance vector recently written from the shared memory and generates flight control variables based on the latest semantic guidance vector, real-time environmental visual observation information, and real-time UAV status information; wherein the second frequency is higher than the first frequency. The UAV is controlled to perform flight missions based on the flight control parameters.

[0006] Optionally, the semantic understanding system includes a first visual encoder, a text encoder, a large language model, and a feature projection layer; it performs inference based on the current environmental visual observation information and the task instructions to generate a semantic guidance vector, including: The first visual encoder encodes the current environmental visual observation information to obtain the first visual feature; The task instructions are encoded using the text encoder to obtain text features; The first visual feature and the text feature are fused and inferred across modally using the large language model to obtain high-level semantic features; The semantic guidance vector is generated by mapping the high-level semantic features through the feature projection layer.

[0007] Optionally, when the task instruction includes multiple levels of sub-tasks, generating a semantic guidance vector by inferring based on the current environmental visual observation information and the task instruction further includes: The task instructions are decomposed into multiple task nodes that are executed sequentially; each task node is associated with a specific task action and environmental features. Generate a semantic guidance vector corresponding to the target task node to be executed; Based on the current environmental visual observation information, if it is confirmed that the task action corresponding to the target task node has been completed, switch to the next task node and generate a semantic guidance vector corresponding to the next task node.

[0008] Optionally, the motion generation system includes a motion expert network, a second visual encoder, and a state encoder; based on the latest semantic guidance vector, real-time environmental visual observation information, and real-time UAV state information, it generates flight control variables, including: The real-time environmental visual observation information is encoded by the second visual encoder to obtain the second visual feature; The state encoder encodes the real-time UAV state information to obtain state features; The latest semantic guidance vector, the second visual feature, and the state feature are used as the generation conditions of the action expert network. The action expert network is then used to perform denoising and generation processes to generate a trajectory sequence. The flight control quantity is generated based on the trajectory points corresponding to the first M time steps in the trajectory sequence; where M is a positive integer greater than or equal to 1.

[0009] Optionally, the UAV control method based on a visual language action model according to claim 1, characterized in that reading the latest semantic guidance vector most recently written from the shared memory includes: Between two runs of the semantic understanding system, the action generation system runs multiple times and repeatedly reads the latest semantic guidance vector most recently written by the semantic understanding system.

[0010] Optionally, the semantic understanding system and the action generation system are trained through the following steps: Using the first training sample, the motion generation system to be trained is trained to learn the flight dynamics characteristics and obstacle avoidance priors of the UAV, and a preliminary trained motion generation system is obtained; wherein, the first training sample includes environmental visual observation information, UAV state information and real trajectory sequence. Using the second training sample, the semantic understanding system to be trained is fine-tuned with visual commands to learn the UAV mission perspective and mission command semantics, thus obtaining a preliminarily trained semantic understanding system; wherein, the second training sample includes aerial view images, mission commands, and the real trajectory sequence corresponding to the mission commands; Using the third training sample, the initially trained semantic understanding system and the initially trained action generation system are jointly fine-tuned to obtain the trained semantic understanding system and the trained action generation system; wherein, the third training sample includes: task instructions containing multi-level tasks, environmental visual observation information, and the real trajectory sequence corresponding to the task instructions. Based on reinforcement learning, the trained action generation system is optimized to obtain a fully trained action generation system.

[0011] Optionally, based on reinforcement learning methods, the trained action generation system is optimized to obtain a fully trained action generation system, including: For a given target state, multiple candidate trajectory sequences are generated through the action expert network of the trained action generation system; wherein, the target state includes environmental visual observation information, UAV status information, and mission instructions; Construct a composite reward function to calculate the score for each candidate trajectory sequence; wherein, the composite reward function includes at least: the proximity reward between the trajectory endpoint corresponding to the candidate trajectory and the target position indicated by the task instruction; Based on the scores of multiple candidate trajectory sequences, the relative advantage signal of each candidate trajectory sequence is calculated, and the parameters of the trained action generation system are updated according to the relative advantage signal to obtain the trained action generation system.

[0012] Optionally, based on the scores of multiple candidate trajectory sequences, a relative advantage signal for each candidate trajectory sequence is calculated, including: Based on the scores of multiple candidate trajectory sequences, the average score and standard deviation of the multiple candidate trajectory sequences are calculated; The difference between the score of each candidate trajectory sequence and the average score is calculated and normalized using the standard deviation to obtain the relative advantage signal of each candidate trajectory sequence.

[0013] Optionally, the parameters of the trained action generation system are updated based on the relative advantage signal, including: Candidate trajectory sequences with positive relative advantage signals are identified as advantageous trajectory sequences, and candidate trajectory sequences with negative relative advantage signals are identified as disadvantageous trajectory sequences. The parameters of the trained action generation system are updated with the goal of increasing the generation probability of the advantageous trajectory sequence and decreasing the generation probability of the disadvantageous trajectory sequence.

[0014] A second aspect of this application discloses a drone control system based on a visual language action model, the system comprising: The communication module is used to receive task instructions described in natural language. Visual sensors are used to collect real-time visual observation information about the environment. Inertial measurement unit (IMU) is used to sense the UAV's status information in real time. The onboard computing unit includes a first processor and a second processor; The first processor is configured to run a semantic understanding system at a first frequency to process current environmental visual observation information from the visual sensor and task instructions from the communication module, and to generate a semantic guidance vector, which is used to characterize task intent and environmental semantics. The semantic understanding system is built based on a visual language action model. The second processor is configured to run the action generation system at a second frequency to read the latest semantic guidance vector recently written from shared memory and generate flight control variables based on the latest semantic guidance vector, real-time environmental visual observation information, and real-time UAV status information; wherein the second frequency is higher than the first frequency; Shared memory, connected between the first processor and the second processor, serves as a data buffer interface for storing and transmitting semantic guidance vectors generated by the semantic understanding system; A flight controller, connected to the onboard computing unit, is used to control the UAV to perform flight missions based on the flight control inputs.

[0015] A third aspect of this application discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the UAV control method based on a visual language action model described in the first aspect of this application.

[0016] A fourth aspect of this application discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the unmanned aerial vehicle control method based on a visual language action model as described in the first aspect of this application.

[0017] A fifth aspect of this application discloses a computer program product, including a computer program that, when executed by a processor, implements the steps of the UAV control method based on a visual language action model as described in the first aspect of this application.

[0018] The embodiments of this application have the following advantages: In this embodiment, by decoupling the semantic understanding system and the action generation system at different frequencies and utilizing shared memory to transmit semantic guidance vectors, computationally intensive high-level semantic reasoning does not need to be completed synchronously in each control cycle. The action generation system can operate independently at a higher frequency than the semantic understanding system, and only needs to read the latest semantic guidance vector cached in shared memory to quickly generate flight control quantities. This significantly reduces control latency while ensuring that complex natural language commands are accurately understood, effectively meeting the stringent real-time response requirements of UAV flight, thereby improving flight stability and safety.

[0019] The architecture of dual-system asynchronous operation is naturally adapted to heterogeneous computing platforms, enabling optimized deployment and efficient utilization of computing resources (for example, a semantic understanding system with a large number of parameters and relatively high inference latency can be deployed on an NPU or GPU, while a lightweight, low-latency action generation system can be deployed on a CPU or DSP). This mechanism isolates the fluctuations or latency that may occur in the semantic understanding process from the control closed loop, avoiding instruction interruptions or jumps caused by high-level inference uncertainties, and enhancing the smoothness and fault tolerance of the overall system operation.

[0020] Furthermore, the semantic guidance vector, serving as the data interface connecting the semantic understanding system and the action generation system, encapsulates the abstract task intent, thus decoupling the semantic generation and action execution processes. This design enhances the system's modularity and maintainability, enabling independent upgrades and replacements of either the semantic understanding system or the action generation system without requiring overall reconstruction. Simultaneously, the compact vectorized representation reduces communication overhead between the semantic understanding system and the action generation system, providing a feasible foundation for implementing complex visual language interaction functions on computationally and power-constrained airborne embedded platforms. Attached Figure Description

[0021] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the steps of a drone control method based on a visual language action model, as provided in an embodiment of this application. Figure 2 This is an overall architecture diagram of a drone control method based on a visual language action model provided in an embodiment of this application; Figure 3 This is an asynchronous collaborative reasoning sequence diagram of a semantic understanding system and an action generation system provided in an embodiment of this application; Figure 4 This is a training flowchart of a semantic understanding system and an action generation system provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a drone control system based on a visual language action model provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To make the above-mentioned objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] To clearly illustrate the technical contributions of this application, the relevant technical solutions are briefly summarized and analyzed below.

[0025] Traditional modular architecture: This solution decomposes the autonomous flight mission of the UAV into a series of independent modules processed sequentially, including environment modeling based on SLAM (Simultaneous Localization and Mapping) / VIO (Visual-Inertial Odometry), target detection and tracking based on algorithms such as YOLO, path planning based on algorithms such as A* (a heuristic search algorithm) / RRT (Rapidly-exploring Random Tree Algorithm), and low-level flight control based on PID (Proportional-Integral-Derivative) / MPC (Model Predictive Control). This system relies on accurate environmental maps and pre-defined rule logic (such as fixed waypoints) to achieve stable fixed-point flight and obstacle avoidance. However, its core deficiency lies in the complete lack of semantic understanding of high-level natural language commands, making it unable to directly respond to open-ended tasks such as "find and follow the person wearing red clothes," resulting in severely insufficient intelligence and mission flexibility.

[0026] Interactive systems based on large language model tool calls: To introduce semantic understanding capabilities, related technical solutions attempt to use large language models as task planners. This approach typically deploys the LLM (Large Language Model) in the cloud, which parses user commands and decomposes them into a series of API (Application Programming Interface) calls to predefined "tools" (such as navigation and detection), which are then distributed to the onboard unit for execution. While this architecture provides the system with some task decomposition and logical reasoning capabilities, the large language model only acts as a remote scheduler and does not participate in the generation of underlying continuous actions, resulting in a deep disconnect between perception and action generation. When faced with tasks requiring precise, real-time visual servoing (such as "flying around a target"), the system struggles to achieve smooth, adaptive control.

[0027] End-to-end visual language action control based on a single large model: This approach aims to construct a unified perception-decision-control model. It employs a large visual language model (such as one based on OpenVLA or RT-2 architecture) as its core, simultaneously inputting visual observations and text commands. Through cross-modal fusion within the model, it directly outputs flight actions or trajectories end-to-end. Theoretically, this method can achieve deep collaboration between command understanding and action generation. However, its tightly coupled single-model architecture exposes fundamental engineering contradictions on UAV platforms: the large model has high single-inference latency (typically on the order of hundreds of milliseconds), severely limiting the output frequency of control commands and failing to meet the millisecond-level high-frequency closed-loop response requirements of UAV flight. This directly leads to an irreconcilable conflict between the deep computation required for high-level semantic understanding and the real-time requirements of low-level control, making the system highly susceptible to instability in dynamic environments.

[0028] Based on an in-depth analysis of relevant technologies, the core technical problem that this application aims to solve is: 1. How to resolve the contradiction between high inference latency of large models and the high real-time control requirements of UAVs. Existing end-to-end VLA solutions require that the generation of each control command must wait for a complete large model inference. On airborne embedded platforms with limited computing power and power consumption, this results in an extremely low output frequency of control commands, which is seriously lagging behind the response speed required for UAV flight control. This causes a fundamental conflict between high-level semantic understanding and low-level real-time control, directly affecting flight stability and safety.

[0029] To address this, the core of this application proposes a dual-system asynchronous decoupled architecture. This architecture splits a single model into a high-frequency low-level action generation system (cerebellum) and a low-frequency high-level semantic understanding system (brain). The two communicate asynchronously via semantic guidance vectors shared in memory. The semantic understanding system handles deep but time-consuming semantic reasoning, outputting a vector encoding the task intent; the action generation system, on the other hand, is lightweight and high-speed, requiring only this vector as a condition to generate smooth trajectories at high frequency by combining real-time sensor data. This decouples the large model's reasoning from the control loop, fundamentally solving the latency bottleneck.

[0030] 2. How to address the "catastrophic forgetting" and insufficient generalization issues caused by end-to-end fine-tuning. Related methods typically use single-task UAV trajectory data to fine-tune a complete VLA model, forcing the model to over-adjust parameters to learn low-level control skills. This damages the valuable general visual recognition and semantic understanding capabilities gained from pre-training on massive amounts of general data. Consequently, the model's performance drops sharply and its generalization ability weakens when faced with complex commands or unfamiliar environments outside the training set.

[0031] Some embodiments of this application employ a multi-stage progressive training strategy. This strategy first freezes the parameters of the high-level semantic understanding system (large model) and trains the motion generation system using only unannotated flight data, enabling it to grasp basic flight dynamics. Subsequently, while maintaining the general knowledge of the large model, it uses aerial perspective data for efficient visual command fine-tuning of its parameters (such as LoRA), adapting it to the UAV domain. Finally, a collaborative fine-tuning of the entire system is performed. This phased, isolated training method effectively preserves the general cognitive capabilities of VLA while injecting domain-specific skills, significantly improving generalization.

[0032] 3. How to address the issues of error accumulation and poor robustness of imitation learning strategies in long-term tasks. Most existing methods rely on open-loop imitation learning from expert demonstration data, lacking proactive error correction mechanisms based on real-time environmental feedback. In real dynamic flight environments, the impact of initial small prediction deviations or external disturbances (such as gusts) accumulates and amplifies over time (distribution shift), ultimately causing the UAV to deviate significantly from its intended flight path or even collide, making navigation success rate and safety difficult to guarantee.

[0033] In some embodiments of this application, a reinforcement learning stage based on group relative policy optimization is introduced on top of imitation learning. In this stage, the model generates multiple candidate trajectory sequences in parallel for the same state and evaluates them based on a composite reward function that includes safety, target proximity, etc. By calculating the relative advantage of trajectories within a group, the model can autonomously learn a more robust flight strategy that surpasses the original expert data, thereby possessing online self-correction and anti-interference capabilities in out-of-distribution states, significantly improving the success rate and safety in complex dynamic environments.

[0034] The following detailed description, in conjunction with the accompanying drawings, of the UAV control method and system based on visual language action model provided in this application, through specific embodiments and application scenarios, will be provided in detail.

[0035] Reference Figure 1 As shown, Figure 1 This is a flowchart illustrating the steps of a drone control method based on a visual language action model, as provided in an embodiment of this application. Figure 1 As shown, an embodiment of this application provides a drone control method based on a visual language action model, which may include steps S110 to S140: Step S110: Obtain the task instructions for natural language description.

[0036] The mission command refers to a command issued in natural language that describes the spatial task that the drone needs to complete, such as "fly to the southeast corner of the building and check the windows," "search for pedestrians wearing blue coats in the park area," or "fly around the central sculpture once." This mission command can be converted by a voice recognition module or directly input in text form through a ground control station or mobile application.

[0037] Step S120: Using a semantic understanding system running at a first frequency, reasoning is performed based on the current environmental visual observation information and the task instructions to generate a semantic guidance vector, and the semantic guidance vector is written into shared memory; wherein, the semantic guidance vector is used to represent the task intent and environmental semantics, and the semantic understanding system is constructed based on a visual language action model.

[0038] The semantic understanding system, also known as a "high-level semantic understanding system" or "brain," is essentially a pre-trained and domain-adapted visual-language action model capable of integrating and understanding visual scenes with textual semantics. The semantic understanding system operates at a primary frequency (the periodic triggering or inference frequency of the semantic understanding system). The range of values ​​for this primary frequency depends on the onboard computing unit's support capability for the semantic understanding system's inference. Its design principle ensures it can understand command intent and respond to significant changes in environmental semantics.

[0039] Environmental visual observation information can be obtained from one or more key images captured by drone-borne cameras (such as RGB cameras or depth cameras) at the current or most recent moment. These images reflect the real-time visual environment in which the drone is located.

[0040] A semantic guidance vector is a fixed-dimensional dense vector generated by a semantic understanding system. It is an abstract, compact, and distributed representation of the high-level task intent and related environmental semantics extracted after deep cross-modal reasoning of the current task instruction and the current environmental visual observation information. For example, for the task instruction "find red vehicles", the semantic guidance vector encodes compound semantics such as "focus on red", "target is vehicle class", and "perform search behavior".

[0041] Shared memory is a memory region provided by the operating system or middleware that can be quickly accessed by multiple processes or threads. Here, it serves as a buffer for semantic guidance vectors, specifically used to connect the two processing processes of the semantic understanding system and the action generation system.

[0042] Specifically, the semantic understanding system performs inference at a first frequency, inputting the current environmental visual observation information (such as the current moment or a recent key image) along with the task instructions obtained in step S110 into the semantic understanding system for inference, generating a semantic guidance vector. Finally, the generated semantic guidance vector is written to a designated area of ​​shared memory.

[0043] Step S130: The action generation system, which operates at a second frequency, reads the latest semantic guidance vector recently written from the shared memory and generates flight control quantities based on the latest semantic guidance vector, real-time environmental visual observation information, and real-time UAV status information; wherein the second frequency is higher than the first frequency.

[0044] The motion generation system (also known as the "low-level motion generation system" or "cerebellum") is a lightweight, low-latency generative policy network that can translate abstract semantic intentions into concrete motion commands (i.e., flight control variables). The motion generation system is designed to operate at a second frequency higher than the first frequency; this second frequency is determined to match the real-time requirements of the UAV flight control loop and ensure smooth and stable control.

[0045] Real-time environmental visual observation information can also be acquired by UAV onboard cameras (such as RGB cameras and depth cameras), but the real-time environmental visual observation information may be different from the "current environmental visual observation information" used for semantic understanding in step S120. The real-time environmental visual observation information refers to the image stream with lower latency or higher frame rate provided to meet high-frequency control.

[0046] Real-time drone status information typically includes real-time data from sensors such as inertial measurement units and barometers, such as three-dimensional position, velocity, attitude, and angular velocity, reflecting the drone's instantaneous motion status.

[0047] Specifically, the action generation system operates continuously in an independent process at a predetermined frequency (e.g., 100 times per second). At the beginning of each control cycle, the action generation system reads the latest semantic guidance vector, which was most recently successfully written by the semantic understanding system, from shared memory. The action generation system uses this latest semantic guidance vector as conditional input and fuses it with real-time environmental visual observation information and real-time UAV state information. Subsequently, through its core generative network (e.g., a Transformer-based diffusion model), it directly outputs a "flight control variable" using a feedforward or iterative denoising approach. This flight control variable can be a trajectory sequence within a future time window, a velocity / attitude command for the next moment, etc.

[0048] The step of reading the latest semantic guidance vector most recently written from the shared memory includes: between two runs of the semantic understanding system, the action generation system runs multiple times and repeatedly reads the latest semantic guidance vector most recently written by the semantic understanding system.

[0049] In this embodiment, since the action generation system operates more frequently than the semantic understanding system, the semantic understanding system writes the latest semantic guidance vector into the shared memory unit after completing one round of inference. Between two runs of the semantic understanding system, the action generation system can repeatedly read the latest semantic guidance vector during multiple runs. In other words, regardless of whether the semantic understanding system is in the process of a new round of slow inference, the action generation system reads the latest available semantic guidance vector, thereby achieving asynchronous collaboration.

[0050] Step S140: Control the UAV to perform flight mission according to the flight control variables.

[0051] Specifically, digitized flight control quantities can be converted into actual physical control quantities, which are then used to control the drone to perform flight missions. For example, if the flight control quantity is a target speed, throttle commands sent to the motors can be calculated using algorithms such as PID or model predictive control; if the control quantity is a trajectory, a trajectory tracker will generate corresponding tracking commands. Ultimately, the drone's actuators respond to these commands to complete flight maneuvers, thereby gradually realizing the natural language tasks given by the user.

[0052] The technical solution adopted in this application decouples the semantic understanding system and the action generation system by operating them at different frequencies and utilizes shared memory to transmit semantic guidance vectors. This eliminates the need for computationally intensive high-level semantic inference to be completed synchronously in each control cycle. The action generation system can operate independently at a higher frequency than the semantic understanding system, quickly generating flight control quantities simply by reading the latest semantic guidance vector cached in shared memory. This significantly reduces control latency while ensuring accurate understanding of complex natural language instructions, effectively meeting the stringent real-time response requirements of UAV flight and thus improving flight stability and safety. The asynchronous dual-system architecture is naturally compatible with heterogeneous computing platforms, enabling optimized deployment and efficient utilization of computing resources. This mechanism isolates potential fluctuations or delays in the semantic understanding process from the control closed loop, avoiding instruction interruptions or jumps caused by uncertainties in high-level inference, and enhancing the smoothness and fault tolerance of the overall system operation. Furthermore, the semantic guidance vector, as the data interface connecting the semantic understanding system and the action generation system, encapsulates abstract task intent, achieving decoupling between semantic generation and action execution processes. This design enhances the system's modularity and maintainability, enabling independent upgrades and replacements of either the semantic understanding system or the action generation system without requiring a complete refactoring. Simultaneously, the compact vectorized representation reduces communication overhead between the semantic understanding system and the action generation system, providing a feasible foundation for implementing complex visual language interaction functions on computationally and power-constrained airborne embedded platforms.

[0053] In an optional embodiment, the semantic understanding system includes a first visual encoder, a text encoder, a large language model, and a feature projection layer. Specifically, step S120 above, "performing inference based on the current environmental visual observation information and the task instructions to generate a semantic guidance vector," may include steps S120-1 to S120-4: Step S120-1: Encode the current environment visual observation information using the first visual encoder to obtain the first visual feature.

[0054] The first visual encoder is typically a deep convolutional neural network or a visual Transformer model. This first visual encoder has been pre-trained on massive image data and can effectively extract high-level visual features such as objects, textures, and spatial relationships.

[0055] Specifically, the semantic understanding system acquires the current keyframe image captured by the UAV's onboard camera (as "current environment visual observation information"). This image is scaled and normalized before being input into the first visual encoder. The first visual encoder extracts features step by step through its multi-layer network structure, ultimately outputting a "first visual feature" that represents the core visual content of the entire image.

[0056] Step S120-2: Encode the task instructions using the text encoder to obtain text features.

[0057] The text encoder is typically part of a large language model or visual language model, such as a Transformer-based tokenizer and word embedding layer. Specifically, the task instruction is input into the text encoder, which segments the text into a series of tokens (sub-words or words), converting each token into a corresponding vector by looking up an embedding table. These vector sequences collectively constitute the text features corresponding to the task instruction, capturing its lexical, grammatical, and preliminary semantic information.

[0058] Step S120-3: Perform cross-modal fusion and inference on the first visual features and the text features using the large language model to obtain high-level semantic features.

[0059] Among them, large language models refer to core backbone networks with cross-modal understanding capabilities, such as multimodal large models based on the Transformer architecture; large language models are used to jointly process visual and linguistic information, and their internal mechanisms (such as cross attention) allow visual features and text features to interact deeply.

[0060] Specifically, the first visual features and textual features are input together into the large language model. Within the large language model, through a cross-modal attention mechanism, each word in the text can "attention" to a relevant region of the image, and vice versa. For example, when processing the task instruction "find the red backpack," the large language model learns to associate the words "red" and "backpack" in the text with visual regions of corresponding colors and shapes in the image. After multiple layers of this interactive reasoning, the large language model ultimately outputs a high-level semantic feature. This high-level semantic feature represents the intrinsic representation obtained after comprehensive reasoning about "how to understand and execute the task instruction in the current visual scene," integrating environmental context and task objective.

[0061] Step S120-4: Map the high-level semantic features through the feature projection layer to generate the semantic guidance vector.

[0062] The feature projection layer is typically a neural network module consisting of one or more fully connected layers, used to perform dimensional transformation and alignment of features.

[0063] Specifically, the high-level semantic features output by large language models may have variable formats and dimensions (e.g., variable-length feature sequences). To ensure stable and efficient integration with downstream, high-frequency action generation systems, these high-level semantic features must be converted to a unified format. The feature projection layer performs this task, taking the high-level semantic features as input and compressing and normalizing them into a fixed-dimensional (e.g., 768-dimensional) semantic guidance vector through linear transformations or nonlinear mappings. This semantic guidance vector, the final and most refined encoding of the task intent, is written into shared memory for use by the action generation system.

[0064] The technical solution adopted in this application extracts high-fidelity features through a visual encoder and a text encoder, respectively, and performs deep cross-modal attention fusion with a large language model. This enables the semantic understanding system to accurately extract high-level semantics strongly related to the task, providing reliable intent guidance for subsequent control. Furthermore, the feature projection layer maps the potentially complex and variable internal representation output by the large language model into a fixed-dimensional semantic guidance vector. This defines a clear, stable, and low-communication-overhead data contract between the semantic understanding system (brain) and the action generation system (cerebellum), which is the cornerstone for ensuring efficient and asynchronous collaborative work between the two systems.

[0065] In an alternative embodiment, considering that task instructions may contain multi-level subtasks—that is, task instructions may be complex descriptions containing multiple consecutive actions or stage objectives, such as "take off, fly to the center of the living room, then fly around the dining table once, and finally land on the sofa"—such task instructions cannot be effectively executed through a single, static semantic intent. This embodiment extends the functionality of the semantic understanding system, enabling it not only to understand the overall task but also to perform task decomposition, sequence planning, and execution status tracking, thereby guiding the UAV to complete complex operations step by step.

[0066] Specifically, when the task instruction contains multiple sub-tasks, the step S120 above, "inferring semantic guidance vectors based on current environmental visual observation information and the task instruction," may further include steps A1 to A3: Step A1: Decompose the task instruction into multiple task nodes that are executed sequentially; wherein each task node is associated with a specific task action and environmental features.

[0067] In this context, multi-level subtasks refer to multiple logically divisible and temporally ordered components within a single overall task instruction. Task nodes are the stage objectives decomposed from the task instruction. Each task node defines the core actions to be performed in that stage (e.g., "takeoff," "fly to," "orbit," "land") and the key environmental features to be perceived (e.g., "central living room space," "dining table object," "sofa plane"). There are sequential constraints between task nodes.

[0068] Specifically, for task instructions containing multiple sub-tasks, the semantic understanding system leverages its powerful logical reasoning and common-sense understanding capabilities to parse the input task instructions, transforming them into a structured task graph or linear sequence. For example, the task instruction "Take off, fly to the center of the living room, then circle the dining table once, and finally land on the sofa" might be decomposed into: Node 1 (Action: Take off, Feature: None), Node 2 (Action: Navigate to, Feature: Central area of ​​the living room), Node 3 (Action: Circle, Feature: Dining table), and Node 4 (Action: Land, Feature: Sofa). This decomposition process is completed when the semantic understanding system initializes its instruction processing, forming an internal task plan.

[0069] Step A2: Generate a semantic guidance vector corresponding to the target task node to be executed.

[0070] When generating a semantic guidance vector, the encoded semantic content focuses on the action intent and expected features of the target task node to be executed. Specifically, the semantic guidance vector corresponding to the target task node can be generated using the methods described in steps S120-1 to S120-4 above; for example, when the target task node is "around the dining table," the generated semantic guidance vector will strongly encode the motion pattern of "around" and the visual target attributes of "dining table." This semantic guidance vector is written to shared memory to guide the behavior of the action generation system in the current stage.

[0071] Step A3: Based on the current environmental visual observation information, and after confirming that the task action corresponding to the target task node has been completed, switch to the next task node and generate a semantic guidance vector corresponding to the next task node.

[0072] Specifically, during each inference iteration, the semantic understanding system, in addition to generating a guidance vector, also performs a completion detection function in parallel, that is, detecting whether the task action corresponding to the target task node has been completed. For example, for the "navigate to the center of the living room" node, the semantic understanding system analyzes the current environmental visual observation information to determine whether the drone is already in an open area that matches the semantic description of "center of the living room". If the detection result indicates that the task action corresponding to the target task node has not been completed, the semantic understanding system keeps the target task node unchanged, and generates a guidance vector for that target node again in the next run. If the detection result indicates that the task action corresponding to the target task node has been completed (for example, confirming that the drone has reached the center of the living room), the semantic understanding system automatically switches the state to the next task node (such as "around the dining table").

[0073] During the inference cycle of the next task node after the state switch, the semantic understanding system generates a completely new semantic guidance vector based on the new target task node and writes it into shared memory. In this way, the intent signal read by the action generation system changes, thereby driving the drone to perform the next stage of behavior.

[0074] The technical solution adopted in this application automatically transforms vague natural language descriptions of task instructions into executable step-by-step plans, enabling UAVs to complete complex operations involving multiple stages and sub-objectives in an orderly manner, thus expanding application scenarios. Furthermore, task progression is based on the real-time semantic judgment of the visual environment by the semantic understanding system. This closed loop of "perception-confirmation-progression" makes task execution more intelligent and reliable, adaptable to environmental uncertainties.

[0075] In an optional embodiment, the motion generation system includes a motion expert network, a second visual encoder, and a state encoder. Specifically, step S130 above, "generating flight control quantities based on the latest semantic guidance vector, real-time environmental visual observation information, and real-time UAV state information," may include steps S130-1 to S130-4: Step S130-1: Encode the real-time environmental visual observation information using the second visual encoder to obtain the second visual feature.

[0076] The second visual encoder can be a lightweight neural network (such as a small CNN or a lightweight ViT). This visual encoder has a streamlined structure and is specifically designed for high-frequency, low-latency real-time processing. Specifically, the action generation system continuously acquires real-time environmental visual observation information (e.g., the latest image frames from the UAV's onboard camera). This real-time environmental visual observation information is input into the second visual encoder, which outputs a second visual feature. This feature reflects the current instantaneous visual environment overview in front of and around the UAV and is the primary basis for obstacle avoidance and immediate response.

[0077] Step S130-2: Encode the real-time UAV state information using the state encoder to obtain state features.

[0078] The state encoder can be a multilayer perceptron. Specifically, the motion generation system acquires real-time UAV state information and inputs it into the state encoder. The state encoder, through learned nonlinear transformations, fuses this data into a state feature rich in dynamic information. This state feature encodes the UAV's instantaneous motion attitude and inertia.

[0079] Step S130-3: Use the latest semantic guidance vector, the second visual feature, and the state feature as the generation conditions of the action expert network, and perform denoising and generation processing through the action expert network to generate a trajectory sequence.

[0080] Among them, motion expert networks are a type of neural network based on conditional generative models (such as diffusion models). The way they work is: given random noise and generation conditions, they transform the noise into a reasonable, future trajectory sequence through a step-by-step denoising process.

[0081] Specifically, the action generation system concatenates or fuses the latest semantic guidance vector from shared memory, the second visual features obtained in step S130-1, and the state features obtained in step S130-2, forming a unified conditional context vector. The action expert network starts with a noisy sequence representing a future time period (e.g., 1.5 seconds) and uses the fused conditional context as the generation condition, performing several steps of denoising and sampling. Under these constraints, the action expert network generates a future trajectory sequence that conforms to the high-level intent (where to go), adapts to the real-time environment (how to avoid obstacles), and considers its own state (how to smoothly transition). This future trajectory sequence typically includes predicted position, velocity, or pose for the next N time steps.

[0082] Step S130-4: Generate the flight control quantity based on the trajectory points corresponding to the first M time steps in the trajectory sequence; where M is a positive integer greater than or equal to 1.

[0083] Specifically, the motion generation system extracts the trajectory points corresponding to the first M time steps (e.g., M=1 corresponds to immediately executing the next step, and M=3 corresponds to executing a trajectory segment in a future time period) from the generated trajectory sequence. These points represent the optimal motion path in the short term. Subsequently, the low-level tracking controller converts these trajectory points into flight control variables, such as target attitude angles or throttle commands. In the next control cycle, the entire S130 process is re-executed based on the latest real-time environmental visual observation information and real-time UAV status information to generate an updated trajectory sequence, thereby achieving closed-loop, responsive model predictive control.

[0084] The technical solution of this application, employing a lightweight dedicated encoder and motion expert network, enables the motion generation system to complete the entire process from multimodal perception to trajectory sequence generation at extremely high operating frequencies (second frequencies), meeting the real-time control requirements of UAVs. Furthermore, the motion expert network, using a generative model, deeply integrates abstract semantic intent (through the latest semantic guidance vector), real-time environmental information (through visual features), and precise self-dynamic constraints (through state features). This ensures that the generated trajectory sequence is semantically correct and physically feasible. In addition, by generating future multi-step trajectories and using the trajectory points corresponding to the first M time steps to generate control variables, the UAV can make smooth preparatory movements for turns and obstacle avoidance in advance, improving flight smoothness, safety, and adaptability to dynamic environments.

[0085] like Figure 2 As shown, Figure 2This is an overall architecture diagram of a UAV control method based on a visual language action model provided in an embodiment of this application. The method is implemented based on a semantic understanding system operating at a first frequency and an action generation system operating at a second frequency. The semantic understanding system includes a first visual encoder, a text encoder, a large language model, and a feature projection layer; the action generation system includes an action expert network, a second visual encoder, and a state encoder.

[0086] Specifically, in each inference cycle, the semantic understanding system encodes the current environmental visual observation information through a first visual encoder to obtain first visual features; it encodes the task instructions through a text encoder to obtain text features; it performs cross-modal fusion and inference on the first visual features and text features through a large language model to obtain high-level semantic features; it maps the high-level semantic features through a feature projection layer to generate semantic guidance vectors; and finally, it writes the semantic guidance vectors into shared memory for the action generation system to read and use. In each control cycle, the action generation system encodes the real-time environmental visual observation information through a second visual encoder to obtain second visual features; it encodes the real-time UAV state information through a state encoder to obtain state features; it uses the latest semantic guidance vector, second visual features, and state features as generation conditions for the action expert network, which performs denoising and generation processing to generate a trajectory sequence; finally, it generates flight control variables based on the trajectory points corresponding to the first M time steps in the trajectory sequence.

[0087] like Figure 3 As shown, by decoupling the semantic understanding system and the action generation system at different frequencies and utilizing shared memory to pass semantic guidance vectors, computationally intensive high-level semantic inference does not need to be completed synchronously in each control cycle. The action generation system can independently run at a higher frequency than the semantic understanding system. Between two runs of the semantic understanding system, the action generation system can run multiple times and repeatedly read the latest semantic guidance vector recently written by the semantic understanding system to quickly generate flight control quantities. While ensuring that complex natural language commands are accurately understood, this significantly reduces control latency, effectively meets the stringent requirements of UAV flight for real-time response, and improves flight stability and safety.

[0088] In one alternative embodiment, a phased, progressive training paradigm is employed to train the semantic understanding system and the action generation system. This allows different systems to focus on learning their core capabilities, followed by gradual alignment and collaborative optimization. This approach injects drone-specific skills while maximizing the retention of general knowledge from the base model, ultimately improving performance in unknown environments through reinforcement learning.

[0089] Specifically, the semantic understanding system and the action generation system are trained through the following steps S210 to S240: Step S210: Using the first training sample, train the motion generation system to be trained to learn the flight dynamics characteristics and obstacle avoidance priors of the UAV, and obtain a preliminarily trained motion generation system; wherein, the first training sample includes environmental visual observation information, UAV state information and real trajectory sequence.

[0090] In this embodiment, the first training samples can be generated on a large scale through a random roaming strategy in a simulated environment, or extracted from open-source real flight logs. The first training samples are unlabeled trajectory data, that is, such data only contains sensor data streams (environmental visual observation information, UAV status information) recorded by the UAV while flying in the environment and their corresponding real trajectory sequences generated by expert systems or skilled operators.

[0091] During this training phase, all parameters of the semantic understanding system to be trained are frozen, preventing it from participating in gradient updates. The training objective is entirely focused on the action generation system to be trained. Specifically, the "environmental visual observation information" and "UAV state information" from the first training sample are input into the action generation system, and the action expert network predicts or reconstructs the trajectory sequence for the next period of time. Mean squared error is then used as the reconstruction loss function to calculate the Euclidean distance between the predicted trajectory sequence and the actual trajectory sequence. This distance is used to update the network parameters of the action generation system to be trained, prompting the action generation system to learn the UAV's flight dynamics characteristics and obstacle avoidance priors (i.e., the motion laws followed by the UAV and the basic rules for safe passage in complex spaces, such as avoiding obstacles and maintaining a smooth path).

[0092] In this way, by training the motion expert network using the first training sample, the motion expert network can implicitly learn the physical motion laws of the drone and the general patterns of safe flight (i.e., "how to fly") from the data. This provides a motion instinct module with basic flight and obstacle avoidance capabilities for subsequent stages.

[0093] Step S220: Using the second training sample, fine-tune the visual command of the semantic understanding system to be trained in order to learn the UAV mission perspective and mission command semantics, and obtain a preliminarily trained semantic understanding system; wherein, the second training sample includes aerial view images, mission commands and the real trajectory sequence corresponding to the mission commands.

[0094] In this embodiment, the second training sample data sources include: public aerial photography dataset conversion, that is, converting the target detection dataset into a question-and-answer format; simulation synthesis data, that is, generating images of specific tasks in a simulation environment and automatically generating text data of future target points for UAV navigation tasks; and a domain corpus containing text data of UAV operation manuals and flight rules, used to enhance the understanding of specific instructions.

[0095] This training phase focuses on improving the domain adaptability of the semantic understanding system while preserving its general knowledge by freezing the parameters of the action generation system during initial training and training only the semantic understanding system. Specifically, aerial view images and task commands from the second training sample are input into the semantic understanding system to be trained. The system outputs a response that conforms to the task command, i.e., the predicted trajectory sequence of the UAV (such as the UAV's future position coordinates). A standard causal language modeling loss function is then used to calculate the difference between the predicted trajectory sequence and the actual trajectory sequence corresponding to the task command, enabling efficient fine-tuning of the parameters of the semantic understanding system using LoRA.

[0096] Thus, by training the semantic understanding system using a second training sample, the system learns the mission perspective and semantics of the UAV. For example, for the mission command "fly towards the red roof in the image," during training, the semantic understanding system needs to learn to identify the "red blocky area" in the aerial image as a "roof" and understand that "fly towards" is a navigation action. This allows the semantic understanding system to adapt to the UAV's unique top-down, wide-angle perspective and deepens its understanding of navigation-related terminology.

[0097] Step S230: Using the third training sample, jointly fine-tune the initially trained semantic understanding system and the initially trained action generation system to obtain the trained semantic understanding system and the trained action generation system; wherein, the third training sample includes: task instructions containing multi-level tasks, environmental visual observation information and the real trajectory sequence corresponding to the task instructions.

[0098] In this embodiment, the third training sample (instruction-trajectory pair data) is high-quality labeled data, which contains complex task instructions that may contain multi-level sub-tasks (such as "first bypass the water tower, then reduce the altitude to find the entrance"), environmental visual observation information, and a complete real trajectory sequence demonstrated by an expert under the instruction.

[0099] This training phase builds upon the training phases S210 and S220, allowing the semantic understanding system and action generation system to learn collaboratively and optimize the complete mapping from task instructions to final actions. Specifically, the parameters of the initially trained action generation system are unfrozen to adapt to subtle semantic changes at higher levels. For the semantic understanding system with a large number of parameters, to avoid destroying its pre-trained knowledge, a low-rank adaptation (LoRA) technique is used to optimize only a small number of adapter parameters (e.g., introducing a small number of trainable parameters in the query and value matrix bypass of the attention layer to achieve efficient parameter fine-tuning). The task instructions from the third training sample and the initial environmental visual observation information are input into the initially trained semantic understanding system to perform inference. The generated semantic guidance vector is written to shared memory. The action generation system reads the latest semantic guidance vector from the shared memory and generates a predicted trajectory sequence based on the latest semantic guidance vector, real-time environmental visual observation information, and real-time UAV state information. The final loss function value is calculated based on the prediction error (the difference between the predicted action trajectory and the actual action trajectory) and the semantic understanding error (ensuring that the model understands the task instructions consistently), so as to fine-tune the initially trained semantic understanding system and the initially trained action generation system.

[0100] Thus, by utilizing a third training sample, the initially trained semantic understanding system and the initially trained action generation system are jointly fine-tuned. This allows the semantic understanding system to learn to output semantic guidance vectors that are more effective in guiding action generation, while the action generation system learns to interpret and utilize these vectors more accurately. This eliminates alignment errors between the semantic understanding system and the action generation system, improving the model's execution coherence in long-term tasks.

[0101] Step S240: Based on the reinforcement learning method, optimize the strategy of the trained action generation system to obtain the trained action generation system.

[0102] In this embodiment, based on reinforcement learning, the trained action generation system optimizes its strategy according to the reward signal for task completion during interaction with the (simulation) environment, enabling it to explore and correct errors in unknown environments.

[0103] In some embodiments, the Group Relative Policy Optimization (GRPO) algorithm in reinforcement learning is used to guide the model to actively explore in the simulation environment by utilizing reward signals from environmental feedback, thereby discovering a better or more robust flight strategy than expert demonstrations, thus solving the error accumulation problem in long-term tasks and improving performance under unknown perturbations.

[0104] Specifically, step S240 above, "based on reinforcement learning methods, performing policy optimization on the trained action generation system to obtain a trained action generation system," may include steps S240-1 to S240-3: Step S240-1 For a given target state, multiple candidate trajectory sequences are generated through the motion expert network of the trained motion generation system; wherein, the target state includes environmental visual observation information, UAV status information and mission instructions.

[0105] In this context, the given target state refers to the observational information provided to the model by the simulation environment at a certain moment during reinforcement learning training. This typically includes environmental visual observations, UAV status information, and mission instructions (such as "fly to the red marker"). Using the trained action generation system's action expert network, multiple (e.g., K) candidate trajectory sequences are generated in parallel by introducing different random noise seeds during the diffusion or decoding process. Each candidate trajectory sequence represents a possible action policy that the action expert network might adopt in the current state. This mechanism enables the model to locally explore the policy space for a given state.

[0106] Step S240-2: Construct a composite reward function to calculate the score for each candidate trajectory sequence; wherein the composite reward function includes at least the proximity reward between the trajectory endpoint corresponding to the candidate trajectory and the target position indicated by the task instruction.

[0107] The proximity between the endpoint of a candidate trajectory and the target location indicated by the task command can be measured by Euclidean distance. The closer the endpoint of the trajectory is to the target location, the higher the score of the candidate trajectory sequence.

[0108] The composite reward function can also include a safety obstacle avoidance reward and a trajectory smoothness reward. The safety obstacle avoidance reward checks whether the candidate trajectory sequence collides with obstacles in the environment; if a collision occurs, a large negative reward (penalty) is applied. The trajectory smoothness reward is used to evaluate the magnitude of the acceleration of the candidate trajectory sequence; excessive speed changes will lead to a negative reward, encouraging the generation of smooth and comfortable flight paths.

[0109] Specifically, for each candidate trajectory, a composite reward function is constructed to score it on each dimension, and the rewards of each dimension are weighted and summed to obtain the score of each candidate trajectory sequence. This score is a composite reward value.

[0110] Step S240-3 calculates the relative advantage signal of each candidate trajectory sequence based on the scores of multiple candidate trajectory sequences, and updates the parameters of the trained action generation system based on the relative advantage signal to obtain the trained action generation system.

[0111] The relative advantage signal measures the performance of a candidate trajectory sequence relative to other candidate trajectory sequences in the same batch. It is the core gradient signal used to guide policy updates. A positive advantage indicates that the selected trajectory sequence is better than the average level and should be strengthened; otherwise, it should be suppressed.

[0112] In some embodiments, calculating the relative advantage signal of each candidate trajectory sequence based on the scores of multiple candidate trajectory sequences includes: calculating the average score and standard deviation of the multiple candidate trajectory sequences based on the scores of the multiple candidate trajectory sequences; calculating the difference between the score of each candidate trajectory sequence and the average score, and normalizing it using the standard deviation to obtain the relative advantage signal of each candidate trajectory sequence.

[0113] For example, candidate trajectory sequences relative advantage signal It can be represented as:

[0114] in, Represents candidate trajectory sequence The rating, This represents the average score of multiple candidate trajectory sequences. This represents the standard deviation of multiple candidate trajectory sequences.

[0115] Thus, by using the relative advantage signal of the candidate trajectory sequences, the scores of the candidate trajectory sequences can be intuitively normalized and transformed into a distribution with a mean of 0 and a standard deviation of approximately 1. When the relative advantage signal of a candidate trajectory sequence is greater than 0, it means that the candidate trajectory sequence outperforms the average performance in this round of sampling. Therefore, the parameters of the trained action generation system can be updated based on the relative advantage signal.

[0116] In some embodiments, updating the parameters of the trained action generation system based on the relative advantage signal includes: determining candidate trajectory sequences with a positive relative advantage signal as advantageous trajectory sequences and candidate trajectory sequences with a negative relative advantage signal as disadvantageous trajectory sequences; updating the parameters of the trained action generation system with the goal of increasing the generation probability of the advantageous trajectory sequences and decreasing the generation probability of the disadvantageous trajectory sequences.

[0117] Specifically, with the goal of increasing the generation probability of dominant trajectory sequences and decreasing the generation probability of suboptimal trajectory sequences, a policy gradient loss value is constructed, and the parameters of the trained action generation system are updated based on this policy gradient loss value. Simultaneously, to maintain training stability and prevent collapse due to excessive policy updates, constraints are introduced during gradient updates, such as limiting the KL divergence between the old and new policies. Then, gradient descent is used to update the parameters of the action expert network.

[0118] The technical solution adopted in this application involves phased isolated training. First, the parameters of the semantic understanding system are frozen to train the action generation system. Then, the semantic understanding system is fine-tuned in a parameter-efficient manner. This ensures that the general visual language cognitive ability is not compromised by low-level control tasks, guaranteeing the model's strong generalization ability to unseen instructions and environments. Furthermore, training the action generation system enables it to master flight dynamics and obstacle avoidance priors, while fine-tuning the visual instructions of the semantic understanding system adapts it to domain characteristics and terminology. The joint fine-tuning stage is a crucial bridge, aligning the semantic guidance vector output by the semantic understanding system with the action trajectory generated by the action generation system in terms of data distribution. This ensures consistency in the understanding and execution of the entire system, improving the continuity of complex, long-cycle tasks. In addition, the reinforcement learning stage enables the action generation system to perform online decision optimization, proactive correction, and anti-interference capabilities in dynamic and uncertain environments, fundamentally enhancing its practicality and security in real-world deployment.

[0119] like Figure 4 As shown, Figure 4 This is a training flowchart of a semantic understanding system and an action generation system provided in an embodiment of this application. This application employs a multi-stage, progressive training strategy, aiming to build the core capabilities of the system step-by-step and efficiently, while avoiding catastrophic forgetting and improving final performance. The process mainly includes the following four key stages: The first stage involves initial training of the motion generation system (i.e., step S210 above). This stage aims to enable the motion generation system (especially its core motion expert network) to master the basic motion instincts of the drone, namely the fundamental dynamics of flight and general obstacle avoidance knowledge. Specifically, using the first training samples, which contain a large amount of unannotated real trajectory data, the parameters of the semantic understanding system (visual language large model) are completely frozen during this stage. Only the motion expert network is trained, enabling it to predict smooth, safe, and physically consistent future trajectories based on current observations.

[0120] The second stage: Domain Adaptation of the Semantic Understanding System (i.e., step S220 above). This stage aims to adapt the general semantic understanding system to the specific top-down, wide-angle aerial perspective of the UAV, enabling it to better understand UAV navigation-related terminology and commands. Specifically, a second training sample is used, consisting of aerial images and corresponding text commands or question-and-answer pairs (e.g., converted or simulated from publicly available aerial datasets). In this stage, the parameters of the pre-trained motion expert network are frozen, and efficient parameter fine-tuning techniques are used to fine-tune the visual commands of the high-level semantic understanding system. For example, only some parameters of its attention layer are fine-tuned, or an adapter such as LoRA is introduced.

[0121] The third stage involves joint fine-tuning of the action generation system and the semantic understanding system (i.e., step S230 above). This stage aims to align the semantic understanding system and the action generation system, optimize their collaborative capabilities, and achieve a seamless end-to-end mapping from natural language commands to flight trajectories. Specifically, a third training sample is used, containing high-quality, manually verified complex navigation task data. Each data point consists of a natural language command that may contain multiple levels of subtasks, the corresponding initial environment observation, and a complete expert demonstration trajectory. In this stage, the parameters of the action expert network are unfrozen, and the high-level semantic understanding system is typically fine-tuned in a parametrically efficient manner (such as LoRA). Through end-to-end training, a composite loss function combining trajectory prediction loss and semantic understanding loss is used to jointly optimize both systems.

[0122] The fourth stage: Reinforcement learning-based policy optimization (i.e., step S240 above). This stage aims to break through the upper limit of imitation learning, endowing the system (especially the action generation system) with the ability to actively explore, self-correct, and optimize policies in unknown or dynamic environments to cope with distribution shift problems in the real world. Specifically, this stage is carried out in a simulation environment, using the system trained in the first three stages as the initial policy, allowing it to interact with the environment. Methods such as group relative policy optimization are employed: for the same target state, multiple candidate trajectories are generated through parallel sampling using an action expert network; each trajectory is evaluated using a composite reward function (comprehensively considering target proximity, safety, smoothness, etc.); and policy parameters are updated by calculating the relative advantage signal of trajectories within the group, encouraging the generation of high-reward trajectories.

[0123] Thus, through the above four stages of training, the final model is able to learn a more robust and intelligent flight strategy that surpasses the original demonstration data, enhancing the system's adaptability and robustness in the face of environmental disturbances, perceived noise, and unseen scenarios.

[0124] To better understand the UAV control method based on visual language action model of this application, two examples (Example 1 and Example 2) are used for illustration below.

[0125] Example 1: Target search and approach in unknown environments based on natural language instructions.

[0126] The scenario describes a user issuing a natural language-based task command to the drone: "Find the red backpack near the park bench and hover it 2 meters above it." The drone is currently in a park environment without a pre-built map, containing dynamic obstacles such as trees and pedestrians.

[0127] The specific implementation process includes the following steps B1 to B3: Step B1: Semantic understanding and guided generation (corresponding to steps S110 and S120 in the above embodiments).

[0128] Step B1-1: Receive the mission instruction "Go to the park bench and find the red backpack, then hover 2 meters above it".

[0129] Step B1-2: The semantic understanding system acquires keyframe images of the current moment from the aircraft's perception sensors (such as airborne cameras) at the first frequency as visual observation information of the current environment.

[0130] Step B1-3: Input the task instructions and keyframe images into the visual language model of the semantic understanding system for inference and generate corresponding semantic guidance vectors.

[0131] Case A (No clear target found): If the visual language model has low confidence in recognizing the presence of a "bench" or "red backpack" in the image (e.g., below a preset threshold), the semantic understanding system generates a semantic guidance vector that encodes the intention of "exploring forward and expanding the search range".

[0132] Case B (Middle landmark found): If the visual language model detects "bench" but does not see "red backpack", then extract the visual features of the bench and generate a semantic guidance vector that encodes the intention to "move closer to the bench and continue searching".

[0133] Step B1-4: The semantic understanding system writes the generated semantic guidance vector into shared memory.

[0134] Step B2: High-frequency action generation and execution (corresponding to steps S130 and S140 in the above embodiments).

[0135] Step B2-1: The motion generation system operates at a second frequency higher than the first frequency and performs the following operations in each control cycle: a. Read the latest semantic bootstrap vector from shared memory, the one most recently written; b. Acquire real-time image streams provided by the camera as real-time environmental visual observation information, as well as real-time UAV status information (including attitude, angular velocity, etc.) from the inertial measurement unit.

[0136] Step B2-2: The action generation system uses semantic guidance vectors, image features (obtained through real-time environmental visual observation information), and UAV state features (obtained through real-time UAV state information) as generation conditions for its internal action expert network (e.g., a generative model based on diffusion Transformer). The action expert network outputs a predicted trajectory sequence for a future time period (e.g., 1.6 seconds) through a denoising generation process.

[0137] Step B2-3: The motion generation system generates flight control quantities based on the trajectory points corresponding to the first M time steps in the trajectory sequence.

[0138] Since the motion expert network has learned a large number of obstacle avoidance priors during the training phase and the input incorporates real-time depth information, when it senses an obstacle (such as a tree) about 3 meters ahead, the trajectory it generates can automatically include smooth detour maneuvers (such as shifting to the left by 0.5 meters) without relying on a separate path planning module.

[0139] Step B3: Online closed-loop error correction for dynamic interference.

[0140] Specifically, during flight, assuming the UAV encounters a sudden strong lateral wind disturbance (e.g., wind speed of 6 m / s), causing the aircraft to tilt and shift unexpectedly. This state change is captured by the IMU within milliseconds (e.g., 20 ms) and input as updated real-time UAV state information into the low-level motion generation system.

[0141] At this point, the high-level semantic understanding system has not yet performed its next inference update. However, based on the abnormal states acquired in real time during high-frequency operation, the low-level action generation system, combined with the currently effective semantic guidance vector, immediately generates a new trajectory containing a reverse compensation component (such as tilting towards the headwind). The UAV quickly adjusts its attitude accordingly, proactively counteracting wind disturbances before the high-level semantic command update, maintaining a stable pose relative to the search target. This demonstrates the rapid anti-interference capability, similar to a "reflection," achieved by the low-level action generation system through high-frequency closed-loop perception, ensuring flight stability.

[0142] The above embodiments demonstrate the process by which a drone executes complex "search-approach-hover" task commands in an unknown, dynamic outdoor park environment, fully verifying the comprehensive advantages of the technical solution presented in this application. By directly parsing fuzzy commands such as "find the red backpack near the bench" that contain spatial relationships and attribute descriptions through a semantic understanding system, and transforming them into specific search strategies (first find the bench, then the backpack), the drone can understand and execute open-ended exploration tasks based on natural language, much like a human, significantly improving its intelligence. During the search process, the action generation system operates independently at high frequency, achieving smooth real-time obstacle avoidance (such as automatically avoiding trees). When encountering strong wind interference, the action generation system utilizes its high-frequency perception and generation capabilities to quickly generate wind-resistant compensation trajectories. Therefore, this architecture successfully decouples slow, deep thinking from rapid reflex actions, fundamentally solving the instability risk caused by the large model inference delay introducing control loops, ensuring flight safety and robustness in unpredictable dynamic environments.

[0143] Example 2: Indoor cross-room navigation based on natural language description.

[0144] Scenario description: The drone is located on the ground in "Room A (living room)". The user issues a complex mission command: "Take off, fly out of the living room door, turn left in the corridor, fly straight along the corridor, pass the kitchen on the right, and finally enter the study at the end of the corridor and land." The environment is characterized by no GPS signal, a narrow space, and the potential presence of dynamic obstacles (such as pedestrians).

[0145] The specific implementation process includes the following steps C1 to C5: Step C1: Instruction parsing and task decomposition (corresponding to the multi-level task processing from step A1 to step A3 in the above embodiment).

[0146] Step C1-1: After receiving the task instruction, the semantic understanding system uses its logical reasoning capabilities to automatically decompose the task instruction into multiple task nodes that are executed sequentially. For example: Node 1: Find and pass through the door frame (exit); the task action is to find the exit, and the environmental feature is a door / door frame; Node 2: Turn left in the corridor and navigate along the wall; the task action is corridor navigation, and the environmental features are extended walls / floors. Node 3: Confirm passing through the kitchen on the right (landmark); the task action is landmark confirmation, and the environmental feature is kitchen / refrigerator; Node 4: Identify the study room and enter / land; the task action is destination identification, and the environmental features are study room / bookshelf / table.

[0147] Step C1-2: The semantic understanding system sets "Node 1" as the currently active target task node and generates the corresponding semantic guidance vector "find and cross the door frame", which is then written to shared memory.

[0148] Step C2: Visual servo traversal of narrow passages (demonstrating fine control under semantic guidance).

[0149] Step C2-1: The motion generation system reads the semantic guidance vector of "find and pass through the door frame" and controls the drone to rotate and scan the environment.

[0150] Step C2-2: When the semantic understanding system recognizes the "door frame" outline in the subsequent keyframe image, the "search" sub-target of decision node 1 is completed, and the semantic guidance vector is updated to "crossing mode" and written to shared memory.

[0151] Step C2-3: Based on the new semantic guidance vector and real-time depth information, the motion generation system generates a trajectory to precisely center the drone as it passes through a narrow doorway (e.g., 80cm wide). Through learned physical characteristics, this trajectory automatically incorporates deceleration and lateral fine-tuning upon approach to ensure safe passage.

[0152] Step C3: Corridor navigation and landmark confirmation (reflecting the progress of the task).

[0153] Step C3-1: After passing through the door frame, the semantic understanding system detects changes in the field of vision (such as the appearance of a narrow space) based on the current environmental visual observation information, activates "Node 2", and generates a semantic guidance vector of "corridor following + left turn".

[0154] Step C3-2: The drone flies along the corridor and turns left under the control of the motion generation system. During flight, when the visual system detects a "kitchen" feature (such as a refrigerator) on the right, the semantic understanding system confirms that node 3 is complete and automatically advances the target task node to node 4 (finding the study).

[0155] Step C4: Dynamic obstacle avoidance and endpoint identification (reflecting system response priority and safety).

[0156] Step C4-1: When the drone flies to the end of the corridor, the semantic understanding system identifies features of the study (such as bookshelves and computer desks) in the field of view ahead.

[0157] Step C4-2: If a pedestrian suddenly steps out, the motion generation system will prioritize detecting the dynamic obstacle through real-time visual changes (optical flow, depth abrupt changes) and immediately interrupt the original navigation trajectory to generate an emergency hovering or avoidance trajectory to ensure safety. After the obstacle disappears, the system resumes normal navigation based on the "enter room" semantic guidance vector.

[0158] Step C5: Precise landing and mission completion.

[0159] Step C5-1: After the drone enters the study, the semantic understanding system detects a flat area suitable for landing and issues a "landing" semantic guidance vector.

[0160] Step C5-2: The motion generation system controls the drone to descend smoothly and uses the downward-facing sensor to shut off the motors when the altitude is below 5cm, thus completing the landing.

[0161] Step C5-3: Send a task completion notification to the user. For example, a task completion notification can be sent to the user via Wi-Fi.

[0162] The above embodiments demonstrate how a drone navigates across rooms in a complex, confined indoor environment without GPS, completing a long sequence of structured language commands. This further highlights the superior performance of the proposed solution in task understanding, long-range planning, and fine-grained control. Specifically, it achieves automatic decomposition and sequential planning of complex long commands. When faced with complex task commands containing multiple continuous actions and landmark constraints, the semantic understanding system can decompose them into a structured sequence of task nodes and establish an internal task state machine. This endows the drone with the ability to understand and execute multi-step complex tasks, enabling it to autonomously complete the entire navigation process from the living room to the study.

[0163] This application also provides a drone control system based on a visual language action model, referring to... Figure 5 As shown, Figure 5 This is a schematic diagram of a drone control system based on a visual language action model, provided in an embodiment of this application. The system includes: The communication module is used to receive task instructions described in natural language. Visual sensors are used to collect real-time visual observation information about the environment. Inertial measurement unit (IMU) is used to sense the UAV's status information in real time. The onboard computing unit includes a first processor and a second processor; The first processor is configured to run a semantic understanding system at a first frequency to process current environmental visual observation information from the visual sensor and task instructions from the communication module, and to generate a semantic guidance vector, which is used to characterize task intent and environmental semantics. The semantic understanding system is built based on a visual language action model. The second processor is configured to run the action generation system at a second frequency to read the latest semantic guidance vector recently written from shared memory and generate flight control variables based on the latest semantic guidance vector, real-time environmental visual observation information, and real-time UAV status information; wherein the second frequency is higher than the first frequency; Shared memory, connected between the first processor and the second processor, serves as a data buffer interface for storing and transmitting semantic guidance vectors generated by the semantic understanding system; A flight controller, connected to the onboard computing unit, is used to control the UAV to perform flight missions based on the flight control inputs.

[0164] Specifically, the communication module serves as the entry point for the system to interact with external users, reliably receiving task instructions described in natural language from users via remote control, ground station, or mobile application. These instructions can be voice (converted to text after initial recognition) or direct text data. Upon receiving the task instructions, they can be transmitted via the internal bus to the onboard computing unit, particularly the primary processor, to initiate the task processing pipeline.

[0165] Visual sensors typically include forward-looking, downward-looking, or multi-view cameras (such as RGB cameras, depth cameras, and event cameras). Their core function is to continuously acquire raw image data of the environment surrounding the drone at a high frame rate, forming environmental visual observation information.

[0166] An inertial measurement unit (IMU) typically includes a three-axis accelerometer and a three-axis gyroscope, used to accurately measure the UAV's own motion state, including three-dimensional acceleration and angular velocity, and then calculate information such as attitude and velocity, i.e., "UAV state information." The UAV state information can be continuously provided to the onboard computing unit's second processor via a high-speed bus (such as SPI or I2C), and is a key feedback necessary for the motion generation system to perform high-frequency, closed-loop control.

[0167] The onboard computing unit is a heterogeneous multi-core or multi-chip computing platform. It is divided into two processing units with different functional and performance requirements, and the "dual-system" architecture is solidified in hardware. The first processor is usually a high-performance processor that supports parallel matrix operations, such as a GPU or NPU. This first processor is specifically configured to run the semantic understanding system. It is periodically triggered at a first frequency, receives task instructions from the communication module and current environmental visual observation information from the visual sensors, performs computationally intensive large-scale visual language model inference, generates semantic guidance vectors, and writes them to shared memory.

[0168] The second processor is typically a low-latency, high-determinism real-time processor, such as a high-performance CPU core, DSP, or FPGA. This second processor is specifically configured to run the action generation system, continuously cycling at a second frequency much higher than the first frequency. In each control cycle, it reads the latest semantic guidance vector written by the first processor from shared memory and combines it with real-time environmental visual observation information and real-time UAV state information to generate flight control quantities.

[0169] Shared memory is a key hardware infrastructure for achieving asynchronous decoupling. It is a physical memory region that is quickly accessed by the first and second processors, acting as an asynchronous data buffer. Specifically, after completing a semantic understanding inference, the first processor writes the generated semantic guidance vector into shared memory. At the beginning of each high-frequency control cycle, the second processor reads the latest memory guidance vector from shared memory. Because the first processor updates infrequently, between two updates, the second processor reads the same old but valid semantic guidance vector, thus ensuring that its control loop is never interrupted while waiting for new intents.

[0170] Flight controllers are typically dedicated microcontroller-based flight control boards responsible for the lowest-level attitude stabilization, motor drive, and safety monitoring. They connect to the onboard computing unit's second processor via a high-speed bus (such as CAN or UART), receiving flight control commands (such as target attitude angle and velocity instructions). Utilizing internal high-speed PID loops and model predictive controllers, they convert these higher-level commands into specific control signals, driving actuators (motors, servos) to precisely control the UAV's flight mission.

[0171] The technical solution adopted in this application embodiment involves customized hardware design based on the software requirements of a dual-system asynchronous architecture. Computationally intensive semantic understanding and reasoning tasks are assigned to a dedicated first processor, while high-real-time control tasks are assigned to a low-latency real-time processor (second processor). Efficient communication is achieved through shared memory, maximizing the performance of the heterogeneous hardware and meeting the stringent requirements of UAVs for computing power, power consumption, and real-time performance. The hardware buffering mechanism of shared memory and the independent high-frequency operating cycle of the second processor ensure a high degree of timing determinism in the output of control commands. Regardless of any accidental delays or fluctuations in the inference of the first processor, control commands can be stably output at a fixed high frequency, fundamentally avoiding uncertainties that may arise from thread scheduling and resource contention in the software architecture, and greatly improving the reliability and safety of the flight control system.

[0172] This application also provides an electronic device, see embodiments thereof. Figure 6 , Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example... Figure 6As shown, the electronic device 600 includes a memory 610 and a processor 620. The memory 610 and the processor 620 are connected via a bus for communication. The memory 610 stores a computer program that can run on the processor 620 to implement the steps of the UAV control method based on a visual language action model as described in the embodiments of this application.

[0173] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the UAV control method based on a visual language action model described in this application.

[0174] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the UAV control method based on a visual language action model described in this application.

[0175] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0176] This application describes embodiments of methods and systems according to embodiments of this application with reference to flowchart illustrations and / or block diagrams. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0177] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0178] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0179] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0180] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0181] The above provides a detailed description of a UAV control method and system based on a visual language action model. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for controlling a drone based on a visual language action model, characterized in that, include: Task instructions for obtaining natural language descriptions; The semantic understanding system, operating at a first frequency, infers based on current environmental visual observation information and the task instructions, generates a semantic guidance vector, and writes the semantic guidance vector into shared memory; wherein, the semantic guidance vector is used to represent the task intent and environmental semantics, and the semantic understanding system is constructed based on a visual language action model; The action generation system, operating at a second frequency, reads the latest semantic guidance vector recently written from the shared memory and generates flight control variables based on the latest semantic guidance vector, real-time environmental visual observation information, and real-time UAV status information; wherein the second frequency is higher than the first frequency. The UAV is controlled to perform flight missions based on the flight control parameters.

2. The UAV control method based on visual language action model according to claim 1, characterized in that, The semantic understanding system includes a first visual encoder, a text encoder, a large language model, and a feature projection layer; Based on the current environmental visual observation information and the task instructions, reasoning is performed to generate a semantic guidance vector, including: The first visual encoder encodes the current environmental visual observation information to obtain the first visual feature; The task instructions are encoded using the text encoder to obtain text features; The first visual feature and the text feature are fused and inferred across modally using the large language model to obtain high-level semantic features; The semantic guidance vector is generated by mapping the high-level semantic features through the feature projection layer.

3. The UAV control method based on visual language action model according to claim 2, characterized in that, When the task instruction contains multiple sub-tasks, the semantic guidance vector is generated by reasoning based on the current environmental visual observation information and the task instruction, and the process further includes: The task instructions are decomposed into multiple task nodes that are executed sequentially; each task node is associated with a specific task action and environmental features. Generate a semantic guidance vector corresponding to the target task node to be executed; Based on the current environmental visual observation information, if it is confirmed that the task action corresponding to the target task node has been completed, switch to the next task node and generate a semantic guidance vector corresponding to the next task node.

4. The UAV control method based on visual language action model according to claim 1, characterized in that, The motion generation system includes a motion expert network, a second visual encoder, and a state encoder; Based on the latest semantic guidance vector, real-time environmental visual observation information, and real-time UAV status information, flight control variables are generated, including: The real-time environmental visual observation information is encoded by the second visual encoder to obtain the second visual feature; The state encoder encodes the real-time UAV state information to obtain state features; The latest semantic guidance vector, the second visual feature, and the state feature are used as the generation conditions of the action expert network. The action expert network is then used to perform denoising and generation processes to generate a trajectory sequence. The flight control quantity is generated based on the trajectory points corresponding to the first M time steps in the trajectory sequence; where M is a positive integer greater than or equal to 1.

5. The UAV control method based on a visual language action model according to claim 1, characterized in that, Reading the latest semantic guidance vector, which was most recently written, from the shared memory includes: Between two runs of the semantic understanding system, the action generation system runs multiple times and repeatedly reads the latest semantic guidance vector most recently written by the semantic understanding system.

6. The UAV control method based on a visual language action model according to any one of claims 1-5, characterized in that, The semantic understanding system and the action generation system are trained through the following steps: Using the first training sample, the motion generation system to be trained is trained to learn the flight dynamics characteristics and obstacle avoidance priors of the UAV, and a preliminary trained motion generation system is obtained; wherein, the first training sample includes environmental visual observation information, UAV state information and real trajectory sequence. Using the second training sample, the semantic understanding system to be trained is fine-tuned with visual commands to learn the UAV mission perspective and mission command semantics, thus obtaining a preliminarily trained semantic understanding system; wherein, the second training sample includes aerial view images, mission commands, and the real trajectory sequence corresponding to the mission commands; Using the third training sample, the initially trained semantic understanding system and the initially trained action generation system are jointly fine-tuned to obtain the trained semantic understanding system and the trained action generation system; wherein, the third training sample includes: task instructions containing multi-level tasks, environmental visual observation information, and the real trajectory sequence corresponding to the task instructions. Based on reinforcement learning, the trained action generation system is optimized to obtain a fully trained action generation system.

7. The UAV control method based on a visual language action model according to claim 6, characterized in that, Based on reinforcement learning methods, the trained action generation system is optimized to obtain a fully trained action generation system, including: For a given target state, multiple candidate trajectory sequences are generated through the action expert network of the trained action generation system; wherein, the target state includes environmental visual observation information, UAV status information, and mission instructions; Construct a composite reward function to calculate the score for each candidate trajectory sequence; wherein, the composite reward function includes at least: the proximity reward between the trajectory endpoint corresponding to the candidate trajectory and the target position indicated by the task instruction; Based on the scores of multiple candidate trajectory sequences, the relative advantage signal of each candidate trajectory sequence is calculated, and the parameters of the trained action generation system are updated according to the relative advantage signal to obtain the trained action generation system.

8. The UAV control method based on a visual language action model according to claim 7, characterized in that, Based on the scores of multiple candidate trajectory sequences, the relative advantage signal of each candidate trajectory sequence is calculated, including: Based on the scores of multiple candidate trajectory sequences, the average score and standard deviation of the multiple candidate trajectory sequences are calculated; The difference between the score of each candidate trajectory sequence and the average score is calculated and normalized using the standard deviation to obtain the relative advantage signal of each candidate trajectory sequence.

9. The UAV control method based on a visual language action model according to claim 7, characterized in that, Based on the relative advantage signal, the parameters of the trained action generation system are updated, including: Candidate trajectory sequences with positive relative advantage signals are identified as advantageous trajectory sequences, and candidate trajectory sequences with negative relative advantage signals are identified as disadvantageous trajectory sequences. The parameters of the trained action generation system are updated with the goal of increasing the generation probability of the advantageous trajectory sequence and decreasing the generation probability of the disadvantageous trajectory sequence.

10. A drone control system based on a visual language action model, characterized in that, include: The communication module is used to receive task instructions described in natural language. Visual sensors are used to collect real-time visual observation information about the environment. Inertial measurement unit (IMU) is used to sense the UAV's status information in real time. The onboard computing unit includes a first processor and a second processor; The first processor is configured to run a semantic understanding system at a first frequency to process current environmental visual observation information from the visual sensor and task instructions from the communication module, and to generate a semantic guidance vector, which is used to characterize task intent and environmental semantics. The semantic understanding system is built based on a visual language action model. The second processor is configured to run the action generation system at a second frequency to read the latest semantic guidance vector recently written from shared memory and generate flight control variables based on the latest semantic guidance vector, real-time environmental visual observation information, and real-time UAV status information; wherein the second frequency is higher than the first frequency; Shared memory, connected between the first processor and the second processor, serves as a data buffer interface for storing and transmitting semantic guidance vectors generated by the semantic understanding system; A flight controller, connected to the onboard computing unit, is used to control the UAV to perform flight missions based on the flight control inputs.

Images