A method for controlling a UAV under language condition based on a world action model

Abstract

The application discloses a kind of unmanned plane control method under language condition based on world action model, first, based on unmanned plane dynamics simulator generates physically consistent unmanned plane state, expert trajectory and flight action, and obtain first perspective RGB observation in combination with high-fidelity rendering environment and three-dimensional Gaussian sputtering rendering environment, from this, the multimodal flight data including vision, state, action and natural language task text are constructed.Second, based on pre-training video diffusion Transformer, construct action center world action model, encode history first perspective image, unmanned plane own state and natural language instruction as conditional information, simultaneously learn action block prediction and future first perspective visual prediction in training stage, and avoid action decoding to leak future image information by block causal attention mask.Then, adopt the five-order trajectory action representation in yaw alignment local coordinate system, convert model output into continuous position, velocity and acceleration reference, so that the generated action is more suitable for the execution of unmanned plane bottom layer trajectory tracking controller.Finally, in the deployment stage, future video generation is omitted, and only local trajectory action block is directly output, and self-guided reasoning is carried out using the overlapping part of adjacent action blocks, so as to reduce reasoning delay, reduce trajectory jump and support unmanned plane to complete target search, target tracking and closed-loop autonomous flight tasks according to natural language instruction.

Description

Technical Field

[0001] This invention belongs to the fields of intelligent control of unmanned aerial vehicles (UAVs), robot learning, and visual-language navigation. Specifically, it relates to a method for training and controlling a world action model that generates executable flight action sequences based on first-person visual observation, the UAV's own state, and natural language commands. This method can be used for target search, target tracking, semantic navigation, and closed-loop autonomous flight control of UAVs. Background Technology

[0002] Multi-rotor drones have been applied in scenarios such as inspection, security, search and rescue, warehousing, and film and television shooting. As the complexity of tasks increases, operators no longer expect drones to perform simple waypoint flights, but rather to understand natural language commands, such as "fly to the yellow mat," "follow the target ahead," or "find the specified object," and to keep the target visible, avoid obstacles, and output smooth control commands that conform to dynamic constraints during flight.

[0003] Existing autonomous flight methods for unmanned aerial vehicles (UAVs) mainly include traditional modular methods and learning-based methods. Traditional methods typically consist of modules such as perception, mapping, localization, trajectory planning, and control. While they can generate relatively stable trajectories in structured scenarios, their long system chains make upstream and downstream errors prone to propagation, and their adaptability to open-category targets and natural language commands is limited. Learning-based methods often replace some perception and decision-making modules with reinforcement learning, imitation learning, or visual-language-action models, improving semantic understanding and reaction speed. However, several problems still exist in UAV scenarios.

[0004] First, many aerial visual language navigation or visual language action methods still use discrete actions, high-level waypoints, motion primitives, or instantaneous velocities as action representations. These interfaces are easy to train, but they are difficult to express the smooth trajectory constraints of UAVs in continuous three-dimensional space, and they cannot directly depict the impact of the current action on the future first-person view.

[0005] Secondly, world models and world-action models learn the consequences of actions by predicting future observations, and have already been applied in tasks such as robotic arms. However, the first-person perspective of drones changes rapidly, and the coupling between actions and future images is strong. If future videos are generated autoregressively during actual flight, inference latency will increase; small errors in the generated images may also be transmitted to the action decoding process, causing trajectory jitter, target loss, or collisions.

[0006] Furthermore, training a drone world motion model requires strictly synchronized multimodal data, including first-person RGB images, natural language commands, the drone's own state, and dynamically feasible flight maneuvers. Ordinary video datasets lack control actions, and common visual language navigation datasets lack continuous drone dynamic constraints. Directly collecting data on a large scale using real drones is costly, risky, and inefficient.

[0007] Therefore, a training and control method for language-conditional UAV flight is needed. This method should utilize future visual consequences for supervision during training, directly output smooth and executable local trajectory actions during inference, and obtain multimodal data aligned with vision, language, state, and action through simulation. Summary of the Invention

[0008] In view of this, the purpose of this invention is to provide a UAV control method based on a world action model under language conditions. The method uses historical first-person view images, the UAV's own state, and natural language commands as inputs to train a model to predict a local trajectory action block. During the training phase, the model learns both action prediction and future first-person view visual prediction simultaneously. During the deployment phase, the model does not generate future videos, but only decodes action blocks and maintains the continuity of adjacent action blocks through a self-guiding mechanism.

[0009] A method for controlling unmanned aerial vehicles (UAVs) under language conditions based on a world action model includes the following steps:

[0010] S1. Constructing Language-Conditioned Multimodal Flight Data: Using a UAV dynamics simulator, physically consistent UAV state, expert trajectory, and action data are generated. Combined with a high-fidelity rendering environment, first-person RGB observations are obtained to form a multimodal dataset containing RGB video, UAV state, flight actions, and natural language task text. S2. Training the Action-Centered World Action Model: A model is built based on a pre-trained video diffusion Transformer, encoding historical first-person perspective images as reference visual lexical units. Encode the drone's own state into state words. Encode natural language task text into language lexical units Map noisy action blocks or action blocks to be predicted to action lexes. Encoding future images into future visual terms ; The model is based on non-linguistic lexical sequences and language lexical units as conditional input It learns to predict actions and future vision. S3. Self-guided action block reasoning: During the model reasoning stage, the current historical image, the UAV's own state, and the language command are input and encoded respectively. The model decodes and outputs local trajectory action blocks. When the model outputs action blocks, it performs self-guided reasoning. Since the suffix of the previous action block and the prefix of the current action block overlap in the generation time, the model uses the suffix of the previous action block to constrain the generation of the prefix of the current action block, ensuring the temporal continuity of adjacent action blocks. S4. Closed-loop UAV control: The current action block output by the model is converted into a local fifth-order polynomial trajectory segment, generating continuous position, velocity and acceleration references, which are sent to the UAV's underlying trajectory tracking controller for execution, enabling autonomous flight of the UAV under language conditions.

[0011] Preferably, the video diffusion Transformer utilizes a block-based causal attention masking mechanism, namely: state terms and reference visual terms only encode the current context and do not access prediction terms; action terms can access state terms, reference visual terms, and action terms, but not future visual terms; future visual terms can access state terms, reference visual terms, and action terms. Through this masking, future visual prediction is conditionalized by action blocks, and action decoding will not leak information from future images.

[0012] Preferably, the multimodal flight data constructed in step S1 includes the following three mission types: Target tracking task: The expert controller controls the drone to follow a moving target and keep the target within the field of view; Language-based target arrival task: In a high-fidelity rendered scene, control the drone to approach the target from a random initial pose according to the target specified by natural language instructions; 3D Gaussian Scene Target Arrival Task: Combine sampled 3D Gaussian scenes and target objects to generate target arrival trajectories under various scene layouts.

[0013] Preferably, the loss function used for model training in step S2 includes action prediction loss and visual consequence prediction loss, and its expression is:

[0014] in:

[0015]

[0016] The motion prediction loss is used to supervise the generation of flight motion blocks in the model. To supervise the loss of future first-person visual consequences caused by actions predicted by the model. The weights are for visual loss. Indicates the expectation. ( ) represents the mapping of the model from the input to the motion velocity field. ( ) represents the model's mapping from the input to the future visual latent lexical velocity field; The target velocity field corresponding to the action word. The target velocity field corresponding to the potential lexical units of the future vision.

[0017] Preferably, the UAV's own state and actions are represented in a yaw-aligned local coordinate system; the x-axis of this local coordinate system is located in the vertical plane where the UAV's x-axis is located, and the z-axis is vertically upward.

[0018] Preferably, each action in the action block represents a local trajectory, including the trajectory endpoint displacement, endpoint velocity, and endpoint acceleration; For each spatial dimension The local trajectory is represented by a fifth-order polynomial:

[0019] in, Indicates the position of the UAV in the local coordinate system as a function of time. A changing function, These are the coefficients corresponding to the five orders of the fifth-order polynomial of the trajectory; Given starting position Starting speed Starting acceleration and the finish line Final speed and terminal acceleration Then, regarding the three spatial dimensions Solve for the coefficients of the fifth-order polynomial:

[0020] in, This represents the trajectory time length.

[0021] Preferably, the self-guided reasoning process in step S3 specifically includes: For the The first reasoning Prefix of time step The inference prefix part of the previous time step of this inference and the suffix of the previous reasoning result The result is obtained by iteratively superimposing the results; for the th The first reasoning The suffix of time step It is obtained by iteratively superimposing the following two parts: the first part is the inference suffix of the previous time step in this inference. The second part is: The specific process is as follows:

[0022]

[0023] in, The length of the action block. The length of the overlapping prefix, determined by the inference delay and the data time step. The starting index in the previous action block that corresponds to the prefix of the current action block. For sampling or denoising time steps, the step size is... .

[0024] The present invention has the following beneficial effects: 1. This method applies a world-action model to language-conditional UAV flight. During training, future first-person view visual predictions are used as additional supervision signals besides actions, enabling the model to improve its visual causal understanding ability by learning the relationship between visual observations and actions.

[0025] 2. Data construction utilizes simulation dynamics, high-fidelity rendering, and 3D Gaussian sputtering rendering to obtain visual, linguistic, state, and action-aligned data. By constructing a high-quality simulation data collection process, the cost of UAV data acquisition is reduced, thereby mitigating the costs and risks associated with directly using real UAVs to collect large-scale training data.

[0026] 3. The motion uses fifth-order polynomial trajectory parameters in a local coordinate system, rather than discrete motion, waypoints, or instantaneous velocity. The generated trajectory can be converted into continuous position, velocity, and acceleration references, making it more suitable for execution by the UAV's underlying controller.

[0027] 4. The self-guiding mechanism constrains the overlapping parts of adjacent action blocks, making the trajectory obtained by continuous reasoning more consistent in time, and reducing the risk of UAV attitude jitter and crash caused by sudden changes in control commands during closed-loop flight.

[0028] 5. During deployment, the model does not generate future videos, but only directly outputs local trajectory action blocks, which reduces inference latency and memory usage, and reduces the impact of future image prediction errors on action decoding. Attached Figure Description

[0029] Figure 1 This is a schematic diagram of the overall process of the motion center world motion model training and control method of the present invention; Figure 2 This is a schematic diagram of the motion center world motion Transformer structure of the present invention; Figure 3 This is a schematic diagram of the hybrid data generation process of the present invention; Figure 4 This is a schematic diagram of the self-guided reasoning of adjacent action blocks in this invention; Figure 5 This is a schematic diagram of the closed-loop flight control platform for unmanned aerial vehicles (UAVs) of the present invention. Detailed Implementation

[0030] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0031] This invention provides a method for controlling unmanned aerial vehicles (UAVs) under language conditions based on a world action model, which mainly includes the following:

[0032] 1. Construction of Language-Conditioned Multimodal Flight Data. Using a UAV dynamics simulator, such as DiffAero, physically consistent UAV state, expert trajectories, and motion data are generated. These are then combined with a high-fidelity rendering environment and a 3D Gaussian sputtering rendering environment to obtain first-person RGB observations. Each trajectory includes at least RGB video, UAV state, flight motions, and natural language task text. Simulation tasks include target tracking, language-conditional target arrival, and 3D Gaussian scene target arrival, covering motion prediction, semantic target search, and scene generalization.

[0033] 2. Action-Centered World Action Model. The model is built upon a pre-trained video diffusion Transformer, comprising a video variational autoencoder, text encoder, state encoder, action encoder, action decoder, and backbone Transformer. The model encodes historical first-person view images as reference visual units, the drone's own state as state units, the action blocks to be predicted as action units, and natural language commands as language units. During training, future images are encoded as future visual units. Action prediction and future visual prediction are performed within the same Transformer.

[0034] 3. Block-based causal attention mask for action decoding. State terms and reference visual terms do not access the action terms to be predicted or future visual terms; action terms only access state terms, reference visual terms, and action terms; future visual terms can access state terms, reference visual terms, action terms, and future visual terms. This ensures that action prediction does not leak future image information, and future image prediction remains constrained by the action block. During inference, future visual terms are removed, retaining only the action decoding branch, thereby reducing computational load and preventing generated image errors from continuing to affect control.

[0035] 4. Smooth Local Trajectory Motion Representation. The model outputs motion not as discrete commands or instantaneous velocities, but as trajectory segment parameters in a local coordinate system. This local coordinate system is aligned with the UAV's yaw, with its x-axis located in the vertical plane containing the body's x-axis, and its z-axis pointing vertically upwards. The UAV's own state can be composed of local velocity, local acceleration, and the body's z-axis direction; the motion can be composed of parameters such as trajectory endpoint displacement, endpoint velocity, and endpoint acceleration, and continuous position, velocity, and acceleration references within a finite time are generated using a fifth-order polynomial.

[0036] 5. Action Block Self-Guided Inference. In continuous closed-loop control, action blocks obtained from two adjacent inferences often overlap in time. To reduce trajectory reference jumps caused by independent sampling, this method uses the suffix of the previous action block to constrain the generation of the prefix of the current action block during denoising or sampling, and only samples the subsequent time periods that do not overlap normally. This process does not change the model structure and does not require additional training.

[0037] 6. Closed-loop UAV control. During deployment, the UAV or external computing device receives the current first-person view image sequence, the UAV's own status, and voice commands. The model outputs local trajectory action blocks. The system converts the action blocks into local fifth-order trajectory segments and then sends them to the trajectory tracking controller for execution. Model inference can be performed on the onboard computing device or by a remote workstation, and the action blocks are returned via network communication.

[0038] like Figure 1 As shown, this method comprises five stages: data acquisition and construction, model training, action block inference, self-guided sampling, and UAV closed-loop execution. The system input consists of historical first-person view images of the UAV, the UAV's own state, and natural language commands. The output is a local trajectory action block that can be executed by the trajectory controller.

[0039] I. Construction of Multimodal Flight Data In one embodiment, the simulation data is based on UAV dynamics systems such as DiffAero and expert controllers. Physically consistent UAV states, trajectories, and actions are first generated, and then first-person view images are obtained through different rendering branches.

[0040] The simulation data includes the following tasks.

[0041] 1. Target Tracking Task. A target moves through a random corridor or scene. An expert controller guides a drone to keep the target within its field of view and follow its movement. This task trains a model to estimate relative motion trends based on historical vision and state, and generates a trajectory to keep the target visible.

[0042] 2. Language-based Target Arrival Task. A target object and interfering objects are placed in a high-fidelity rendered scene. A verbal command specifies the target to be reached, and the drone approaches the target from a random initial pose. This task is used to train a model to map natural language target descriptions to first-person visual content.

[0043] 3. 3D Gaussian Scene Target Arrival Task. This task combines sampled 3D Gaussian scenes and target objects to generate target arrival trajectories in various scenes. It aims to increase the variety of scene layout, object appearance, and visual background.

[0044] Each original trajectory includes at least rgb.mp4, obs.npy, rpyva.npy, and task.txt. rgb.mp4 is a first-person perspective RGB video, obs.npy contains the UAV's observations and its own state, rpyva.npy contains the action sequence fitted by expert trajectories, and task.txt contains a natural language task description. During data preprocessing, video frames, states, actions, text encodings, and metadata are organized into training data, and normalized statistics for actions and states are calculated.

[0045] II. Observation Space and Action Space In one embodiment, the model at time... Receive natural language instructions Current status of the drone And historical first-person view image sequences sampled at time intervals:

[0046] in, Number of historical reference images The time sampling interval is denoted as .

[0047] The model output length is Action blocks:

[0048] Each action in an action block represents a local trajectory, rather than a single-step discrete action or an instantaneous velocity command.

[0049] In one training implementation, the number of historical reference images Time sampling interval Action block length The simulation data was acquired at 30 Hz, and the video was sampled at 3 times the time interval, so each action block corresponds to 24 low-level action steps and 8 future vision frames.

[0050] The UAV's own state and actions are represented in a yaw-aligned local coordinate system. The x-axis of this local coordinate system lies in the vertical plane containing the UAV's own x-axis, and the z-axis points vertically upwards. The UAV's own state is represented as:

[0051] in, For local velocity, For local acceleration, This represents the unit vector of the body along the z-axis in the local coordinate system.

[0052] For each spatial dimension The local trajectory segments are represented using fifth-order polynomials:

[0053] in, The coefficients correspond to the five orders of the trajectory polynomial; the action is represented as:

[0054] in, and These are the velocity and acceleration at the end of the trajectory, respectively. These represent the length, pitch angle, and yaw angle of the line connecting the endpoint and the starting point of the local trajectory in the local coordinate system, respectively, and are used to describe the relative displacement of the trajectory endpoint:

[0055] Given starting position Starting speed Starting acceleration and the finish line Final speed and terminal acceleration Then, it is possible to consider three spatial dimensions. Solve for the coefficients of the fifth-order polynomial:

[0056] in, The trajectory time length is denoted as . The resulting local trajectory segment is continuous in position, velocity, and acceleration, and can be directly converted into the reference input of the trajectory tracking controller.

[0057] III. Action Center World Action Model Structure like Figure 2 As shown, in one embodiment, the action-centric world action model is built based on a pre-trained video diffusion Transformer. The model includes the following modules.

[0058] 1. Video Coding Module. This module uses a video variational autoencoder to encode historical images and future images from the training phase into spatiotemporal latent terms, and incorporates 3D rotational position encoding. Historical images correspond to reference visual terms. Future images correspond to future visual terms .

[0059] 2. State Coding Module. This module uses a multilayer sensor to encode the drone's own state. Mapped to state lexical .

[0060] 3. Motion encoding and motion decoding module. The motion encoding module decodes noisy or unpredictable motion blocks. Mapped to action lexical The motion decoding module reconstructs the denoised velocity field corresponding to the motion block based on the Transformer output.

[0061] 4. Text Encoding Module. This module uses a pre-trained text encoder to encode natural language instructions. Encoding as language lexical units And serve as the cross-attention condition for Transformer.

[0062] 5. Core Transformer. This module receives state lexical units, reference visual lexical units, action lexical units, and future visual lexical units from the training phase, forming a non-linguistic lexical sequence:

[0063] Language lexical As conditional inputs, the model learns action prediction and future visual prediction within the same sequence.

[0064] To prevent action prediction from using future visual information, this method employs a block-based causal attention mask. State and reference visual terms encode only the current context and do not access prediction terms; action terms can access state, reference, and action terms, but not future visual terms; future visual terms can access state, reference, and action terms. Through this mask, future visual prediction is conditionally defined by action blocks, ensuring that action decoding does not leak information from future images.

[0065] IV. Model Training Methods In one embodiment, the loss function used for model training includes action prediction loss and visual consequence prediction loss. The model can employ a diffusion model or flow matching to learn denoising for action terms and future visual terms. Let... For the target noise or velocity field corresponding to the action word, If the target noise or velocity field corresponds to the future visual latent word, then the loss function is:

[0066] in:

[0067]

[0068] The motion prediction loss is used to supervise the generation of flight motion blocks in the model. To supervise the loss of future first-person visual consequences caused by actions predicted by the model. The weights are for visual loss. Indicates the expectation. ( ) represents the mapping of the model from the input to the motion velocity field. () represents the model's mapping from input to the future visual latent lexical velocity field; the visual prediction loss is only used during the training phase to constrain the model's spatiotemporal representation and action priors; future videos are not generated during the inference phase.

[0069] In one training implementation, the model uses a pre-trained Wan family video diffusion Transformer as its backbone. Video encoders, text encoders, and video generation priors provide spatiotemporal visual modeling capabilities; new state encoders, motion encoders, and motion decoders are added to adapt to UAV flight control. The state encoder, motion encoder, and motion decoder can employ multilayer perceptrons with hidden dimensions of 128 or 256.

[0070] The training data covers target tracking, Isaac high-fidelity target arrival, and 3D Gaussian scene target arrival. After training, the deployed system can load only the branches required for action decoding, omitting future vision decoding.

[0071] V. Reasoning and Self-Guided Action Block Generation During deployment, the system receives the most recent The model takes a first-person view frame, its current state, and natural language instructions. It encodes historical images, states, and language, and generates action blocks without inputting future visual lexical units. Because the inference process does not generate future video, its computational cost and latency are lower than a full-video predictive world action model.

[0072] In continuous closed-loop control, the model repeats inference at a fixed frequency. For example... Figure 4 As shown, let the first... The action block generated by the second inference has not yet been fully executed. If the next inference has already begun, the action blocks generated by the two inferences will overlap in time. If each action block is sampled independently, the overlapping parts may be inconsistent, and the reference trajectory received by the underlying controller will show abrupt changes.

[0073] like Figure 4 As shown, this method incorporates self-guidance during the action denoising or sampling process. For the first... In the second inference, within the time window from the start to the end of the inference, the actions performed by the drone still obey the first inference. The result of the first inference; the result generated after the inference is completed. The inference result still includes the actions the drone should have performed during the inference period. Because the actions output by the model in a single inference are temporally continuous—that is, the temporally adjacent actions within a single action block are relatively similar—to ensure the success of the second inference, the inference result must be consistent with the actions that were performed during the inference period. The next reasoning ends; the source of the drone's control signals is the action block. Switch to action block The control commands remain continuous, in the first... During the next inference process, model outputs within overlapping prefix intervals are discarded. , and instead use the first The reasoning results of the next inference within the corresponding time window The model is guided to output the first value in the non-overlapping interval. The next inference involves actions that are relatively continuous in time; for the remaining non-overlapping suffix intervals, the current model still samples normally based on the latest observations, state, and language. This process can be represented as:

[0074]

[0075] in, The length of the action block. The length of the overlapping prefix, determined by the inference delay and the data time step. The starting index in the previous action block that corresponds to the prefix of the current action block. For sampling or denoising time steps, the step size is... .

[0076] From the first formula above, we can see that for the... The first reasoning The prefix (overlapping part) of the time step The inference prefix part of the previous time step of this inference And the suffix (overlapping part) of the previous reasoning result. The results are obtained by iteratively superimposing the superimposed results. From the second formula above, we can see that for the... The first reasoning The suffix of time step It is obtained by iteratively superimposing the following two parts: the first part is the inference suffix of the previous time step in this inference. The second part is: ; This self-guiding process does not require retraining the model or changing its structure.

[0077] VI. Closed-loop flight control like Figure 5 As shown, in one embodiment, the UAV platform includes a first-view camera, an onboard computing device, a flight controller, and a state measurement module. Model inference can be performed on the onboard computing device or on a remote workstation. When using a remote workstation for inference, the UAV sends image, state, and voice task information via a local area network or other communication link, and the remote workstation returns action blocks.

[0078] After receiving the action block, the system calculates a local fifth-order trajectory segment based on the endpoint displacement, velocity, and acceleration of the action. This trajectory segment is then converted into reference inputs for the underlying trajectory tracking controller, such as position, velocity, acceleration, and attitude-related references. The underlying controller tracks this trajectory according to the current UAV state and outputs motor, attitude, or acceleration control commands.

[0079] In one specific embodiment, the model inference server receives first-view image history, the current state of the UAV, and task text, and outputs an action block of length 24. The UAV requests model inference at a frequency lower than the data sampling frequency and continuously executes the previously generated local trajectory between adjacent inferences. Since the action block length is greater than the single inference interval, a self-guiding mechanism is used to ensure the continuity of new and old action blocks in overlapping areas.

[0080] VII. Experimental Examples In one experimental implementation, training data was acquired at 30 Hz, and the video was downsampled at 3 times the time interval. Each sample in the model used 9 historical reference images, 8 future visual supervision frames, and 24 action blocks. The model backbone employed a 1.3B parameter-level video spread Transformer, while the state encoder, motion encoder, and motion decoder used a multilayer perceptron.

[0081] In simulation experiments, the model performed target tracking and language-conditional target search tasks in a high-fidelity simulation environment. Compared with the model using only a single frame of first-person view image, the model using 9 historical images can better estimate relative motion and scene changes, thereby improving the target tracking success rate, the target search success rate, and reducing the collision rate.

[0082] In real-world flight experiments, the UAV platform can consist of a UAV frame, a first-person view camera, an onboard computing unit, and a state measurement system. Natural language commands specify the target object, and the model generates local trajectory action blocks based on visual history, state, and commands. A trajectory tracking controller then guides the UAV towards the target. Experimental results demonstrate that this method can connect visual language task understanding, action block generation, and UAV closed-loop control, enabling language-conditional flight on a physical UAV.

[0083] In summary, the above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for controlling unmanned aerial vehicles (UAVs) under language conditions based on a world action model, characterized in that, Includes the following steps: S1. Constructing Language-Conditioned Multimodal Flight Data: Using a UAV dynamics simulator, physically consistent UAV state, expert trajectory, and action data are generated. Combined with a high-fidelity rendering environment, first-person RGB observations are obtained to form a multimodal dataset containing RGB video, UAV state, flight actions, and natural language task text. S2. Training the Action-Centered World Action Model: A model is built based on a pre-trained video diffusion Transformer, encoding historical first-person perspective images as reference visual lexical units. Encode the drone's own state into state words. Encode natural language task text into language lexical units Map noisy action blocks or action blocks to be predicted to action lexes. Encoding future images into future visual terms ; The model is based on non-linguistic lexical sequences and language lexical units as conditional input It learns to predict actions and future vision. S3. Self-guided action block reasoning: During the model reasoning stage, the current historical image, the UAV's own state, and the language command are input and encoded respectively. The model decodes and outputs local trajectory action blocks. When the model outputs action blocks, it performs self-guided reasoning. Since the suffix of the previous action block and the prefix of the current action block overlap in the generation time, the model uses the suffix of the previous action block to constrain the generation of the prefix of the current action block, ensuring the temporal continuity of adjacent action blocks. S4. Closed-loop UAV control: The current action block output by the model is converted into a local fifth-order polynomial trajectory segment, generating continuous position, velocity and acceleration references, which are sent to the UAV's underlying trajectory tracking controller for execution, enabling autonomous flight of the UAV under language conditions.

2. The method according to claim 1, characterized in that, The video diffusion Transformer utilizes a block-based causal attention masking mechanism, where: state terms and reference visual terms only encode the current context and do not access prediction terms; action terms can access state terms, reference visual terms, and action terms, but not future visual terms; future visual terms can access state terms, reference visual terms, and action terms. Through this masking, future visual prediction is conditionalized by action blocks, and action decoding will not leak information from future images.

3. The method according to claim 1, characterized in that, The multimodal flight data constructed in step S1 includes the following three mission types: Target tracking task: The expert controller controls the drone to follow a moving target and keep the target within the field of view; Language-based target arrival task: In a high-fidelity rendered scene, control the drone to approach the target from a random initial pose according to the target specified by natural language instructions; 3D Gaussian Scene Target Arrival Task: Combine sampled 3D Gaussian scenes and target objects to generate target arrival trajectories under various scene layouts.

4. The method according to claim 1, characterized in that, The loss function used in model training in step S2 includes action prediction loss and visual consequence prediction loss, and its expression is: in: The motion prediction loss is used to supervise the generation of flight motion blocks in the model. To supervise the loss of future first-person visual consequences caused by actions predicted by the model. The weights are for visual loss. Indicates the expectation. ( ) represents the mapping of the model from the input to the motion velocity field. ( ) represents the model's mapping from the input to the future visual latent lexical velocity field; The target velocity field corresponding to the action word. The target velocity field corresponding to the potential lexical units of the future vision.

5. The method according to claim 1, characterized in that, The UAV's own state and actions are represented in a yaw-aligned local coordinate system; the x-axis of this local coordinate system is located in the vertical plane where the UAV's x-axis is located, and the z-axis is vertically upward.

6. The method according to claim 1, characterized in that, Each action in an action block represents a local trajectory, including the trajectory endpoint displacement, endpoint velocity, and endpoint acceleration; For each spatial dimension The local trajectory is represented by a fifth-order polynomial: in, Indicates the position of the UAV in the local coordinate system as a function of time. A changing function, These are the coefficients corresponding to the five orders of the fifth-order polynomial of the trajectory; Given starting position Starting speed Starting acceleration and the finish line Final speed and terminal acceleration Then, regarding the three spatial dimensions Solve for the coefficients of the fifth-order polynomial: in, This represents the trajectory time length.

7. The method according to claim 1, characterized in that, The self-guided reasoning process in step S3 is specifically as follows: For the The first reasoning Prefix of time step The inference prefix part of the previous time step of this inference and the suffix of the previous reasoning result The result is obtained by iteratively superimposing the results; for the th The first reasoning The suffix of time step It is obtained by iteratively superimposing the following two parts: the first part is the inference suffix of the previous time step in this inference. The second part is: The specific process is as follows: in, The length of the action block. The length of the overlapping prefix, determined by the inference delay and the data time step. The starting index in the previous action block that corresponds to the prefix of the current action block. For sampling or denoising time steps, the step size is... .

Images