Unmanned aerial vehicle task execution method based on visual language action model semantic target

Abstract

The application discloses a UAV task execution method based on visual language action model semantic target, and belongs to the field of artificial intelligence and UAV control. The method aims to solve the problems of poor flight safety guarantee, uninterpretable decision process and difficulty in dealing with sudden obstacles caused by the end-to-end method of the existing VLA model directly outputting low-level control instructions, and specifically comprises the following steps: acquiring a multi-modal input data stream in real time at the current time; processing the multi-modal input data stream by using a visual language action model to obtain an intermediate semantic target; acquiring real-time environment perception data collected by an environment perception sensor carried by a UAV; taking the intermediate semantic target as input and taking the real-time environment perception data as constraint, performing safety motion planning on the UAV to generate a flight trajectory reaching the intermediate semantic target, which is dynamically feasible and collision-free; and controlling the UAV to execute the flight trajectory.

Description

Technical Field

[0001] This invention belongs to the field of autonomous control technology for unmanned aerial vehicles (UAVs), and particularly relates to a method for UAV mission execution based on semantic targets of visual language action models. Background Technology

[0002] The autonomy of drones is crucial for their ability to perform tasks in complex environments. Currently, drone control primarily relies on rigid pre-programmed waypoints or remote control requiring significant human intervention. These methods struggle to adapt to dynamic and unpredictable field environments and cannot comprehend the high-level, ambiguous natural language commands from operators. With the development of multimodal artificial intelligence, VLA (Vision-Language-Action) models have demonstrated powerful semantic understanding and action generation capabilities. Recent research has explored the use of fine-tuned VLA models for drone control, enabling drones to respond to commands such as "fly over that door" or "fly around the tree."

[0003] However, these existing VLA drone control schemes generally employ an end-to-end imitation learning paradigm. In this paradigm, the VLA model is trained as a "black box," directly mapping the input first-person view images and verbal commands to the drone's low-level motion commands (such as a four-dimensional velocity vector). This end-to-end approach has serious technical drawbacks:

[0004] Lack of safety guarantees: VLA models are statistical models trained on massive amounts of data, and their outputs lack mathematically verifiable safety. When a drone flies at high speed in three-dimensional space, if the model encounters a scene not seen in the training dataset, its output actions may lead to catastrophic collisions.

[0005] Poor robustness and generalization: As existing research has shown, the low-level control assumptions of ground robots do not hold true for the complex 3D flight dynamics of UAVs. Forcing VLA models to simultaneously learn to "understand commands" and "learn to fly" increases the training burden on the model, potentially leading to a decline in performance in either aspect.

[0006] Deployment delay issue: Large VLA models consume enormous amounts of computing power, making real-time deployment on UAV onboard computers difficult. Therefore, existing technologies rely on "ground-air collaboration," i.e., inference is performed on ground station servers. However, this introduces unavoidable network communication latency, which is extremely dangerous for lower-level flight control systems that require rapid responses.

[0007] In summary, there is an urgent need for a new control architecture that can leverage the powerful semantic understanding capabilities of the VLA model while ensuring the safety, real-time performance, and robustness of UAV flight. Summary of the Invention

[0008] The core objective of this invention is to address the lack of security and robustness in existing VLA (Visual Language Action) drone control methods due to their end-to-end architecture. This invention proposes a drone mission execution method based on a visual language action model semantic target. Instead of directly generating low-level actions, the VLA model generates a high-level intermediate target semantic, thereby decoupling semantic understanding from safe execution.

[0009] According to a first aspect of the embodiments of this application, a method for executing unmanned aerial vehicle (UAV) missions based on semantic targets of visual language action models is provided, including:

[0010] The system acquires multimodal input data streams in real time, including natural language commands, real-time visual data, and real-time status data. The natural language commands represent the tasks to be performed by the UAV, the real-time visual data is collected by the UAV's first-person view camera, and the real-time status data are the UAV's flight dynamics parameters.

[0011] The multimodal input data stream is processed using a visual language action model to obtain intermediate semantic targets;

[0012] The system acquires real-time environmental perception data collected by the environmental perception sensors on the UAV, uses the intermediate semantic target as input and the real-time environmental perception data as constraints, performs safe motion planning for the UAV, and generates a dynamically feasible and collision-free flight trajectory to reach the intermediate semantic target.

[0013] Control the drone to execute the stated flight path.

[0014] Furthermore, the intermediate semantic target includes at least one of the following:

[0015] One or more three-dimensional space waypoints in the global coordinate system or the UAV body coordinate system;

[0016] A relative pose associated with a specific object in the environment;

[0017] A semantic label representing the state of a task.

[0018] Furthermore, an optimization-based trajectory planner or a sampling-based trajectory planner is used to plan the safe movement of the drone.

[0019] Furthermore, the environmental perception sensor is a depth camera or a lidar, and the real-time environmental perception data is a depth map or a 3D point cloud.

[0020] Furthermore, the visual language action model includes a visual encoder, a text encoder, a state encoder, a fusion module, and a target decoder. The visual encoder, text encoder, and state encoder are used to extract features from real-time visual data, natural language instructions, and real-time state data in the multimodal input data stream, respectively. The fusion module is used to deeply fuse the features extracted by the three encoders. The target decoder generates a high-level intermediate semantic target based on the fused features.

[0021] Furthermore, the visual language action model is trained through imitation learning, and its training method includes:

[0022] Obtain a training dataset containing visual data, language commands, and expert flight trajectories;

[0023] Semantic targets are extracted from the expert's flight trajectory and used as training labels;

[0024] The training objective is set to minimize the difference between the intermediate semantic target output by the visual language action model and the training label;

[0025] The visual language action model is trained until the training objective is met.

[0026] Furthermore, semantic targets are extracted from the expert's flight trajectory, including:

[0027] Task types are classified according to language instructions, and the task types include at least arrival tasks, circling tasks, and following tasks;

[0028] Preprocessing of expert flight trajectories includes removing outliers and smoothing the trajectory;

[0029] When the task type is an arrival task, the speed of the expert trajectory at the end of several consecutive time steps is calculated. If it is consistently less than the predetermined speed threshold, it is considered a task termination state, and the average position of that time period is calculated as the semantic target.

[0030] When the task type is a circling task, the turning angle of three adjacent points in the expert trajectory is calculated. If the turning angle is greater than a predetermined angle, the midpoint of these three points is taken as the key waypoint, thus obtaining the key waypoint sequence as the semantic target of the circling trajectory.

[0031] When the task type is a follower task, the target object position is obtained through the object detection algorithm, the relative offset between the expert's flight position and the target object position is calculated, and the time average is used as the semantic target.

[0032] According to a second aspect of the embodiments of this application, a drone control system is provided, including a ground station and a drone, wherein the drone is equipped with a first-person view camera and an environmental perception sensor.

[0033] The ground station is equipped with a visual language action model to acquire multimodal input data streams in real time, including natural language commands, real-time visual data, and real-time status data. The natural language commands represent the tasks to be performed by the UAV. The real-time visual data is collected by the UAV's first-person view camera. The real-time status data includes the UAV's pose, velocity, and angular velocity. The visual language action model is used to process the multimodal input data streams to obtain intermediate semantic targets.

[0034] The UAV includes an onboard computing unit and a low-level controller. The onboard computing unit is used to acquire real-time environmental perception data collected by the environmental perception sensors on the UAV, and, with the intermediate semantic target as input and the real-time environmental perception data as constraints, to perform safe motion planning for the UAV and generate a dynamically feasible and collision-free flight trajectory to reach the intermediate semantic target. The low-level controller is used to control the UAV to execute the flight trajectory.

[0035] According to a third aspect of the embodiments of this application, an electronic device is provided, comprising:

[0036] One or more processors;

[0037] Memory, used to store one or more programs;

[0038] When the one or more programs are executed by the one or more processors, the one or more processors perform the method as described in the first aspect.

[0039] According to a fourth aspect of the embodiments of this application, a computer-readable storage medium is provided that stores computer instructions thereon, which, when executed by a processor, implement the steps of the method as described in the first aspect.

[0040] The technical solutions provided by the embodiments of this application may include the following beneficial effects:

[0041] High security and verifiability: This invention separates the black-box decision-making of VLA from the white-box execution of the planner. Flight safety is mathematically guaranteed by a verifiable safe motion planning module, completely eliminating the security vulnerabilities of end-to-end methods.

[0042] High robustness and real-time performance: The VLA model does not require high-frequency operation. It only needs to output a new intermediate semantic target when the semantic decision needs to be updated. The obstacle avoidance task, which has extremely high real-time requirements, is independently completed by a lightweight, deployable onboard computer safety motion planning module. This perfectly solves the safety problem caused by VLA inference latency in existing technologies.

[0043] Reduced training burden: VLA models no longer need to learn complex flight dynamics or evasion techniques; they can focus solely on mapping language and vision to semantic targets. This significantly reduces the difficulty of model training and the reliance on massive amounts of expert flight data.

[0044] Interpretability: When a flight fails, the problem can be clearly diagnosed: whether the VLA model gave the wrong intermediate semantic target or the planner failed to find the path. Attached Figure Description

[0045] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0046] Figure 1 This is a flowchart illustrating a method for executing unmanned aerial vehicle (UAV) missions based on a visual language action model semantic target, according to an exemplary embodiment.

[0047] Figure 2 This is a schematic diagram illustrating a visual language action model according to an exemplary embodiment.

[0048] Figure 3 This is a schematic diagram illustrating a safe motion planning process according to an exemplary embodiment.

[0049] Figure 4 This is a schematic diagram of the architecture of an unmanned aerial vehicle system according to an exemplary embodiment.

[0050] Figure 5 This is a block diagram illustrating a drone mission execution device based on a visual language action model semantic target, according to an exemplary embodiment.

[0051] Figure 6 This is a schematic diagram of an electronic device according to an exemplary embodiment. Detailed Implementation

[0052] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application.

[0053] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0054] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0055] like Figure 1 As shown, this invention provides a method for executing UAV missions based on semantic targets of a visual language action model, comprising the following steps:

[0056] S1: Get the current time in real time The multimodal input data stream includes:

[0057] Real-time visual data Typically from the first-person view camera of the drone, such as RGB images or depth maps.

[0058] Natural Language Commands : Instructions given by the operator at the start of a mission or during flight, such as "fly to the roof of that red building".

[0059] Real-time status data The drone flight control system provides its own six degrees of freedom attitude, velocity, angular velocity, etc.

[0060] S2: The multimodal input data stream is processed using the Visual Language Action (VLA) model to obtain the intermediate semantic target;

[0061] Based on the safety requirements of UAV control systems, this application redefines the functional positioning of the VLA model, transforming it from an action generation module to a semantic target generation module. By modifying the action decoder to a target decoder, the model output meets the input interface requirements of the safety motion planning module, thereby establishing a decoupled structure between the learning decision layer and the verifiable control layer. This allows the subsequent safety planning module to independently generate flight trajectories that satisfy dynamic constraints. In specific implementation, such as... Figure 2As shown, the VLA model includes a visual encoder, a text encoder, a state encoder, a fusion module, and a target decoder. The visual encoder, text encoder, and state encoder are used to extract features from real-time visual data, natural language instructions, and real-time state data in the multimodal input data stream, respectively. The fusion module is used to deeply fuse the features extracted by the three encoders. The target decoder generates a high-level intermediate semantic target based on the fused features.

[0062] In one embodiment, the internal structure of the VLA model includes:

[0063] Visual encoder: In this embodiment, the first-person view camera simultaneously acquires RGB images and depth maps. Therefore, a pre-trained DinoV2 or SigLIP model is used to extract RGB visual representations rich in spatial and semantic information from real-time visual data. At the same time, a depth map encoder such as Depth Anything or Uni3D is used to extract features from the depth map information. The two-dimensional visual representations and depth information are spatially aligned and fused to construct visual features with three-dimensional geometric perception capabilities.

[0064] In practice, if the first-person view camera only acquires RGB images, the visual encoder will only extract RGB visual representations; if the first-person view camera only acquires depth maps, the visual encoder will only extract depth information.

[0065] Text encoder: Employs Llama Tokenizer to convert natural language instructions into language feature embeddings;

[0066] State encoder: Employs a multilayer perceptron to encode real-time state data into state features. The real-time state data includes flight dynamics parameters such as the UAV's pose and velocity.

[0067] Fusion Module: A multimodal large model such as Llama2 7B is used as the core fusion unit. This module deeply fuses visual features, linguistic features and state features through a cross-attention mechanism to generate a fused representation that understands the command intent, scene content and the current state of the drone.

[0068] Target Decoder: Unlike the action decoder in existing VLA models that directly outputs low-level actions, the decoder in this invention is modified into a target decoder. Its function is to parse and fuse representations and output high-level intermediate semantic targets. . It can be one or a series of three-dimensional spatial coordinate points, representing the next key waypoint or final destination pointed to by the language command; it can also be related to specific objects in the environment, such as VLA models. and The relative pose associated with the "red building" jointly identified; or it can be a semantic label representing the task state, such as "target identified" or "task completed". Specifically, when When the target is a 3D spatial coordinate point or relative pose, the target decoder first outputs the target's position or attention heatmap in a 2D pixel coordinate system based on the fused representation. Then, combining the aligned depth information (or RGB visual representation) and the camera intrinsic and extrinsic parameter matrices held by the UAV's onboard computing unit, it converts the 2D pixel coordinates into 3D physical coordinates or relative pose in the UAV's body coordinate system or global coordinate system through back projection mapping. If the intermediate semantic target is a semantic label representing the task state, then the multimodal large model needs to be used to determine, for example, whether the destination has been reached, based on the 3D physical coordinates or relative pose output by the target decoder.

[0069] The following further illustrates several specific forms of the intermediate semantic target to address natural language instructions of varying complexity.

[0070] Scenario 1: Single waypoint

[0071] When a command such as "fly to the red door" is received, the VLA model combines the "red door" identified in the visual data with the status data to interpret the command intent and output a single 3D coordinate point. As an intermediate semantic target, a safe trajectory to that point is then planned.

[0072] Scenario 2: Multi-step waypoint sequence

[0073] When given an instruction such as "fly around the building once", the VLA model is trained to adapt to the current state. Based on the visual understanding of the "building", a waypoint sequence is output in an autoregressive manner. Then, in sequence, As continuous target points, segmented planning and execution of safe trajectories are performed.

[0074] Scenario 3: Dynamic relative pose

[0075] When given an instruction such as "keep following the white car 5 meters behind," the VLA model continues to run, locating the position of the "white car" in each frame of visual data. Its output intermediate semantic target is a dynamically calculated relative pose, for example... ,in This represents the target offset vector "5 meters behind", which needs to be updated frequently in subsequent step S3. Real-time trajectory replanning is performed on dynamic target points to achieve language command tracking of dynamic targets.

[0076] Scenario 4: Semantic Tags

[0077] For non-navigational commands, such as "find the person in the red shirt," the output of the VLA model can be a semantic label. or At this point, the intermediate semantic target can be used to trigger other behavioral logic of the drone. In one embodiment, when the intermediate semantic target... When using semantic labels, subsequent step S3 does not use a single 3D point as the terminal target, but instead selects the corresponding planning mode based on the semantic labels and generates a reference trajectory or control objective function that matches the mode:

[0078] when When the planning mode is switched to hovering mode, the subsequent step S3 generates a hovering trajectory that gradually converges the speed and angular velocity of the drone to zero while maintaining a safe distance.

[0079] when At this point, the planning mode switches to search mode. Specifically, the onboard computing unit activates its built-in global autonomous exploration module, such as a frontier-based exploration algorithm or a regional coverage path planning algorithm. This exploration module dynamically calculates the boundary points or coverage waypoints of the unknown environment based on the currently frequently updated local environment map, and inputs them as temporary guide waypoints into subsequent step S3. During the implementation of subsequent step S3, the temporary guide waypoints are used as tracking targets, and local obstacle avoidance replanning is performed under the constraints of real-time environment representation. During the execution of this exploration trajectory, the VLA model continuously infers from the newly acquired real-time multimodal data until its output switches to search mode. This enables continuous and safe searching in unknown environments without relying on a single, clearly defined target point.

[0080] In practice, the training process of the VLA model is as follows: supervised fine-tuning is performed using a demonstration dataset containing visual data, language commands, and expert flight trajectories. The key is to avoid using the low-level actions recorded in the dataset. Instead of being used as a label, it is processed post-processed according to language instructions. Extract the semantic target corresponding to the instruction from the complete expert trajectory. .

[0081] In one embodiment, the post-processing includes the following steps:

[0082] Step 1: Task type identification: Classify the task type according to the language instruction L. The task type includes arrival tasks, surrounding tasks and following tasks.

[0083] Step 2: Trajectory Preprocessing: Processing the expert flight trajectory Preprocessing is performed, including outlier removal and trajectory smoothing, where smoothing can be achieved using moving average filtering or low-pass filtering.

[0084] Step 3: Semantic target extraction: When the task type is an arrival task, the speed at each moment in the expert trajectory is calculated. If there are N consecutive time steps at the end of the trajectory that satisfy... Then the average position of that time period As the final destination waypoint, that is, as the semantic target, where The preset speed threshold is set empirically based on the drone's flight dynamics characteristics and sensor noise levels.

[0085] When the task type is a loop-type task, key points are extracted from the expert trajectory. Specifically, the turning angles of three adjacent points in the trajectory are calculated. If satisfied Then the point These are the key waypoints. The final sequence of key waypoints is obtained. As a semantic target sequence surrounding the trajectory, The preset angle threshold is set empirically based on the drone's flight dynamics characteristics and sensor noise levels.

[0086] When the task type is a follow-up task, the location of the target object is first obtained through an object detection algorithm. Then calculate the relative offset between the expert's flight position and the target object's position. Subsequently, the time average of this relative offset over the entire following trajectory is calculated and used as the semantic target label for the relative pose. Where T is the total discrete time steps of the expert's flight trajectory segment.

[0087] When the task type is a search-type task, a pre-trained object detection algorithm is used to analyze the first-person real-time visual data during the expert demonstration process. Perform frame-by-frame detection. If the confidence level of a target object detected in the current frame and adjacent time windows is higher than a preset threshold, then extract the semantic label for that time period. For wandering search trajectory segments where no target was detected, their semantic labels are extracted as follows: .

[0088] The training objective is to minimize the intermediate semantic target output by the visual language action model. With tags The differences between them are specifically addressed using a multi-task joint loss function:

[0089]

[0090] Specifically, when the task type is a navigation task (arrival, circling, following) and the semantic target is spatial coordinates or relative pose, the regression loss is activated. The geometric distance between the predicted coordinates and the true coordinates is calculated using mean squared error or Huber Loss; when the task type is a search task and the semantic target is a state label, the classification loss is activated. The difference in probability distribution is calculated using cross-entropy loss. and These are the weighting coefficients.

[0091] S3: Acquire real-time environmental perception data collected by the environmental perception sensor on the UAV, and based on the intermediate semantic target, real-time status data, and real-time environmental perception data, perform safe motion planning for the UAV to generate a dynamically feasible and collision-free flight trajectory to reach the intermediate semantic target.

[0092] Specifically, the second environmental sensor can be a LiDAR, millimeter-wave radar, depth camera, etc., and the real-time environmental perception data it collects can be a real-time 3D point cloud or depth map.

[0093] In one embodiment, this step is a cyclical step, with a cyclical cycle. This refers to the trajectory planning cycle, in which trajectory planning is performed once per cycle. The triggering method includes at least one of time-triggered and event-triggered methods.

[0094] Time-triggered: at a fixed frequency The process is performed periodically, with the fixed frequency matching the refresh rate of the environmental perception sensor and the onboard computing power.

[0095] Event Trigger: A replanning event is immediately triggered when any of the following conditions are met: 1) Local environment map update (new point cloud / depth map arrives); 2) The drone's status deviates from the current trajectory by more than a threshold (e.g., position deviation exceeds...). Or the heading deviation exceeds ); 3) Predicted trajectory will violate safe distance constraints within the forward time window; 4) Intermediate semantic objectives Changes occur after updates or target feasibility modifications.

[0096] Furthermore, to decouple semantic decision-making from secure execution, a semantic target update cycle is set in one embodiment. With planning cycle They are independent of each other, among which In other words, the VLA model outputs semantic targets at a low frequency, while this step runs independently at a high frequency to complete real-time obstacle avoidance and dynamic feasible trajectory generation, with a planning cycle of The semantic update cycle is set to 10ms–100ms (10–100Hz). Take 0.2s–2s (0.5–5Hz).

[0097] At the beginning of each planning cycle, intermediate semantic targets, real-time environmental perception data, and real-time state data are acquired to generate a dynamically feasible and collision-free flight trajectory from the current position to the intermediate semantic targets. Specifically, as follows: Figure 3 As shown, the process may include the following:

[0098] First, the real-time environmental perception data is denoised and processed, and then projected onto a frequently updated local environmental map, such as an Euclidean Signed Distance Field (ESDF) map. This map can be used to... It has the time complexity to quickly find the distance from any point to the nearest obstacle.

[0099] Subsequently, the received intermediate semantic target Conduct feasibility studies and implement safety safeguards. If a query of the ESDF map reveals... If the target is located inside an obstacle or does not meet the set safe distance constraints, then target feasibility correction is performed: in the local environment map, the distance is found using gradient descent or local search algorithms. The nearest, collision-free free space point is used as the new corrected target point. If no feasible point can be found within the maximum search threshold, or if the target point is significantly beyond the current flight dynamics constraints of the UAV, the safe motion planning module will stop generating the local trajectory and send an "target unreachable" abnormal feedback back to the VLA model. This triggers the VLA model to re-infer and issue a new semantic target based on the latest visual and state data, thus forming a closed loop of decision-making and execution.

[0100] Finally, if the target point is feasible or the correction is successful, the corrected intermediate semantic target is then defined in the map. With the goal of using real-time status data Starting from this point, search for or optimize a mathematically provable, collision-free, smooth flight trajectory that satisfies the dynamic constraints of the UAV. .

[0101] In specific implementations, the trajectory planning algorithm includes optimization-based trajectory planning algorithms or sampling-based trajectory planning algorithms. Optimization-based trajectory planning algorithms include, but are not limited to: Model Predictive Control (MPC) trajectory optimization, trajectory generation and ESDF constraint optimization based on polynomials / minimum snaps, and CHOMP / TrajOpt type trajectory optimization. Sampling-based trajectory planning algorithms include, but are not limited to: dynamically constrained fast expanding random trees, BIT* / FMT, etc.

[0102] S4: Control the drone to execute the stated flight path;

[0103] The generated security trajectory The data is sent to the drone's underlying controller for precise tracking and execution. The system then returns to step S100, forming a complete closed-loop control.

[0104] During flight, the VLA model can continuously monitor flight progress and FPV images at a low frequency based on flight progress, and generate and issue the next intermediate semantic target when making a decision. Meanwhile, the onboard safety motion planning module operates independently at a high frequency, continuously responding to sudden dynamic obstacles to ensure that the UAV always flies safely in the general direction specified by the VLA model.

[0105] In one embodiment, the VLA model can be deployed on a ground station. Based on this, this application provides an unmanned aerial vehicle (UAV) system, including a ground station and an UAV, wherein the UAV is equipped with a first-person view camera and an environmental perception sensor.

[0106] The ground station is equipped with a visual language action model to acquire multimodal input data streams in real time, including natural language commands, real-time visual data, and real-time status data. The natural language commands represent the tasks to be performed by the UAV. The real-time visual data is collected by the UAV's first-person view camera. The real-time status data includes the UAV's pose, velocity, and angular velocity. The visual language action model is used to process the multimodal input data streams to obtain intermediate semantic targets.

[0107] The UAV includes an onboard computing unit and a low-level controller. The onboard computing unit is used to acquire real-time environmental perception data collected by the environmental perception sensors on the UAV, and, with the intermediate semantic target as input and the real-time environmental perception data as constraints, to perform safe motion planning for the UAV and generate a dynamically feasible and collision-free flight trajectory to reach the intermediate semantic target. The low-level controller is used to control the UAV to execute the flight trajectory.

[0108] Specifically, the ground station is typically a personal computer or server equipped with a high-performance computing unit. It runs computationally intensive, heavyweight VLA semantic target models, sends text commands to the UAV, and receives real-time visual and status data from the UAV. The UAV is equipped with a lightweight onboard computing unit, such as an NVIDIA Jetson, which is responsible for the high-frequency operation of the real-time-critical safety motion planning module and the underlying controller. In this embodiment, the natural language commands can be text directly entered by the operator through an input box, or text converted from the operator's speech input via a microphone. In this case, the ground station needs to be equipped with a microphone.

[0109] The two components interact via a low-latency wireless communication link. During operation, the UAV uploads real-time first-person visual images and its own state data to the ground station. Upon receiving natural language commands, the VLA model at the ground station infers from the uploaded data and sends the calculated intermediate semantic targets back to the UAV via the link. This "ground-air collaboration" architecture resolves the contradiction between the high computational demands of the VLA model and the limited computational power of the airborne platform.

[0110] In one exemplary embodiment, the drone uploads real-time first-person visual images and its own status data to the ground station via an uplink. Upon receiving the operator's natural language command "fly around the house once," the ground station's VLA model performs data fusion and inference, and sends the calculated intermediate semantics—containing the house's location and planned path information—to the drone via a downlink. The drone then combines the intermediate semantics with its own sensor data to automatically plan a safe execution path and execute the operator's command to fly around the house once.

[0111] In another embodiment, to adapt to scenarios with extremely high requirements for communication latency or where tasks need to be performed offline, the method is deployed entirely on the same high-performance recording and computing unit of the UAV.

[0112] To achieve efficient operation on airborne units, the VLA semantic target model needs to be lightweighted beforehand, for example, by using model distillation, weight quantization, or parameter pruning techniques to reduce model size and computational complexity.

[0113] Under this architecture, all data flows, including steps S100 to S500, are completed in a closed loop within the onboard computing unit, without the need for transmission via external wireless communication links. This eliminates the risks associated with communication latency and instability, allowing the closed-loop control frequency of the entire VLA semantic decision-making and safety planning process to reach a higher level, thereby improving the UAV's reaction speed and robustness in dynamic environments.

[0114] In summary, the architecture of this invention combines the advantages of two domains: the VLA model is responsible for the semantic decision of "where to go," utilizing powerful multimodal reasoning capabilities to resolve ambiguous language instructions and complex visual scenes into clear geometric targets. The safe motion planning module is responsible for the safe execution of "how to get there," utilizing real-time environmental perception and mature optimization algorithms to ensure absolute safety during flight. This decoupled hybrid architecture is an effective way to achieve robust, safe, and intelligent autonomous control of unmanned aerial vehicles (UAVs).

[0115] Corresponding to the aforementioned embodiments of the UAV mission execution method based on visual language action model semantic targets, this application also provides embodiments of the UAV mission execution device based on visual language action model semantic targets.

[0116] Figure 5 This is a block diagram illustrating a drone mission execution device based on a visual language action model semantic target, according to an exemplary embodiment. (Refer to...) Figure 5 The device may include:

[0117] The data acquisition module 21 is used to acquire the multimodal input data stream at the current moment in real time, including natural language commands, real-time visual data and real-time status data. The natural language commands represent the tasks to be performed by the UAV. The real-time visual data is collected by the UAV's first-person view camera. The real-time status data includes the UAV's pose, speed and angular velocity.

[0118] The semantic target generation module 22 is used to process the multimodal input data stream using a visual language action model to obtain intermediate semantic targets;

[0119] The safe motion planning module 23 is used to acquire real-time environmental perception data collected by the environmental perception sensor on the UAV, take the intermediate semantic target as input and the real-time environmental perception data as constraints, perform safe motion planning for the UAV, and generate a dynamically feasible and collision-free flight trajectory to reach the intermediate semantic target.

[0120] Flight control module 24 is used to control the UAV to execute the flight trajectory.

[0121] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0122] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0123] Accordingly, this application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the UAV mission execution method based on the semantic target of the visual language action model as described above.

[0124] Accordingly, this application also provides an electronic device, including: one or more processors; a memory for storing one or more programs; and when the one or more programs are executed by the one or more processors, causing the one or more processors to implement the UAV mission execution method based on the semantic target of the visual language action model as described above. Figure 6 The diagram shown is a hardware structure diagram of any data processing-capable device, which is a UAV mission execution device based on a visual language action model semantic target, according to an embodiment of the present invention. Except for... Figure 6 In addition to the processor, memory, and network interface shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0125] Accordingly, this application also provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the UAV mission execution method based on the semantic target of a visual language action model as described above. The computer-readable storage medium can be an internal storage unit of any data-processing device as described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units of any data-processing device and external storage devices. The computer-readable storage medium is used to store the computer program and other programs and data required by the data-processing device, and can also be used to temporarily store data that has been output or will be output.

[0126] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

Claims

1. A method for executing UAV missions based on semantic targets using a visual language action model, characterized in that, include: The system acquires multimodal input data streams in real time, including natural language commands, real-time visual data, and real-time status data. The natural language commands represent the tasks to be performed by the UAV, the real-time visual data is collected by the UAV's first-person view camera, and the real-time status data are the UAV's flight dynamics parameters. The multimodal input data stream is processed using a visual language action model to obtain intermediate semantic targets; The system acquires real-time environmental perception data collected by the environmental perception sensors on the UAV, uses the intermediate semantic target as input and the real-time environmental perception data as constraints, performs safe motion planning for the UAV, and generates a dynamically feasible and collision-free flight trajectory to reach the intermediate semantic target. Control the drone to execute the stated flight path.

2. The method according to claim 1, characterized in that, The intermediate semantic target includes at least one of the following: One or more three-dimensional space waypoints in the global coordinate system or the UAV body coordinate system; A relative pose associated with a specific object in the environment; A semantic label that represents the state of a task.

3. The method according to claim 1, characterized in that, Use an optimization-based trajectory planner or a sampling-based trajectory planner to plan safe motion for the drone.

4. The method according to claim 1, characterized in that, The environmental perception sensor is a depth camera or a lidar, and the real-time environmental perception data is a depth map or a 3D point cloud.

5. The method according to claim 1, characterized in that, The visual language action model includes a visual encoder, a text encoder, a state encoder, a fusion module, and a target decoder. The visual encoder, text encoder, and state encoder are used to extract features from real-time visual data, natural language instructions, and real-time state data in the multimodal input data stream, respectively. The fusion module is used to deeply fuse the features extracted by the three encoders. The target decoder generates a high-level intermediate semantic target based on the fused features.

6. The method according to claim 1, characterized in that, The visual language action model is trained through imitation learning, and its training methods include: Obtain a training dataset containing visual data, language commands, and expert flight trajectories; Semantic targets are extracted from the expert's flight trajectory and used as training labels; The training objective is set to minimize the difference between the intermediate semantic target output by the visual language action model and the training label; The visual language action model is trained until the training objective is met.

7. The method according to claim 6, characterized in that, Semantic targets are extracted from the expert's flight trajectory, including: Task types are classified according to language instructions, and the task types include at least arrival tasks, circling tasks, and following tasks; Preprocessing of expert flight trajectories includes removing outliers and smoothing the trajectory; When the task type is an arrival task, the speed of the expert trajectory at the end of several consecutive time steps is calculated. If it is consistently less than the predetermined speed threshold, it is considered a task termination state, and the average position of that time period is calculated as the semantic target. When the task type is a circling task, the turning angle of three adjacent points in the expert trajectory is calculated. If the turning angle is greater than a predetermined angle, the midpoint of these three points is taken as the key waypoint, thus obtaining the key waypoint sequence as the semantic target of the circling trajectory. When the task type is a follower task, the target object position is obtained through the object detection algorithm, the relative offset between the expert's flight position and the target object position is calculated, and the time average is used as the semantic target.

8. A drone control system, characterized in that, It includes a ground station and a drone, the drone being equipped with a first-person view camera and environmental perception sensors; The ground station is equipped with a visual language action model to acquire multimodal input data streams in real time, including natural language commands, real-time visual data, and real-time status data. The natural language commands represent the tasks to be performed by the UAV. The real-time visual data is collected by the UAV's first-person view camera. The real-time status data includes the UAV's pose, velocity, and angular velocity. The visual language action model is used to process the multimodal input data streams to obtain intermediate semantic targets. The UAV includes an onboard computing unit and a low-level controller. The onboard computing unit is used to acquire real-time environmental perception data collected by the environmental perception sensors on the UAV, and, with the intermediate semantic target as input and the real-time environmental perception data as constraints, to perform safe motion planning for the UAV and generate a dynamically feasible and collision-free flight trajectory to reach the intermediate semantic target. The low-level controller is used to control the UAV to execute the flight trajectory.

9. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-7.

10. A computer-readable storage medium storing computer instructions thereon, characterized in that, When executed by the processor, this instruction implements the steps of the method as described in any one of claims 1-7.

Images