Unmanned aerial vehicle flight task execution method and device based on visual language model, medium

Abstract

The application discloses a kind of unmanned aerial vehicle flight task execution methods, equipment, medium based on visual language model, the method includes: constructing simulation visual language data set for unmanned aerial vehicle flight task;Visual language model is trained based on simulation visual language data set;Including: visual language model is with natural language instruction, several frames historical RGB image as input, with natural language instruction corresponding unmanned aerial vehicle continuous action trajectory prediction value as output;According to the deviation between unmanned aerial vehicle continuous action trajectory prediction value, unmanned aerial vehicle continuous action trajectory true value, loss function is constructed to train visual language model;And, visual language model is with natural language instruction, initial time's RGB image, several frames historical RGB image as input, with task completion prediction label as output;According to the classification loss between task completion prediction label, task completion true label, visual language model is trained;Well-trained visual language model is applied to unmanned aerial vehicle flight task.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) navigation technology, and particularly relates to a method, device, and medium for executing UAV flight missions based on a visual language model. Background Technology

[0002] With the rapid development of drone technology, its applications have expanded from simple aerial photography for entertainment to critical fields such as environmental monitoring, emergency rescue, logistics delivery, and infrastructure inspection. In these increasingly complex and dynamic mission scenarios, the ability to simply execute pre-set waypoints is far from sufficient for drones. There is an urgent need for them to possess the intelligent capability to understand high-level human natural language commands and achieve a closed-loop "perception-understanding-decision" process in unknown environments. Achieving human-machine interactive autonomous navigation based on natural language, enabling drones to accurately understand human intentions and autonomously complete tasks in complex environments, is a crucial direction for current drone technology development.

[0003] However, in real-world applications, achieving high-precision and robust UAV-VLN (Visual Language Navigation) still faces numerous challenges. Existing target-oriented autonomous navigation methods are mainly divided into two categories: traditional modular methods based on a "perception-planning-control" separation architecture, and end-to-end navigation methods based on machine learning. Both of these methods have significant limitations when dealing with complex real-world scenarios.

[0004] The first category is the traditional modular navigation method. This type of method typically follows a strict sequential process: first, target detection algorithms (such as YOLO, Faster R-CNN, etc.) are used to identify the target object in the command and obtain its position information; then, a path planning module (such as A*, RRT*, etc.) generates the trajectory; finally, the flight control module performs the tracking. The main drawback of this architecture lies in its rigid "detect first, plan later" process and the lack of semantic understanding. On the one hand, the target detection module usually relies on training data of predefined categories, making it difficult to generalize to non-specific targets containing complex attribute descriptions or contextual relationships, resulting in a significant semantic gap; on the other hand, this modular design leads to the accumulation of errors at each level, and even a small deviation in the perception module can cause the complete failure of subsequent planning and control. Furthermore, this method has poor adaptability to dynamic environments; once the target is lost, the entire perception and planning process often needs to be restarted, lacking robustness in task execution.

[0005] The second category comprises end-to-end or near-end-to-end navigation methods based on large-scale visual language models (VLMs), which have emerged in recent years. These methods attempt to leverage the powerful cross-modal understanding and common-sense reasoning capabilities of VLMs to directly establish a mapping from images and language commands to flight control commands. Despite demonstrating significant potential, directly applying general-purpose VLMs to UAV navigation tasks still faces two core technological bottlenecks: First, there is a significant domain gap between the model training data and the actual application scenarios of UAVs, resulting in insufficient navigation and positioning accuracy. The performance of VLM (Virtual Modeling) is highly dependent on the quality of the training data. Although there are some relevant simulation platforms and datasets in academia (such as OpenUAV and AerialVLN), most of them are based on traditional triangular meshes for environment rendering. They cannot reproduce the real physical world in terms of lighting, texture, material reflection, and minute geometric details, making it difficult to meet the requirements of high-fidelity Sim-to-Real (Simulation to Reality) transfer. At the same time, general-purpose VLMs are mostly trained based on internet image and text pairs, lacking first-person perspective flight data. This data deficiency makes it difficult for existing models to accurately match language commands with visual observations at the pixel level when facing real UAV navigation scenarios, and they cannot output accurate target bounding boxes, which seriously restricts the final accuracy of navigation.

[0006] Secondly, existing methods lack dynamic evaluation and real-time replanning mechanisms for task completion, resulting in poor task continuity. Most existing VLM navigation schemes make decisions based on single-frame images or extremely short time windows, lacking a deep understanding of the task's temporal state. In actual flight, as the UAV moves, the target's position and size in the field of view change drastically, and may even become obstructed. Existing methods cannot accurately determine whether the task has met the command requirements (such as "approaching" or "arriving") based solely on current observations, often relying on unreliable heuristic rules or non-existent prior maps. Because they cannot accurately assess task progress, the system cannot trigger effective real-time replanning when faced with target loss or environmental changes, resulting in extremely low mission success rates for UAVs in complex dynamic environments.

[0007] Third, existing methods lack a unified architecture for multiple navigation tasks, limiting the overall generalization ability of the system. Current research often designs models and algorithms separately for specific tasks such as navigation obstacle avoidance, target tracking, precision landing, and environmental exploration. These tasks lack a shared representation space and decision-making mechanism, making it difficult to form a reusable capability system. In practical applications, UAVs need to dynamically switch or even execute multiple tasks in parallel during the same flight. However, existing methods typically rely on independent modules or single-task strategies, lacking cross-task collaboration and transfer capabilities. This fragmented design not only increases system complexity and maintenance costs but also makes it difficult for models to adapt to new scenarios, tasks, or ontology configurations, thus significantly limiting robustness and scalability in complex and open environments.

[0008] In summary, how to construct a high-fidelity simulation environment to eliminate the domain differences between Sim-to-Real and empower UAVs with the ability to make real-time judgments and replanning of mission status in unknown environments is a key issue that UAV visual language navigation technology urgently needs to address. Summary of the Invention

[0009] To address the shortcomings of existing technologies, embodiments of the present invention provide a method, device, and medium for executing unmanned aerial vehicle (UAV) flight missions based on a visual language model.

[0010] In a first aspect, embodiments of the present invention provide a method for executing unmanned aerial vehicle (UAV) flight missions based on a visual language model, the method comprising: Construct a simulated visual language dataset for UAV flight missions; Training a visual language model based on a simulated visual language dataset; including: The visual language model takes natural language commands and several frames of historical RGB images as input, and outputs the predicted continuous drone trajectory corresponding to the natural language commands. A loss function is constructed based on the deviation between the predicted and actual continuous drone trajectory values ​​to train the visual language model. The visual language model takes natural language instructions, the initial RGB image, and several historical RGB images as input, and the task completion prediction label as output; the visual language model is trained based on the classification loss between the task completion prediction label and the actual task completion label. The trained visual language model is applied to drone flight missions.

[0011] In a second aspect, embodiments of the present invention provide an electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to execute the above-described method for executing UAV flight missions based on a visual language model.

[0012] Thirdly, embodiments of the present invention provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described method for executing UAV flight missions based on a visual language model.

[0013] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a method for executing UAV flight missions based on a visual language model. By constructing a simulated visual language dataset for UAV navigation, the visual language model can not only generalize and understand various complex natural language commands, but also output continuous UAV trajectory with extremely high accuracy, thereby providing accurate waypoints for downstream trajectory planners. Simultaneously, this invention innovatively introduces a temporal comparison mechanism based on the "initial frame - current frame," achieving dynamic evaluation of flight mission progress by inputting and comparing the visual differences between the initial and current states in parallel. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart of a UAV flight mission execution method based on a visual language model provided in an embodiment of the present invention; Figure 2 A schematic diagram of a UAV flight mission execution method based on a visual language model provided in an embodiment of the present invention; Figure 3 A schematic diagram of a simulation scenario provided in an embodiment of the present invention; Figure 4 A schematic diagram of a simulation deployment experiment provided in an embodiment of the present invention; Figure 5 A schematic diagram of a physical deployment experiment provided for an embodiment of the present invention; Figure 6 This is a schematic diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0016] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] It should be noted that, unless otherwise specified, the features in the following embodiments and implementation methods can be combined with each other.

[0018] like Figure 1 and Figure 2 As shown, this embodiment of the invention provides a method for executing UAV flight missions based on a Visual Language Model (VLM), the method comprising the following steps: Step S1: Construct a simulation visual language dataset for UAV flight missions such as obstacle avoidance navigation, target tracking, precision landing, environmental exploration, and high-maneuver flight.

[0019] Furthermore, the simulated visual language dataset mainly revolves around UAV flight tasks such as obstacle avoidance navigation, target tracking, landing and resting, environmental search, and high-maneuver flight. It includes natural language commands generated in the UAV simulation scenario and their corresponding UAV trajectory multimodal samples. The UAV trajectory multimodal samples include RGB images, UAV poses, and corresponding UAV action trajectories. The UAV simulation scenario is composed of 3D Gaussian sputtering data and traditional triangular mesh data mixed in a preset ratio.

[0020] Specifically, such as Figure 3 As shown, this invention uses Unity as the core simulation platform. In this example, the drone simulation scene data consists of 3D Gaussian Splatting (3DGS) data and traditional triangular mesh data, supplemented by a suitable amount of real scene data, with the three components mixed in a ratio of 2:6:1. The 3DGS scene is generated through 3D reconstruction of self-acquired high-definition videos of real indoor environments, capable of photorealistically reproducing complex lighting, shadows, material reflections, and minute geometric details in the real world. This is crucial for ensuring the robustness of the model's visual feature learning and reducing the domain gap between simulation and reality. The mesh data is used to quickly construct diverse, large-scale structured environments.

[0021] Regarding the generation of natural language commands and UAV trajectory, this example employs manual annotation of the natural language command L for each atomic task and its corresponding trajectory starting point. Each atomic task includes various sophisticated flight modes such as navigation and obstacle avoidance, hovering observation, perching, in-situ corner scanning, target tracking, environmental exploration, and high-maneuver flight. Simultaneously, a collision-free, smooth UAV position and attitude trajectory is generated based on the trajectory starting point. .

[0022] Furthermore, a sample from the simulated visual language dataset It can be represented as: In the formula, L represents a natural language instruction. For the set of cameras, here it refers to the four directions {front, back, left, right}; This is a 640x480 pixel RGB image of the t-th sample; For the drone trajectory corresponding to the t-th sample, In this example, N=50. Let i be the 6D position and attitude of the UAV at time i, specifically: , Let i be the 3D position in the world coordinate system at time i. Let represent the roll angle, pitch angle, and yaw angle at time i, respectively.

[0023] Step S2, training a visual language model based on a simulated visual language dataset; including: The visual language model takes natural language commands and several historical RGB images (e.g., the previous 10 frames based on the current frame) as input, and outputs the predicted continuous drone trajectory corresponding to the natural language commands. A loss function is constructed based on the deviation between the predicted and actual continuous drone trajectory values ​​to train the visual language model; and... The visual language model takes natural language instructions, the initial RGB image, and several historical RGB images (e.g., the previous 10 frames based on the current frame as historical RGB images) as input and task completion prediction labels as output. The visual language model is trained based on the classification loss between the task completion prediction labels and the actual task completion labels.

[0024] Specifically, step S2, training the visual language model, includes the following steps: Phase 1: The visual language model uses natural language instructions L and several historical RGB images based on the current frame (e.g., the previous 10 frames based on the current frame as historical RGB images). As input, the predicted value of the continuous motion trajectory of the UAV corresponding to the natural language command. For output; the expression is as follows: A smoothed L1 loss is constructed to measure the deviation between the predicted value of the continuous drone trajectory corresponding to the natural language command and the actual value of the continuous drone trajectory, thereby training the visual language model. The expression is as follows: In the formula, This represents the loss function for the first stage of training the visual language model. For weight hyperparameters, For continuous motion trajectory regression loss, This is the predicted value of the continuous motion trajectory of the drone. This represents the true value of the continuous motion trajectory of the drone.

[0025] It should be noted that the first stage of visual language model training is navigation target localization and output formatting training, which aims to train the visual language model (VLM) to accurately identify and locate command targets from a single frame image.

[0026] Phase Two: The visual language model uses natural language instructions L and the initial RGB image. And several historical RGB images based on the current frame (e.g., the previous 10 frames based on the current frame as historical RGB images). Using historical RGB images as input and the initial RGB image as key and value, based on the cross-frame attention mechanism in visual language models, the algorithm calculates the similarity between the query and the key to determine the degree of change of the historical RGB image relative to the initial RGB image, thereby outputting the task completion prediction label. .

[0027] It should be noted that the second stage of visual language model training focuses on robustness and temporal understanding training for task completion judgment. Its core is to address the issue of drones failing to accurately determine task status when the viewpoint changes, thus endowing the visual language model with temporal understanding and progress perception capabilities. Furthermore, by forcing the visual language model to compare the visual differences between the initial frame and historical frames, and combining this with the intent in the natural language command L, the visual language model can learn a deep semantic understanding of task progress. Through its internal cross-frame attention mechanism, the visual language model can calculate the relative size and positional relationship between the target object in the current field of view and the initial target object, thereby determining states such as "approaching," "arriving," or "moving away," ultimately outputting a more reliable task completion judgment. Its training loss function takes the following form: In the formula For classifying losses, Indicates the task completion prediction label. This indicates the true label of task completion. This time-series comparison training paradigm greatly enhances the robustness of task completion judgment, laying a solid foundation for the subsequent closed-loop control and replanning mechanism of UAVs.

[0028] Step S3: Apply the trained visual language model to UAV flight missions such as obstacle avoidance navigation, target tracking, landing and resting, environmental search, and high-maneuver flight.

[0029] To enhance the drone's ability to prevent crashes due to obstacle encounters during flight, this example also incorporates flight safety strategies. These strategies utilize a pre-trained visual language model and a finite state machine (FSM) to execute drone flight missions. The specific steps are as follows: The system acquires the RGB image at the current moment in real time and converts it into a depth map. For example, in this instance, the MoGe-2 model is used to process the RGB image to obtain the depth map. The depth map is processed by a finite state machine (FSM) to determine whether there is a risk of collision in the predicted value of the continuous motion trajectory of the drone output by the visual language model. If there is a risk of collision, the predicted value of the continuous motion trajectory of the UAV output by the visual language model is corrected by using a finite state machine based on the obstacle data in the depth map.

[0030] In addition, the finite state machine includes an IDLE state, flight mission states including navigation, landing, tracking, exploration, or high-maneuver flight, replanning, and task completion states.

[0031] Furthermore, in the standby state, the finite state machine, in response to the natural language instruction L, acquires the RGB image at the initial moment. Switch to flight mission states including navigation, landing, tracking, exploration, or high-maneuver flight.

[0032] In the flight mission state, the trained visual language model responds to the natural language command L and the RGB image at the initial moment. And several historical RGB images based on the current frame. Output the predicted value of the continuous motion trajectory of the UAV corresponding to the natural language command. And task completion prediction labels; the visual language model is based on natural language instructions, the initial RGB image, and several frames of historical RGB images. The system determines the task completion prediction label. When the task completion prediction label is True, it switches to the task completion state, where the UAV performs various actions (including navigation, landing, tracking, exploration, or high-maneuver flight).

[0033] Furthermore, in the tracking state, when the visual language model cannot output the bounding box prediction value of the target corresponding to the natural language instruction (i.e., when the tracking target is lost), the finite state machine switches to the replanning state and executes a preset autonomous search strategy until the visual language model outputs the continuous motion trajectory prediction value corresponding to the natural language instruction and the bounding box prediction value of the tracking target, and the finite state machine switches to the tracking state; wherein, the autonomous search strategy includes performing a 360-degree slow rotation scan in place and / or retreating and raising the height to obtain a wider field of view.

[0034] It should be noted that by executing a pre-defined autonomous search strategy, the system ensures high continuity and a high final success rate even after the target is briefly occluded or lost. Specifically, when the target detection temporarily fails due to occlusion, changes in lighting, or viewing angle deviation, the system does not directly determine that the task has failed or stopped. Instead, it takes over through a replanning state and continuously and proactively acquires new observation information, thus eliminating task interruption caused by short-term perception interruptions. 360° in-situ rotation scanning can expand the field of view coverage without introducing additional displacement risks, increasing the probability of re-capturing the target; moving back and raising the height can increase the line of sight, reduce near-range occlusion, and expand the visible area, making it more likely that the target will re-enter the field of view, thereby shortening the "loss-reappearance" recovery time.

[0035] Example 1 Figure 4 Images showing the results of the simulation deployment experiment are displayed. Figure 4 It intuitively demonstrates the ability of visual language models to execute complex natural language commands, perform long-sequence navigation, and explore the environment in a simulation environment. Figure 4 The simulation deployment experiment was divided into three specific task scenarios: home, commercial (coffee shop / restaurant) and office. Each scenario compared and demonstrated the follow-up perspective (third person, showing the drone's position in the environment) and the airborne perspective (first person, the actual view seen by the drone).

[0036] Task 1 involved navigation within a home environment. The natural language command was "Go to the bookshelf first, then to the door, then across the sofa, and stop in the shower." The footage shows the drone taking off from the study area, identifying and flying towards the door, then smoothly traversing the sofa area in the living room, and finally landing precisely in the small shower room. This demonstrates that the visual language model can successfully handle continuous long sequences of navigation and obstacle avoidance tasks.

[0037] Task 2 involves spatial logic and exploration in a commercial environment. The natural language instruction is "Go to the right side of the chair, check the plants, and finally find the coffee machine on the counter." The drone first locates the chair and adjusts its position to its "right side," then flies to the planting area to check the plants, and finally navigates to the bar area and identifies the coffee machine. This demonstrates that the visual language model can understand navigation tasks with spatial logical relationships (such as "the right side of the chair") and has the ability to "explore the environment" (such as finding a specific object like the coffee machine).

[0038] Task 3 involved multi-target localization in an office environment. The natural language command was "Go to the bookshelf, then find the computer, and finally go to the desk and chair in that order." The drone navigated through the complex office facilities, successfully locating and approaching the bookshelf, computer, desk, and chair. This demonstrates that the visual language model can reliably follow and accurately execute complex natural language commands in scenarios with many obstacles (dense desks and chairs).

[0039] Example 2 Figure 5 This is a schematic diagram of a physical deployment experiment provided in an embodiment of the present invention. It is used to evaluate the performance of the visual language model in a real physical environment, verify the ability of the visual language model to perform complex tasks in the real world, including obstacle avoidance navigation, target tracking, precise landing, and environmental search, and verify that the visual language model can correctly decompose long sequences of natural language instructions with logical relationships, and has strong versatility and generalization.

[0040] The natural language command for Task 1 was "Search for the electric fan, then find the tree with the dark green bag, and finally land on the yellow mat." The footage shows the drone navigating the indoor area, accurately locating and approaching the electric fan and the specific tree, before smoothly landing in the designated yellow target area. This validates the visual language model's ability to search the environment and accurately land on its habitat.

[0041] The natural language instructions for Task 2 were "Go to the black chair on the right, then search the square item box, and finally go to the vicinity of the two white cylinders." The footage shows that the drone demonstrated an understanding of spatial relative positions (such as "on the right") and was able to identify the specific square item box and white cylinders from a variety of objects.

[0042] The natural language instruction in Task 3 was "Find the smaller yellow cabinet, then pass through the square frame and stop on the right side of the black artificial hill." The footage shows that the drone not only compared the sizes of the two cabinets to find the "smaller one," but also precisely passed through the narrow square frame (obstacle avoidance), finally stopping on a specific side of the artificial hill. This verifies the visual language model's logical ability to decompose sequential subtasks and process multi-step complex instructions.

[0043] The natural language command in Task 4 was "track the person wearing gray clothes." The footage showed the drone maintaining a certain altitude and distance, moving in sync with the person in gray clothes walking ahead. This verified the target tracking capability of the visual language model in a real-world scenario.

[0044] In summary, this invention provides a method for UAV flight mission execution based on a visual language model. By constructing a simulated visual language dataset for UAV navigation, the visual language model can not only generalize and understand various complex natural language commands, but also output continuous UAV trajectory with extremely high accuracy, thereby providing accurate waypoints for downstream trajectory planners. Furthermore, this invention innovatively introduces a temporal comparison mechanism based on the "initial frame - current frame," achieving dynamic evaluation of flight mission progress by inputting and comparing the visual differences between the initial and current states in parallel. Moreover, this invention improves the mission continuity of UAVs in dynamic and occluded environments.

[0045] According to embodiments of the present invention, the present invention also provides an electronic device and a readable storage medium.

[0046] Figure 6 A schematic block diagram of an electronic device that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0047] The electronic device includes a computing unit 101, which can perform various appropriate actions and processes according to a computer program stored in ROM 102 or a computer program loaded into RAM 103 from storage unit 108. RAM 103 may also store various programs and data required for the operation of the electronic device. The computing unit 101, ROM 102, and RAM 103 are interconnected via bus 104. I / O interface 105 is also connected to bus 104.

[0048] Multiple components in the electronic device are connected to the I / O interface 105, including: an input unit 106, such as a keyboard, mouse, etc.; an output unit 107, such as various types of displays, speakers, etc.; a storage unit 108, such as a disk, optical disk, etc.; and a communication unit 109, such as a network card, modem, wireless transceiver, etc. The communication unit 109 allows the electronic device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0049] The computing unit 101 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 101 performs the various methods and processes described above. For example, in some embodiments, the methods in the multidimensional early warning system for pressure injuries can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 108. In some embodiments, part or all of the computer program can be loaded and / or installed on an electronic device via ROM 102 and / or communication unit 109. When the computer program is loaded into RAM 103 and executed by the computing unit 101, one or more steps of the methods in the multidimensional early warning system for pressure injuries described above can be performed. Alternatively, in other embodiments, the computing unit 101 can be configured to perform the methods in the multidimensional early warning system for pressure injuries by any other suitable means (e.g., by means of firmware).

[0050] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0051] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0052] In the context of this invention, a readable storage medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A readable storage medium can be a machine-readable signal medium or a machine-readable storage medium. A readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0053] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including voice input, speech input, or tactile input).

[0054] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0055] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0056] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only.

[0057] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A method for executing unmanned aerial vehicle (UAV) flight missions based on a visual language model, characterized in that, The method includes: Construct a simulated visual language dataset for UAV flight missions; Training a visual language model based on a simulated visual language dataset; including: The visual language model takes natural language commands and several frames of historical RGB images as input, and outputs the predicted continuous drone trajectory corresponding to the natural language commands. A loss function is constructed based on the deviation between the predicted and actual continuous drone trajectory values ​​to train the visual language model. The visual language model takes natural language instructions, the initial RGB image, and several historical RGB images as input, and the task completion prediction label as output; the visual language model is trained based on the classification loss between the task completion prediction label and the actual task completion label. The trained visual language model is applied to drone flight missions.

2. The UAV flight mission execution method based on a visual language model according to claim 1, characterized in that, The simulated visual language dataset includes natural language commands and their corresponding drone trajectory multimodal samples generated in drone simulation scenarios and / or real drone scenarios; wherein, the drone trajectory multimodal samples include RGB images and drone motion trajectories; the drone motion trajectory includes the drone's position and attitude at each moment, and the attitude includes roll angle, pitch angle, and yaw angle attitude; And / or, The drone simulation scene is composed of 3D Gaussian sputtering data and traditional triangular mesh data mixed in a preset ratio.

3. The UAV flight mission execution method based on a visual language model according to claim 1, characterized in that, The process of constructing a loss function and training a visual language model based on the deviation between the predicted values ​​of continuous drone trajectory and the actual values ​​of continuous drone trajectory includes: A smooth L1 loss is constructed to measure the deviation between the predicted value of the continuous action trajectory of the UAV corresponding to the natural language command and the actual value of the continuous action trajectory of the UAV, so as to train the visual language model.

4. The UAV flight mission execution method based on a visual language model according to claim 1, characterized in that, The visual language model takes natural language instructions, an initial RGB image, and several frames of historical RGB images as input, and outputs a predicted label upon task completion. The process includes: The visual language model takes natural language instructions, the initial RGB image, and several historical RGB images based on the current frame as input. Based on the cross-frame attention mechanism in the visual language model, it uses several historical RGB images as queries and the initial RGB image as keys and values. By calculating the similarity between the query and the key, it determines the degree of change of the historical RGB images relative to the initial RGB image, and thus outputs the task completion prediction label.

5. The UAV flight mission execution method based on a visual language model according to claim 1, characterized in that, The process of applying a trained visual language model to drone flight missions includes: The drone flight mission is executed based on a trained visual language model and a finite state machine; specifically, the following steps are performed: The system acquires the RGB image at the current moment in real time and converts it into a depth map. By processing the depth map using a finite state machine, it is possible to determine whether the predicted values ​​of the continuous motion trajectory of the drone output by the visual language model pose a collision risk. If there is a risk of collision, the predicted value of the continuous motion trajectory of the UAV output by the visual language model is corrected by using a finite state machine based on the obstacle data in the depth map.

6. The UAV flight mission execution method based on a visual language model according to claim 5, characterized in that, The finite state machine includes a standby state, flight mission states including navigation state, landing state, tracking state, exploration state or high-maneuver flight state, replanning state and mission completion state.

7. The UAV flight mission execution method based on a visual language model according to claim 6, characterized in that, In the standby state, the finite state machine responds to natural language commands, acquires the RGB image at the initial moment, and switches to flight mission states including navigation state, landing state, tracking state, exploration state, or high-maneuver flight state. In the flight mission state, the trained visual language model responds to natural language commands, the RGB image at the initial moment, and several historical RGB images based on the current frame, and outputs the predicted value of the continuous action trajectory of the UAV corresponding to the natural language command and the predicted label of mission completion. When the task completion prediction label is true, switch to the task completion state.

8. The UAV flight mission execution method based on a visual language model according to claim 7, characterized in that, In the tracking state, when the trained visual language model is unable to output the bounding box prediction value of the tracking target, i.e. when the tracking target is lost, the finite state machine switches to the replanning state and executes the preset autonomous search strategy until the visual language model outputs the bounding box prediction value of the tracking target corresponding to the natural language command, and the finite state machine switches to the tracking state. The autonomous search strategy includes performing a 360-degree slow rotation in place and / or moving backward and increasing the height.

9. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the UAV flight mission execution method based on the visual language model as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the UAV flight mission execution method based on the visual language model as described in any one of claims 1-8.

Images