Unmanned aerial vehicle automatic driving system and method based on cloud collaborative space-time AI

Abstract

The invention relates to the field of unmanned aerial vehicle automatic driving, and particularly discloses an unmanned aerial vehicle automatic driving system and method based on cloud collaborative space-time AI. The system comprises an unmanned aerial vehicle airborne device arranged at an edge end, and a first instruction generation device, a data management device and a model training device which are arranged at a cloud end. The unmanned aerial vehicle airborne device comprises a data acquisition module for acquiring flight data and airspace spatio-temporal data of an unmanned aerial vehicle, and generating spatio-temporal coded data based on a three-dimensional grid; the control instruction generation module processes the space-time coded data, a first instruction and the like by using the edge end space-time model to generate a control instruction; and the flight control module adjusts the flight state according to the control instruction. And the first instruction generation device processes the space-time coded data by using the cloud space-time model to generate a first instruction. A data management device generates a training data set. The model training device trains and generates a cloud spatio-temporal model. According to the scheme, flight is safer and more flexible.

Description

Technical Field

[0001] The present application relates to the field of drone autonomous driving technology, and in particular to a drone autonomous driving system and method based on cloud-based collaborative spatiotemporal AI. Background Art

[0002] Drones are the core terminal carriers of low-altitude economic activities. With the rapid and large-scale development of low-altitude economic activities such as logistics distribution, agricultural plant protection, emergency rescue, safety inspection and urban air traffic, how to safely fly autonomously in the increasingly complex low-altitude airspace environment and high-density integrated flight scenarios is the biggest technical challenge facing drones, and it is also a challenge faced by the low-altitude economy.

[0003] The flight control of drones beyond visual range can only rely on autonomous driving algorithms and models. However, the existing drone autonomous driving technology mainly focuses on the single-machine autonomous control mode, which is unable to obtain and process a large amount of data from multiple drones and the external environment in real time, and the computing power is insufficient, so it can only solve the autonomous driving needs in some known and certain flight scenarios. In complex low-altitude airspace environments and high-density fusion flight scenarios, there are still significant deficiencies in the basic requirements of drone autonomous driving, such as intelligent flight control, efficient real-time computing analysis, and system safety, and it is impossible to achieve completely autonomous and safe flight.

[0004] Therefore, a new drone autopilot system is needed to solve the above problems. Summary of the invention

[0005] The present application provides a drone autopilot system and method based on cloud-based collaborative spatiotemporal AI, in an effort to solve or at least alleviate at least one of the above problems.

[0006] According to one aspect of the present application, a drone autopilot system based on cloud-based collaborative spatiotemporal AI is provided, comprising: one or more drone-mounted devices, wherein each drone-mounted device comprises: a data acquisition module, adapted to acquire flight data and airspace spatiotemporal data of the drone during flight, and based on a three-dimensional grid, process the flight data and airspace spatiotemporal data to generate spatiotemporal coded data; a control instruction generation module, adapted to process the spatiotemporal coded data, flight mission data, low-altitude airspace environment data and a first instruction using an edge-end spatiotemporal model to generate a control instruction; a flight control module, adapted to adjust the flight state of the drone according to the control instruction; a first instruction generation device, coupled to the data acquisition module and the control instruction generation module respectively, adapted to acquire the spatiotemporal coded data, and process the spatiotemporal coded data using a cloud-based spatiotemporal model to generate the first instruction; a data management device, coupled to the data acquisition module and the control instruction generation module respectively, adapted to acquire the spatiotemporal coded data, and generate a training data set in combination with historical data; a model training device, coupled to the data management device, adapted to train and generate the cloud-based spatiotemporal model using the training data set.

[0007] Optionally, in the system according to the present application, the data management device is also suitable for fine-tuning the cloud-side spatiotemporal model to obtain the edge-side spatiotemporal model, and sending it to the control instruction generation module; optimizing the cloud-side spatiotemporal model using the operating results and execution feedback of the edge-side spatiotemporal model, wherein the cloud-side spatiotemporal model and the edge-side spatiotemporal model are based on the Transformer architecture.

[0008] Optionally, in the system according to the present application, in the unmanned aerial vehicle onboard device, the data acquisition module also includes a data processing unit, and the data processing unit is suitable for preprocessing the flight data separately, and the preprocessing includes: spatiotemporal alignment, anomaly detection, and normalization; according to the geographic location information and time information of each flight data, mapping it to a corresponding three-dimensional grid, and encoding each three-dimensional grid to obtain spatiotemporal encoded data.

[0009] Optionally, in the system according to the present application, in the drone onboard device, the data acquisition module includes: positioning equipment, video acquisition equipment, inertial measurement unit, lidar, attitude reference system, infrared sensor; the flight data includes: flight position data, driving data, flight airspace data, video data.

[0010] Optionally, in the system according to the present application, the data management device is suitable for obtaining the spatiotemporal coding data of the corresponding unmanned aerial vehicle airborne device via the data acquisition module; obtaining historical data, including: historical environmental information, historical flight mission records, historical flight status data, communication and collaboration records, system operation logs, wherein the historical environmental information includes: geographic information, weather, dynamic obstacles, environmental risk records, the historical flight mission records include: historical flight mission planning, flight mission type, flight mission resource consumption, the historical flight status data includes: historical flight trajectory, flight attitude records, power system data, battery performance data; combining the historical data and the spatiotemporal coding data to generate a training data set.

[0011] Optionally, in the system according to the present application, the model training device is suitable for labeling the training data set according to task categories to obtain labeled data, wherein the task categories include perception tasks, prediction tasks, planning and decision-making tasks; constructing a pre-trained model, which is based on a large language model and includes a feature extraction component, a perception task component, a prediction task component, and a planning and decision-making component; based on the labeled data, the pre-trained model is trained to obtain a cloud-based spatiotemporal model.

[0012] Optionally, in the system according to the present application, for the perception task, the labeled data include: target position and type in the image, dynamic objects and static objects in the point cloud, and abnormal data in the inertial measurement data; for the prediction task, the labeled data include: dynamic target trajectory, future state of the time series; for the planning and decision-making task, the labeled data include: obstacle avoidance behavior, key nodes of path planning, wherein the obstacle avoidance behavior includes turning, acceleration, and deceleration.

[0013] Optionally, in the system according to the present application, the flight control module is also suitable for controlling the acquisition parameter configuration of the data acquisition module.

[0014] According to another aspect of the present application, a method for autonomous driving of a drone based on cloud-based collaborative spatiotemporal AI is provided, wherein the method is suitable for execution in a drone onboard device, wherein the drone onboard device is connected to a server, and wherein the method comprises: collecting flight data during the flight of the drone, wherein the flight data comprises: flight position data, driving data, and flight airspace data; processing the flight data based on a three-dimensional grid to generate spatiotemporal coded data; utilizing an edge-end spatiotemporal model to process the spatiotemporal coded data, flight mission data, low-altitude airspace environment data, and a first instruction to generate a control instruction; and adjusting the flight state of the drone according to the control instruction, wherein the first instruction is determined by the server at least based on the spatiotemporal coded data.

[0015] According to another aspect of the present application, a method for autonomous driving of a drone based on cloud-based collaborative spatiotemporal AI is provided, wherein the method is suitable for execution in a server, wherein the server is connected to one or more drone onboard devices, and the method comprises: utilizing a cloud-based spatiotemporal model to process spatiotemporal coded data from the drone onboard device to generate a first instruction, so that the drone onboard device can determine a control instruction for adjusting the flight state of the drone in combination with the first instruction; optimizing the cloud-based spatiotemporal model according to the operating results and execution feedback of the edge-end spatiotemporal model in the drone onboard device; wherein the cloud-based spatiotemporal model comprises a feature extraction component, a perception task component, a prediction task component, and a planning and decision-making component, and the cloud-based spatiotemporal model is trained by a training data set generated from the spatiotemporal coded data and historical data.

[0016] According to another aspect of the present application, a computing device is provided, comprising: one or more processor memories; one or more programs, wherein the one or more programs are stored in the memories and configured to be executed by one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0017] According to another aspect of the present application, a computer-readable storage medium storing one or more programs is provided. The one or more programs include instructions. When the instructions are executed by a computing device, the computing device executes any of the methods described above.

[0018] According to another aspect of the present application, a computer program product is provided, comprising a computer program / instruction, wherein the computer program / instruction implements the steps of the above-mentioned method when executed by a processor.

[0019] In summary, according to the solution of this application, a complete spatiotemporal AI large model is established based on the Transformer architecture, and the powerful model computing power of the server side and the real-time computing advantages of the airborne edge side are combined to realize the large model deployment of cloud-based collaborative spatiotemporal AI. Based on this, a drone automatic driving system is constructed. During the drone automatic driving process, flight data and airspace spatiotemporal data can be collected through the data acquisition module to obtain spatiotemporal coding data, and the edge-side spatiotemporal model can be used to perform real-time computational reasoning on the spatiotemporal coding data; at the same time, the cloud (server) spatiotemporal AI model performs computational reasoning on the real-time spatiotemporal coding data, and synchronously generates a first instruction for the edge-side spatiotemporal model to perform computational reasoning, and finally generates a control instruction to control the flight status of the drone in real time. This solution can enhance the drone's global perception capability and improve flight safety.

[0020] In addition, this solution can effectively reduce the edge data processing requirements of drones, reduce the energy consumption of drones, and thus improve the flight endurance of drones. At the same time, it can reduce the high-frequency transmission of massive and large-capacity indiscriminate information of drones and improve the efficiency of information processing.

[0021] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to achieve the above and related purposes, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings, which indicate various ways in which the principles applied herein can be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. By reading the following detailed description in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. Throughout the application, the same reference numerals generally refer to the same parts or elements.

[0023] Figure 1 A schematic diagram of a drone autopilot system 100 based on cloud-based collaborative spatiotemporal AI according to some embodiments of the present application is shown; Figure 2 A schematic diagram of the structure of a spatiotemporal model 200 according to some embodiments of the present application is shown; Figure 3 A schematic diagram of a computing device 300 according to some embodiments of the present application is shown; Figure 4 A schematic diagram of a method 400 for autonomous driving of a drone based on cloud-based collaborative spatiotemporal AI according to some embodiments of the present application is shown; Figure 5 A schematic diagram of a drone autonomous driving method 500 based on cloud-based collaborative spatiotemporal AI according to some embodiments of the present application is shown. DETAILED DESCRIPTION

[0024] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0025] This application aims to provide a drone autopilot system based on cloud-based collaborative spatiotemporal AI, combining the powerful model computing capabilities of cloud servers and the real-time computing advantages of drone onboard devices (onboard edge terminals), bidirectional verification, and improved system security. At the same time, based on AI technology, a large language model is used to build a deep learning model with spatiotemporal attributes (referred to as a spatiotemporal model, including a cloud-based spatiotemporal model and an edge-side spatiotemporal model), and the spatiotemporal coding data is processed to determine the control instructions indicating the flight status of the drone. During the drone autopilot process, real-time calculation and reasoning can be performed, and the drone flight control system can be controlled in real time, making drone flights safer and more flexible.

[0026] Figure 1 A schematic diagram of a drone autopilot system 100 based on cloud-based collaborative spatiotemporal AI according to some embodiments of the present application is shown.

[0027] refer to Figure 1 The system 100 is composed of one or more drone onboard devices 110 and a server 120. Figure 1 Two drone-mounted devices 110 are shown, and it should be understood that the embodiments of the present application do not limit the number of drone-mounted devices 110 and servers 120.

[0028] The drone onboard device 110 is arranged on the drone, and depending on the desired configuration, the drone onboard device 110 further includes: a data acquisition module 112, a control instruction generation module 114, and a flight control module 116 coupled to each other. The components in the drone onboard device 110 are described below.

[0029] The data acquisition module 112 collects flight data and airspace spatiotemporal data of the UAV during flight, and processes the flight data and airspace spatiotemporal data based on a three-dimensional grid to generate spatiotemporal coded data.

[0030] In some embodiments, the data acquisition module 112 includes: positioning equipment (such as GPS / Beidou / Galileo positioning equipment), video acquisition equipment, IMU inertial measurement unit, LiDAR laser radar, heading reference system, infrared sensor, etc. The flight data and airspace spatiotemporal data collected by these data acquisition modules include: flight position data (including longitude and latitude, altitude), driving data (including speed, acceleration, heading angle, pitch angle, etc.), flight airspace data (geographic real scene data, building data, obstacle data, etc.), and video data. Flight data is the relevant data during the flight of the drone. Usually, the flight data has geographic location information and time information, wherein the geographic location information indicates the geographic location associated with the flight data, and the time information is usually the collection time of the flight data.

[0031] In this embodiment, the data acquisition module 112 further includes a data processing unit (not shown), which is used to process the above-mentioned collected data to generate spatiotemporal coded data.

[0032] First, the flight data and airspace spatiotemporal data are preprocessed separately, including spatiotemporal alignment, anomaly detection, and normalization. The following are some specific processing methods for preprocessing.

[0033] Spatiotemporal alignment (i.e., spatiotemporal synchronization and calibration of flight data): align data with different sampling frequencies to the same time axis; use coordinate transformation to uniformly convert the geographic location information of the data to the Beidou grid coordinate system to ensure spatial consistency.

[0034] Anomaly detection (i.e., detection and correction of abnormal data): For flight data at different times, continuous time data is generated through time interpolation (such as linear interpolation or spline interpolation) to ensure that each time point has corresponding correct flight data. Anomaly detection also includes: anomaly detection of a series of data collected by each data acquisition module (for example, inertial measurement data), such as identifying outliers through statistical model-based detection (such as outlier detection algorithm, Z-score, DBSCAN clustering) methods, and correcting outliers based on historical data or adjacent data points.

[0035] Normalization: For each type of data, convert it into a distribution with a mean of 0 and a variance of 1 based on the spatial dimension or time series; or convert the data into values ​​in a specified range (such as [0, 1]) so that it is not affected by unit differences.

[0036] Afterwards, according to the geographic location information and time information of each flight data and airspace spatiotemporal data, it is mapped to the corresponding three-dimensional grid, and each three-dimensional grid is encoded to obtain spatiotemporal encoded data.

[0037] According to the implementation mode of the present application, the low-altitude airspace where drones are active is divided into multiple three-dimensional grids according to the three-dimensional grid division rules and coding specifications, and a unique grid code is generated for each three-dimensional grid. Low-altitude airspace generally refers to the flight area below 1,000 meters. Three-dimensional grid coding methods such as geospatial grid codes and Beidou grid codes can be used to divide the three-dimensional grid and generate grid codes. Then, according to the latitude and longitude altitude (i.e., geographic location information) and time information of the flight data, it is allocated to the corresponding three-dimensional grid so that each flight data is associated with the corresponding three-dimensional grid. In other words, each three-dimensional grid is equivalent to a "container" that can box multiple data points. The boxing operation is quickly located according to the grid code, combined with efficient spatial index structures such as hash tables or KD-trees. When all flight data and airspace spatiotemporal data are mapped to their respective three-dimensional grids, the mapped three-dimensional grids are encoded to obtain spatiotemporal coded data.

[0038] In some embodiments, the three-dimensional grid has different precisions. When mapping the flight data to the three-dimensional grid, the flight data can be mapped to the three-dimensional grids of different precisions according to the precision and source of the flight data. For example, the flight data from the high-resolution sensor (data acquisition module) is mapped to the three-dimensional grid with a finer precision range, and the flight data from the low-resolution sensor (data acquisition module) is mapped to the three-dimensional grid with a coarser precision range.

[0039] In addition, after mapping the above data into three-dimensional grids, you can also use weighted interpolation or spatial interpolation algorithms (such as Kriging interpolation and bilinear interpolation) to fuse data between three-dimensional grids to ensure data continuity.

[0040] The control instruction generation module 114 uses the edge-end spatiotemporal model to process the spatiotemporal coding data, the flight mission data, the low-altitude airspace environment data and the first instruction to generate a control instruction.

[0041] In this embodiment, the flight mission data includes, for example, target point information, airspace information, flight time, energy consumption, etc.; the low-altitude airspace environment data includes, for example, the current flight weather information, etc. Usually, when the UAV performs a flight mission, it can obtain the corresponding flight mission data and low-altitude airspace environment data. The first instruction is obtained by the server 120 based on the spatiotemporal coding data obtained in real time, using the cloud-based spatiotemporal model calculation and analysis, and is synchronized to the control instruction generation module 114.

[0042] The edge-end spatiotemporal model is based on a large language model. More specifically, the edge-end spatiotemporal model is based on the Transformer architecture. According to the implementation mode of the present application, the edge-end spatiotemporal model and the cloud-end spatiotemporal model are in a hierarchical collaborative relationship with clear division of tasks. The two together constitute an efficient, multi-level AI system, combining the powerful computing power of the cloud server and the real-time performance of the edge-end drone airborne device to achieve the purpose of overall performance optimization and intelligence enhancement. Specifically, the cloud-end spatiotemporal model is mainly responsible for global, high-computation tasks, which are characterized by global perception, complex calculations, and long-term planning. The edge-end spatiotemporal model is responsible for local, real-time tasks, which are characterized by real-time response, local optimization, and data reduction.

[0043] The cloud-side spatiotemporal model is trained and generated on the server 120, and then the parameters of the cloud-side spatiotemporal model are fine-tuned to obtain an edge-side spatiotemporal model that is more suitable for edge-side deployment. In other words, the basic structure of the edge-side spatiotemporal model and the cloud-side spatiotemporal model is the same, but there are differences in parameter settings.

[0044] Figure 2 A schematic diagram of the structure of a spatiotemporal model 200 according to some embodiments of the present application is shown. As mentioned above, the structure of the spatiotemporal model can be applied to an edge spatiotemporal model or a cloud spatiotemporal model. To simplify the description, it is collectively referred to as a spatiotemporal model here.

[0045] According to the task category of the spatiotemporal model, the spatiotemporal model 200 includes a feature extraction component 210, a perception task component 220, a prediction task component 230, and a planning and decision component 240, such as Figure 2 As shown. Among them, the feature extraction component 210 is used as the bottom module, and the Transformer architecture is used to extract features from the input data, wherein information weighting is performed based on the attention mechanism of the Transformer. The perception task component 220 (including the visual perception subnetwork, the sensor perception fusion network, the real scene environment perception subnetwork, etc.), the prediction task component 230, and the planning and decision component 240 (including the reinforcement learning subnetwork, the generation network, etc.) are used as high-level task-related modules, wherein the perception task component 220 and the prediction task component 230 are coupled to the feature extraction component 210, and the extracted features are further processed and calculated and analyzed to identify the target object, anomaly detection, and predict the flight trajectory. The planning and decision component 240 is coupled to the perception task component 220 and the prediction task component 230, and dynamic decisions and action optimization adjustments are made according to the perception results and prediction results, and finally an instruction (the instruction can be a control instruction or a first instruction) is output, such as route optimization results, obstacle avoidance actions, flight attitude adjustment operations, etc.

[0046] In this embodiment, the cloud spatiotemporal model processes the real-time spatiotemporal coded data from one or more drone airborne devices 110, generates a first instruction, and sends it to the control instruction generation module 114 of the corresponding drone airborne device 110. The first instruction is a global instruction representing a global plan.

[0047] The edge-end spatiotemporal model processes the local information (including: spatiotemporal coding data, flight mission data, low-altitude airspace environment data) and the first instruction to generate a control instruction. In other words, the control instruction generation module 114 combines the local information and the global instruction (first instruction) from the server 120 to generate a control instruction indicating the flight status of the drone.

[0048] In some other embodiments, when the network connection between the drone airborne device 110 and the server 120 is interrupted or the command is delayed, the control command generation module 114 can also generate a control command based on local information (for example, the first command in the input edge-end spatiotemporal model can be missing). In this way, the drone airborne device 110 completes the given task independently.

[0049] The flight control module 116 adjusts the flight state of the drone according to the control instructions.

[0050] The control instructions mainly include dynamic decision-making and action optimization adjustments. In some embodiments, the control instructions include the following four categories: flight control, mission execution, communication coordination, and emergency response. The specific instructions are as follows.

[0051] Flight control: Instructions related to basic flight actions of the drone, such as attitude control, position adjustment, hovering control, obstacle avoidance, route adjustment, etc.

[0052] Task execution: operations directly related to the task, such as shooting, inspection, or delivery.

[0053] Communication and collaboration: Supports interaction and data synchronization between drones and the cloud, such as formation adjustment and reporting to the cloud.

[0054] Emergency response: handle emergencies and ensure the safety of drones, such as fault switching, automatic return, etc.

[0055] In some embodiments, the flight control module 116 also includes an operating system ROS (Robot Operating System) and a flight control system, which realize data sharing and command transmission through a communication protocol (mainly serial communication) to ensure that the drone performs tasks safely and efficiently. The communication protocol adopts the MAVLink protocol. The communication process is: ROS generates high-level control instructions and sends them to the flight control system through the MAVLink protocol. After receiving the instructions, the flight control system parses them into low-level control signals to drive the drone to execute; the flight control system continuously sends the real-time flight data of the drone to ROS, ROS obtains the flight data and publishes it to the corresponding ROS topic, and at the same time synchronizes the flight data to the control instruction generation module through the data acquisition module 112 for calculation and analysis.

[0056] In some other embodiments, the flight control module 116 can also control the acquisition parameter configuration of the data acquisition module 112. For example, ROS controls the on / off and acquisition parameter configuration of the data acquisition module 112 through a system interface and a program.

[0057] Continue as Figure 1 The server 120 (i.e., the cloud) includes a first instruction generating device 122, a data management device 124, and a model training device 126 coupled to each other. In some embodiments, each device can be implemented by one or more servers. The first instruction generating device 122, the data management device 124, and the model training device 126 are described below.

[0058] The first instruction generating device 122 is respectively coupled to the data acquisition module 112 and the control instruction generating module 114 of the drone onboard device 110. The first instruction generating device 122 obtains the spatiotemporal coding data from each data acquisition module 112, and processes the spatiotemporal coding data using the cloud spatiotemporal model to generate the first instruction.

[0059] For the structure of the cloud-side spatiotemporal model, please refer to the previous description of the edge-side spatiotemporal model (such as Figure 2 According to the implementation mode of the present application, the cloud-based spatiotemporal model is deployed in the cloud, which is suitable for global planning of low-altitude airspace.

[0060] At the same time, the server 120 centrally trains the spatiotemporal model, optimizes the algorithm and generates a new model, and sends the optimized model (ie, the edge-end spatiotemporal model) to the drone airborne device 110. The training process of the cloud-based spatiotemporal model is described below.

[0061] The data management device 124 is coupled to the data acquisition module 112 of the drone onboard device 110. The data management device 124 obtains the spatiotemporal coded data from the data acquisition module 112 and combines it with the historical data to generate a training data set.

[0062] Specifically, the data management device 124 obtains the spatiotemporal coding data of the corresponding drone airborne device via the data acquisition module 112. In some embodiments, the spatiotemporal coding data acquired by the data acquisition module 112 is uploaded to the server 120 in real time.

[0063] At the same time, the data management device 124 obtains historical data, including: historical environmental information, historical flight mission records, historical flight status data, communication and collaboration records, and system operation logs. Among them, the historical environmental information includes: geographic information, weather, dynamic obstacles, and environmental risk records; the historical flight mission records include: historical flight mission planning, flight mission type, and flight mission resource consumption; the historical flight status data includes: historical flight trajectory, flight attitude records, power system data, and battery performance data.

[0064] Afterwards, the data management device 124 combines the historical data and the spatiotemporal coding data to generate a training data set, and sends the training data set to the model training device 126 for model training.

[0065] The model training device 126 uses the training data set to train and generate a cloud spatiotemporal model. The training process includes the following three steps.

[0066] The first step is to label the training data set according to the task category to obtain the labeled data. The task categories include perception tasks, prediction tasks, planning and decision-making tasks.

[0067] Specifically, for perception tasks, the annotated data includes: the location and type of the target in the image, the dynamic and static objects in the point cloud, and the abnormal data in the inertial measurement data. For prediction tasks, the annotated data includes: the trajectory of the dynamic target (such as position, speed, direction), and the future state of the time series. For planning and decision-making tasks, the annotated data includes: obstacle avoidance behavior (such as turning, acceleration, deceleration), and key nodes of path planning (such as starting point, end point, and intermediate nodes).

[0068] In some other embodiments, data enhancement is first performed on the training data set, and then the enhanced training data set is labeled.

[0069] Specifically, data enhancement includes: image enhancement, point cloud enhancement, and sequence enhancement. Among them, image enhancement operations include rotation, cropping, blurring, and illumination changes; point cloud enhancement operations include random sampling, noise addition, and sparseness; and sequence data enhancement operations include data cropping and noise injection.

[0070] The second step is to build a pre-training model. The pre-training model is based on the large language model. The pre-training model also includes coupled feature extraction components, perception task components, prediction task components, and planning and decision components (such as Figure 2 as shown).

[0071] In some embodiments, the pre-trained model may be based on a LM4STD model capable of processing time series and spatiotemporal data, but is certainly not limited thereto.

[0072] The third step is to train the pre-trained model based on the labeled data to obtain the cloud-based spatiotemporal model.

[0073] Specifically, self-supervised learning and multi-task joint training are used to design corresponding training tasks for different tasks. For example, for perception tasks, the training task design can be target detection (Masked Region Modeling), that is, randomly masking image areas to predict targets. For prediction tasks, the training task design can be time series prediction (Masked Time Prediction), that is, randomly masking time points to predict future values. For planning and decision-making tasks, the training task design can be path planning tasks, that is, incremental path prediction (planning trajectory completion). And so on.

[0074] The pre-trained model is trained through the established tasks, including: freezing the feature extraction component and only fine-tuning the high-level task-related components (i.e., perception task component, prediction task component, and planning and decision-making component); data set adaptation, adjusting the input format and model structure to adapt to specific task data; fine-tuning for application areas such as perception, prediction, and planning tasks, etc.

[0075] The training of the model is well known to those skilled in the art and will not be elaborated here. After the training is completed, the trained model is deployed in the first instruction generating device 122 as a cloud spatiotemporal model.

[0076] In addition, the data management device 124 is also coupled to the control instruction generation module 114 of the drone onboard device 110. The data management device 124 also fine-tunes the cloud-side spatiotemporal model to obtain the edge-side spatiotemporal model and sends it to the control instruction generation module 114.

[0077] In some embodiments, the fine-tuning of the cloud-based spatiotemporal model mainly involves adjusting the model parameters so that the spatiotemporal model can adapt to the edge drone airborne device. Specifically, the fine-tuning can be, for example, efficient parameter fine-tuning, such as: reducing model training parameters (LoRA (Low-Rank Adaptation)); inserting a lightweight adaptation layer (Adapter Tuning); pruning and quantization, including: trimming redundant network parameters; compressing parameters from floating point numbers to low-precision integers, etc.

[0078] At the same time, the data management device 124 will also use the operating results and execution feedback of the edge-end spatiotemporal model to optimize the cloud-end spatiotemporal model.

[0079] This process is achieved through real-time data feedback, dynamic model updates and incremental optimization, ensuring that the cloud-based spatiotemporal model continuously improves its adaptability and performance.

[0080] After the edge spatiotemporal model completes reasoning and execution, two types of data are generated, namely: reasoning results and execution feedback. The data management device 124 evaluates the uploaded online results and analyzes the performance of the edge spatiotemporal model, including: reasoning accuracy, performance bottlenecks, execution anomalies, etc. After that, the optimization process is triggered, and the data management device 124 combines the real-time uploaded data with the historical data to form a new training data set, realizes annotation optimization and abnormal data supplementation, and enhances the cloud spatiotemporal model capabilities or optimizes or fine-tunes the model for specific scenarios through incremental learning, multi-task learning, scenario-specific optimization, etc. The optimized model is then distributed to the control instruction generation module 114 in full or incremental manner through the data management device 124.

[0081] In summary, the data management device 124 realizes two-way interaction with the control instruction generation module 114 through the network, realizes model services such as model deployment, algorithm / service calling, model version management, grayscale release, quality assurance monitoring, and policy configuration, and realizes the collaboration between the cloud-side spatiotemporal model and the edge-side spatiotemporal model.

[0082] According to the system 100 of the present application, a complete spatiotemporal model is established based on the attention mechanism of the Transformer architecture, and the spatiotemporal model deployment of cloud-based collaborative spatiotemporal AI is realized by combining the powerful model computing capabilities of the server side and the real-time computing advantages of the airborne edge side. Based on this, a drone autopilot system is constructed, which can perform real-time calculation and reasoning during the drone autopilot process, and control the drone flight control system in real time, making drone flight safer and more flexible.

[0083] In addition, the system 100 can effectively reduce the edge data processing requirements of the drone, reduce the energy consumption of the drone, and thus improve the flight endurance of the drone. At the same time, the server processes the real-time spatiotemporal coding data and synchronously generates the first instruction to the control instruction generation module, which can enhance the global perception capability of the drone and improve flight safety. In addition, the high-frequency transmission of massive and large-capacity indiscriminate information of the drone is reduced, and the efficiency of information processing is improved.

[0084] The system 100 according to the present application can be implemented by multiple computing devices. Figure 3 FIG. 3 is a schematic diagram of a computing device 300 according to some embodiments of the present application. It should be noted that: Figure 3 The computing device 300 shown is only an example. In practice, the computing device used to implement the method of the present application can be any type of device, and its hardware configuration can be different from the above. Figure 3 The computing device 300 shown is the same as Figure 3 The computing device 300 shown is different. In practice, the computing device used to implement the embodiments of the present application can be Figure 3 The hardware components of the computing device 300 shown are added or deleted, and the present application does not limit the specific hardware configuration of the computing device.

[0085] like Figure 3 In a basic configuration, computing device 300 includes at least one processing unit 302 and system memory 304. According to one aspect, depending on the configuration and type of computing device, processing unit 302 can be implemented as a processor. System memory 304 includes, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. According to one aspect, system memory 304 includes operating system 305 and program modules 306.

[0086] According to one aspect, operating system 305 is suitable for controlling the operation of computing device 300, for example. Furthermore, examples are practiced in conjunction with graphics libraries, other operating systems, or any other application programs, and are not limited to any particular application or system. Figure 3 This basic configuration is illustrated in FIG. 3 by those components within dashed line 308. According to one aspect, computing device 300 has additional features or functionality. For example, according to one aspect, computing device 300 includes additional data storage devices (removable and / or non-removable), such as magnetic disks, optical disks, or tapes. Such additional storage Figure 3 3 is illustrated by removable storage 309 and non-removable storage 310.

[0087] As stated above, according to one aspect, program modules are stored in the system memory 304. According to one aspect, the program modules may include one or more application programs, and the present application does not limit the type of application program, for example, the application program may include: email and contact application programs, word processing application programs, spreadsheet application programs, database application programs, slide show application programs, drawing or computer-aided application programs, web browser application programs, etc.

[0088] According to one aspect, the examples may be practiced on a circuit comprising discrete electronic components, a packaged or integrated electronic chip containing logic gates, a circuit utilizing a microprocessor, or a single chip containing electronic components or a microprocessor. Figure 3 Each or many components shown in can be integrated in a system on chip (SOC) on a single integrated circuit to practice examples. According to one aspect, such a SOC device may include one or more processing units, a graphics unit, a communication unit, a system virtualization unit, and various application functions, all of which are integrated (or "burned") into a chip substrate as a single integrated circuit. When operated via SOC, the functions described in the present application can be operated via a dedicated logic integrated with other components of a computing device 300 on a single integrated circuit (chip). Other technologies that can perform logical operations (such as AND, OR, and NOT) can also be used to practice embodiments of the present application, including but not limited to mechanical, optical, fluid, and quantum technologies. In addition, embodiments of the present application can be practiced in a general-purpose computer or in any other circuit or system.

[0089] According to one aspect, the computing device 300 may also have one or more input devices 312, such as a keyboard, a mouse, a pen, a voice input device, a touch input device, a VR motion capture input device, etc. Output devices 314 may also be included, such as a display, a speaker, a printer, etc. The aforementioned devices are examples and other devices may also be used. The computing device 300 may include one or more communication connections 316 that allow communication with other computing devices 318. Examples of suitable communication connections 316 include, but are not limited to: RF transmitters, receivers and / or transceiver circuits; Universal Serial Bus (USB), parallel and / or serial ports.

[0090] The term computer-readable medium as used in this application includes computer storage media. Computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (e.g., computer-readable instructions, data structures, or program modules). System memory 304, removable storage 309, and non-removable storage 310 are all examples of computer storage media (i.e., memory storage). Computer storage media may include random access memory (RAM), read-only memory (ROM), electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices, or any other products that can be used to store information and can be accessed by computing device 300. According to one aspect, any such computer storage medium can be part of computing device 300. Computer storage media do not include carrier waves or other propagated data signals.

[0091] According to one aspect, communication media is implemented by computer readable instructions, data structures, program modules, or other data in a modulated data signal (e.g., a carrier wave or other transport mechanism), and includes any information delivery media. According to one aspect, the term "modulated data signal" describes a signal that has one or more characteristics set or changed in a manner that encodes information in the signal. By way of example and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

[0092] In some embodiments of the present application, the computing device 300 is configured to execute the unmanned aerial vehicle automatic driving method 400 based on cloud-based collaborative spatiotemporal AI according to the present application. Among them, the program module 306 includes multiple program instructions for executing the above method 400, and these program instructions can instruct the processing unit 302 to execute the corresponding method.

[0093] Figure 4 The flowchart of the method 400 for autonomous driving of a drone based on cloud-based collaborative spatiotemporal AI according to some embodiments of the present application is shown. The method 400 is suitable for execution in the drone airborne device 110 of the aforementioned system 100. The description of the method 400 is complementary to the drone airborne device 110, and the relevant parts are not repeated here.

[0094] like Figure 4 As shown, the method 400 starts at step S410 .

[0095] In S410, the flight data and airspace spatiotemporal data of the UAV during flight are collected, including flight position data, driving data, and flight airspace data.

[0096] In S420, based on the three-dimensional grid, the flight data and the airspace spatiotemporal data are processed to generate spatiotemporal coded data.

[0097] In S430, the spatiotemporal coding data, the flight mission data, the low-altitude airspace environment data and the first instruction are processed using the edge spatiotemporal model to generate a control instruction, wherein the first instruction is determined by the server at least based on the spatiotemporal coding data.

[0098] In S440, the flight state of the UAV is adjusted according to the control instruction.

[0099] In some embodiments of the present application, the computing device 300 is configured to execute the unmanned aerial vehicle automatic driving method 500 based on cloud-based collaborative spatiotemporal AI according to the present application. Among them, the program module 306 includes multiple program instructions for executing the above method 500, and these program instructions can instruct the processing unit 302 to execute the corresponding method.

[0100] Figure 5 The flowchart of the method 500 for autonomous driving of a drone based on cloud-based collaborative spatiotemporal AI according to some embodiments of the present application is shown. The method 500 is suitable for execution in the server 120 of the aforementioned system 100. The description of the method 500 is complementary to the server 120, and the relevant parts are not repeated here.

[0101] like Figure 5 As shown, the method 500 starts at step S510. In S510, the cloud-based spatiotemporal model is used to process the spatiotemporal coded data from the drone airborne device to generate a first instruction, so that the drone airborne device can determine a control instruction for adjusting the drone flight state in combination with the first instruction.

[0102] Among them, the cloud-based spatiotemporal model includes feature extraction components, perception task components, prediction task components, planning and decision-making components, and the cloud-based spatiotemporal model is trained by a training data set generated by spatiotemporal coding data and historical data.

[0103] Then, in S520, the cloud-side spatiotemporal model is optimized according to the operation results and execution feedback of the edge-side spatiotemporal model in the drone onboard device.

[0104] According to the solution of this application, a complete spatiotemporal model is established based on the Transformer architecture, and the powerful model computing power of the server side and the real-time computing advantages of the airborne edge side are combined to realize the deployment of cloud-coordinated spatiotemporal AI models. Based on this, a drone automatic driving system is constructed. During the drone automatic driving process, flight data can be collected through the data acquisition module to obtain spatiotemporal coding data, and the edge-side spatiotemporal model can be used to perform real-time computational reasoning on the spatiotemporal coding data; at the same time, the server performs computational reasoning on the real-time spatiotemporal coding data, and synchronously generates a first instruction for the edge-side spatiotemporal model to perform computational reasoning, and finally generates a control instruction to control the flight status of the drone in real time. This solution can enhance the drone's global perception capability and improve flight safety.

[0105] The various techniques described herein may be implemented in combination with hardware or software, or a combination thereof. Thus, the method and apparatus of the present application, or certain aspects or portions of the method and apparatus of the present application may be in the form of program codes (i.e., instructions) embedded in a tangible medium, such as a removable hard disk, a USB flash drive, a floppy disk, a CD-ROM, or any other machine-readable storage medium, wherein when the program is loaded into a machine such as a computer and executed by the machine, the machine becomes a device for practicing the present application.

[0106] In the case where the program code is executed on a programmable computer, the computing device generally includes a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The memory is configured to store the program code; the processor is configured to execute the method of the present application according to the instructions in the program code stored in the memory.

[0107] By way of example and not limitation, readable media include readable storage media and communication media. Readable storage media stores information such as computer readable instructions, data structures, program modules or other data. Communication media generally embody computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transmission mechanism, and include any information transfer medium. Any combination of the above is also included within the scope of readable media.

[0108] In the description provided herein, algorithms and displays are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the examples of the present application. According to the above description, it is obvious to construct the structure required for such systems. In addition, the present application is not directed to any specific programming language either. It should be understood that the content of the present application described herein can be realized by various programming languages, and the description of the specific language above is to disclose the preferred implementation of the present application.

[0109] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0110] Those skilled in the art will appreciate that the modules or units or components of the devices in the examples disclosed herein may be arranged in the devices described in the embodiment, or alternatively may be located in one or more devices different from the devices in the examples. The modules in the foregoing examples may be combined into one module or may be divided into multiple submodules.

[0111] Those skilled in the art will appreciate that the modules in the devices in the embodiments can be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments can be combined into one module or unit or component, and in addition they can be divided into multiple submodules or subunits or subassemblies. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification and all processes or units of any method or device disclosed in this manner can be combined in any combination. Unless otherwise clearly stated, each feature disclosed in this specification can be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0112] Furthermore, those skilled in the art will appreciate that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present application and to form different embodiments.

[0113] In addition, some of the embodiments are described herein as methods or combinations of method elements that can be implemented by a processor of a computer system or by other devices that perform the functions. Therefore, a processor with necessary instructions for implementing the method or method elements forms a device for implementing the method or method elements. In addition, the elements described herein of the device embodiments are examples of devices that are used to implement the functions performed by the elements for the purpose of implementing the disclosure.

[0114] As used herein, unless otherwise specified, the use of ordinal numbers "first," "second," "third," etc. to describe common objects merely refers to different instances of similar objects and is not intended to imply that the objects so described must have a given order in time, space, order, or in any other manner. In addition, the quantifier "plurality" means "two" and / or "more than two."

Claims

1. The drone autopilot system based on cloud-based collaborative spatiotemporal AI is characterized by: include: One or more drone onboard devices, wherein each drone onboard device comprises a data acquisition module, a control instruction generation module and a flight control module, wherein: The data acquisition module is suitable for collecting flight data and airspace spatiotemporal data during the flight of the UAV, and processing the flight data and airspace spatiotemporal data based on the three-dimensional grid to generate spatiotemporal coded data; The control instruction generation module is adapted to process the spatiotemporal coding data, the flight mission data, the low-altitude airspace environment data and the first instruction using the edge-end spatiotemporal model to generate a control instruction; The flight control module is adapted to adjust the flight state of the UAV according to the control instruction; A first instruction generating device, coupled to the data acquisition module and the control instruction generating module respectively, adapted to acquire the spatiotemporal coded data and process the spatiotemporal coded data using a cloud spatiotemporal model to generate the first instruction; A data management device, coupled to the data acquisition module and the control instruction generation module respectively, adapted to acquire the spatiotemporal coded data and generate a training data set in combination with historical data; The model training device is coupled to the data management device and is suitable for using the training data set to train and generate the cloud-based spatiotemporal model.

2. The system according to claim 1, characterized in that The data management device is also suitable for: Fine-tune the cloud-side spatiotemporal model to obtain the edge-side spatiotemporal model, and send it to the control instruction generation module; The cloud-side spatiotemporal model is optimized using the running results and execution feedback of the edge-side spatiotemporal model. Among them, the cloud-side spatiotemporal model and the edge-side spatiotemporal model are based on the Transformer architecture.

3. The system according to claim 1, characterized in that In the drone airborne device, the data acquisition module further includes a data processing unit, and the data processing unit is adapted to: Preprocessing the flight data and the airspace spatiotemporal data respectively, wherein the preprocessing includes: spatiotemporal alignment, anomaly detection, and normalization; According to the geographic location information and time information of each flight data and airspace spatiotemporal data, it is mapped to the corresponding three-dimensional grid, and each three-dimensional grid is encoded to obtain spatiotemporal encoded data.

4. The system according to claim 3, characterized in that In the drone onboard device, The data acquisition module includes: positioning equipment, video acquisition equipment, inertial measurement unit, laser radar, heading reference system, infrared sensor; The flight data and airspace spatiotemporal data include: flight position data, driving data, flight airspace data, and video data.

5. The system according to claim 4, characterized in that The data management device is suitable for: Acquiring the time-space coding data of the corresponding UAV airborne device via the data acquisition module; Acquiring historical data, including: historical environmental information, historical flight mission records, historical flight status data, communication and collaboration records, and system operation logs, wherein the historical environmental information includes: geographic information, weather, dynamic obstacles, and environmental risk records; the historical flight mission records include: historical flight mission planning, flight mission type, and flight mission resource consumption; the historical flight status data includes: historical flight trajectories, flight attitude records, power system data, and battery performance data; The historical data and the spatiotemporal coded data are combined to generate a training data set.

6. The system according to claim 1, characterized in that The model training device is suitable for: Annotating the training data set according to task categories to obtain annotated data, wherein the task categories include perception tasks, prediction tasks, and planning and decision-making tasks; Constructing a pre-training model, wherein the pre-training model is based on a large language model and includes a feature extraction component, a perception task component, a prediction task component, and a planning and decision component; Based on the labeled data, the pre-trained model is trained to obtain the cloud-based spatiotemporal model.

7. The system according to claim 6, characterized in that For the perception task, the annotated data includes: the location and type of the target in the image, the dynamic and static objects in the point cloud, and the abnormal data in the inertial measurement data; For the prediction task, the labeled data includes: dynamic target trajectory, future state of time series; For the planning and decision-making tasks, the annotated data includes: obstacle avoidance behaviors and key nodes of path planning, wherein the obstacle avoidance behaviors include turning, accelerating, and decelerating.

8. The system of claim 1, wherein: The flight control module is also suitable for controlling the acquisition parameter configuration of the data acquisition module.

9. A method for autonomous driving of a drone based on cloud-based collaborative spatiotemporal AI, the method being suitable for execution in a drone onboard device, the drone onboard device being connected to a server, the method comprising: Collect flight data and airspace spatiotemporal data of drones during flight, including: flight position data, driving data, and flight airspace data; Based on the three-dimensional grid, the flight data and the airspace spatiotemporal data are processed to generate spatiotemporal coded data; The spatiotemporal coding data, the flight mission data, the low-altitude airspace environment data and the first instruction are processed by using the edge spatiotemporal model to generate a control instruction; According to the control instruction, the flight state of the UAV is adjusted, Wherein, the first instruction is determined by the server at least based on the spatiotemporal coding data.

10. A method for autonomous driving of a drone based on cloud-based collaborative spatiotemporal AI, the method being suitable for execution in a server connected to one or more drone-mounted devices, the method comprising: Using the cloud-based spatiotemporal model, the spatiotemporal coded data from the drone airborne device is processed to generate a first instruction, so that the drone airborne device can determine a control instruction for adjusting the flight state of the drone in combination with the first instruction; Optimizing the cloud-based spatiotemporal model according to the running results and execution feedback of the edge-side spatiotemporal model in the drone-mounted device; The cloud-based spatiotemporal model includes a feature extraction component, a perception task component, a prediction task component, and a planning and decision-making component, and the cloud-based spatiotemporal model is trained by a training data set generated by the spatiotemporal coding data and historical data.

11. A computing device, characterized in that: include: one or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising instructions for executing the method according to claim 9 or 10.

12. A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which when executed by a computing device, cause the computing device to perform the method according to claim 9 or 10.

13. A computer program product comprising a computer program / instructions, wherein: When the computer program / instructions are executed by a processor, the method according to claim 9 or 10 is implemented.

Images