A Cesium-based multi-machine, multimodal remote sensing monitoring and 3D visualization system and method

Abstract

This invention discloses a multi-aircraft, multi-modal remote sensing monitoring and 3D visualization system and method based on Cesium. The system includes: a data acquisition and communication module, a backend data processing module, a data management and publishing module, a spatial analysis module, a monitoring module, a video stream synchronization module, and a frontend display module. The system receives multi-aircraft flight status and payload data in real time, achieves status data fusion through time alignment and anomaly removal, and constructs continuous and smooth flight paths; it achieves lightweight video frame stitching, and projects the stitched image onto a 3D surface in combination with the UAV's pose. This invention supports spatial analysis for auxiliary decision-making in UAV missions, and realizes automatic publishing and multi-source fusion display of multi-modal remote sensing data. It achieves integrated visualization and linkage analysis of multi-source remote sensing information, which can improve the situational awareness, spatial positioning accuracy, and real-time data processing of multi-aircraft collaborative missions. It has the characteristics of strong real-time performance, high data fusion degree, and excellent 3D display effect.

Description

Technical Field

[0001] This invention relates to the fields of unmanned aerial vehicle (UAV) remote sensing, geographic information systems, and 3D visualization technologies, and more specifically, to a Cesium-based multi-drone multimodal remote sensing monitoring and 3D visualization system and method. Background Technology

[0002] With the widespread application of UAV remote sensing technology in agriculture, environmental monitoring, and emergency rescue, the information perception capabilities of single-unit, single-payload operations are weak, and the efficiency of remote sensing operations over large areas is low. In the future, multi-unit collaborative operations based on mobile platforms will become an important development direction for UAV remote sensing. Therefore, UAV management systems face the challenge of data diversity brought about by multi-unit collaboration and multi-sensor integration. UAVs not only transmit real-time flight status data but also collect multimodal remote sensing data (such as color imagery, multispectral data, infrared data, and laser point clouds) through various payloads.

[0003] In existing technologies, most drone monitoring systems are in the form of two-dimensional maps, with weak three-dimensional visualization and interactive capabilities, making it difficult to accurately reproduce the flight situation; multi-source data are isolated from each other, lacking a fusion and linkage display mechanism; the preprocessing of multimodal data relies on manual labor, with insufficient automation and real-time performance, failing to meet the real-time processing, publishing, and visualization needs in multi-drone operation scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide a Cesium-based multi-drone, multi-modal remote sensing monitoring and 3D visualization system and method for unmanned aerial vehicles (UAVs), addressing technical problems in existing multi-UAV remote sensing systems such as difficulty in unified access and management of multi-source asynchronous data, insufficient linkage between flight status and mission information, limited spatial analysis and mission support capabilities, and difficulty in real-time fusion and display of video information with geospatial data. This invention constructs a closed-loop collaborative system for multi-UAV remote sensing, encompassing "acquisition—fusion—analysis—display—return control," covering multiple modules including data acquisition, data parsing and management, automated data service publishing, spatial analysis and decision support, flight path and mission monitoring, video stream stitching and 3D projection. It also transmits mission adjustment commands and interactive operations based on analysis results back to the backend for processing, meeting the high real-time and high interactivity requirements for data fusion, visualization, and mission monitoring in multi-UAV collaborative operations. This invention further addresses the technical challenges of integrating and displaying real-time video data with 3D geographic scenes by proposing a browser-based method for real-time video frame stitching and 3D synchronous visualization. Through feature matching, spatiotemporal synchronization, and image projection strategies, the video information is synchronously mapped with the UAV's attitude trajectory and geographical location. This enables low-latency, highly robust real-time 3D video overlay display without relying on high-performance servers, thereby improving the UAV's situational awareness and spatial positioning accuracy in complex scenarios.

[0005] To achieve the above objectives, the first aspect of the present invention provides a Cesium-based multi-machine, multi-modal remote sensing monitoring and 3D visualization system, comprising:

[0006] The data acquisition and communication module is used to collect real-time flight data and mission metadata of the UAV and acquire multimodal remote sensing data. The real-time flight data of the UAV includes flight status data and video stream data.

[0007] The backend data processing module is used to preprocess real-time flight data, multimodal remote sensing data, and mission metadata of UAVs, and provides data interfaces;

[0008] The data management and publishing module is used to process multimodal remote sensing data based on the data interface, generate corresponding map services and 3D data services according to the data type, and publish them.

[0009] The spatial analysis module is used to perform spatial analysis by calling map services or 3D data services based on preprocessed real-time flight data, mission metadata and pre-saved terrain data, to assess mission feasibility and generate mission feasibility reports and optimized execution plans.

[0010] The monitoring module is used to perform continuous verification and interval sampling of waypoints based on real-time UAV flight data, mission metadata, mission feasibility reports, and optimized execution plans, construct three-dimensional waypoint curves, and generate real-time waypoint layers.

[0011] The video stream synchronization module is used to receive flight status data and video stream data, and to stitch and fuse the UAV video frames through a front-end lightweight algorithm;

[0012] The front-end display module is used to build a 3D geographic scene based on the Cesium engine and comprehensively display the analysis results of each module.

[0013] In one implementation, the data acquisition and communication module is further used for:

[0014] A thematic data distribution strategy is introduced, and independent message channels are established according to the drone number and data type to transmit data from multiple drones in parallel.

[0015] The collected data undergoes preliminary analysis, time alignment, and compression before being pushed to the backend data processing module.

[0016] In one implementation, the backend data processing module includes:

[0017] The preprocessing unit is used to clean, standardize, and perform temporal and spatial correction on real-time flight data, multimodal remote sensing data, and mission metadata.

[0018] The association unit establishes a relationship between flight status data and mission metadata in real-time flight data and then stores them in the database in a unified manner.

[0019] The data forwarding and synchronization management unit is used to establish a bridge channel with the MQTT message queue and push processed data in conjunction with WebSocket.

[0020] The interface support unit provides interface support for the automated publishing of multimodal remote sensing data.

[0021] In one implementation, the data management and publishing module is also used for:

[0022] Automatic generation of time-series indexes and pyramid slice structures for multi-temporal UAV imagery;

[0023] An interface was established with the video stream synchronization module to enable the real-time generated stitched images to be mounted to the GeoServer server.

[0024] In one implementation, the spatial analysis module is specifically used for:

[0025] The system calls upon map services or 3D data services to perform buffer analysis, visibility analysis, optimal path calculation, and area measurement. Buffer analysis includes calculating the safety boundary of the flight path or mission area; visibility analysis includes assessing the monitoring coverage of the UAV in the mission area; optimal path calculation includes generating and optimizing the flight path based on 3D terrain and obstacles; and area measurement includes measuring the length and area of ​​the flight mission area.

[0026] In one implementation, the safety boundary range of the flight path or mission area is calculated as follows:

[0027]

[0028] Where (x, y) are the coordinates of any point in space, and n is the number of route points, with the i-th route point as the starting point. Generate a circular safe zone with radius r centered at the center. All circular safe zones are then processed by union operation. Connect them to form a continuous route buffer zone. , , For waypoints The coordinates.

[0029] In one implementation, assessing the surveillance coverage of the drone in the mission area includes:

[0030] The terrain model is divided into regular grids. Visibility is determined for the line connecting each grid point Q and the UAV observation point O. Specifically, discrete sampling is performed along the line connecting O and Q, and the terrain elevation value of each sampling point is extracted from the 3D terrain data. If the terrain elevation of any point is higher than the height of the line of sight at the same location, the line of sight is considered to be blocked; otherwise, the point is visible.

[0031] Among all visible points, those that satisfy the field-of-view constraint constitute the actual visible area of ​​the UAV at a specified altitude. The field-of-view constraint is:

[0032]

[0033] in, For the field of view, For the maximum observation distance, From the perspective of This is the orientation vector of the drone.

[0034] In one implementation, generating an optimized flight path based on three-dimensional terrain and obstacles includes:

[0035] The UAV mission area is discretized into a three-dimensional voxel grid, where each voxel stores terrain height, obstacle occupancy status, and meteorological data, forming a set of cost functions to comprehensively reflect terrain characteristics;

[0036] By setting weight parameters, the impact of different factors on path planning can be adjusted. These factors include terrain elevation difference cost, obstacle collision cost, and energy consumption cost.

[0037] Using three-dimensional Alternatively, the Dijkstra algorithm can perform a heuristic search in the cost graph to obtain the route with the minimum cumulative cost, which can then be used as the optimized route.

[0038] In one implementation, the video stream synchronization module is specifically used for:

[0039] For two adjacent video frames, a preset algorithm is used to detect feature points;

[0040] Feature matching is performed on feature points of two adjacent video frames;

[0041] A homography matrix is ​​constructed using a random sampling consistency method to eliminate erroneous matching points;

[0042] By using the homography matrix, the coordinate system of the next frame image is mapped to that of the previous frame image. A weighted fusion algorithm is then used to smooth the overlapping areas to obtain the stitched image.

[0043] The stitched images are projected onto the Cesium 3D Earth scene based on the UAV's position information and attitude parameters, forming a real-time video overlay layer synchronized with the flight path and mission status.

[0044] Based on the same inventive concept, a second aspect of this invention provides a Cesium-based multi-machine, multi-modal remote sensing monitoring and 3D visualization method, comprising:

[0045] The data acquisition and communication module is used to collect real-time flight data and mission metadata of the UAV to obtain multimodal remote sensing data. The real-time flight data of the UAV includes flight status data and video stream data.

[0046] The backend data processing module is used to preprocess real-time flight data, multimodal remote sensing data, and mission metadata of UAVs, and provides a data interface.

[0047] The data management and publishing module processes multimodal remote sensing data based on the data interface, generates corresponding map services and 3D data services according to the data type, and publishes them.

[0048] The spatial analysis module utilizes preprocessed real-time flight data, mission metadata, and pre-saved terrain data to call map services or 3D data services for spatial analysis, assess mission feasibility, and generate mission feasibility reports and optimized execution plans.

[0049] The monitoring module uses real-time UAV flight data, mission metadata, mission feasibility reports, and optimized execution plans to perform continuity verification and interval sampling of waypoints, construct a three-dimensional waypoint curve, and generate a real-time waypoint layer.

[0050] The video stream synchronization module receives flight status data and video stream data, and the drone video frames are stitched and fused using a front-end lightweight algorithm;

[0051] The front-end display module utilizes the Cesium engine to construct a 3D geographic scene and comprehensively displays the analysis results from each module.

[0052] Compared with the prior art, the advantages and beneficial technical effects of the present invention are as follows:

[0053] (1) Proposal of a full-link multi-UAV monitoring system architecture: A multi-UAV multimodal remote sensing monitoring and 3D visualization system based on Cesium was constructed. It is a full-link multi-UAV monitoring system architecture based on a 3D geographic engine, realizing closed-loop integration from multi-source data acquisition and reception, data parsing and fusion, pre-emptive spatial analysis assistance, task monitoring to 3D visualization display. The system adopts a modular design and asynchronous message-driven mechanism, unifying multi-UAV status data, flight path information, task metadata, real-time video streams and multimodal remote sensing images into the same spatiotemporal coordinate system, realizing real-time docking, parsing, fusion and 3D dynamic expression of multi-UAV asynchronous data streams. This invention utilizes the Cesium 3D Earth engine to change the limitations of traditional 2D map monitoring systems, such as fragmented data chains, static 2D interface monitoring, and separation of video and geographic scenes. It can comprehensively and three-dimensionally display the flight paths, attitudes and multi-source remote sensing data of multiple UAVs in a full-domain three-dimensional visualization, improving the spatial cognition and real-time situation visualization capabilities of multi-UAV collaborative tasks.

[0054] (2) Multi-source and multi-modal data end-to-end fusion and automation system: This invention proposes a real-time fusion and automated service publishing system for multi-source and multi-modal remote sensing data in scenarios involving vehicle-mounted dual airports and multi-UAV multi-payload collaborative operations. Unlike existing technologies where multiple types of sensor data require manual processing and publishing, and are difficult to display in three-dimensional linkage, this invention achieves an integrated pipeline of "multi-source access—unified parsing—fusion into the database—automatic publishing—three-dimensional linkage display" in cross-sensor and multi-platform scenarios. The system forms a multi-source heterogeneous remote sensing data fusion framework through the collaboration of the backend data processing and management module and the multi-modal remote sensing data management and publishing module, and has multiple capabilities such as unified spatiotemporal reference alignment, data format standardization, task semantic association, cross-modal index management, and automated map / 3D service publishing. It supports the seamless integration and interactive display of real-time status data (location, battery level), mission flight path, real-time video stream, and multi-source remote sensing data (images, vectors, models) in the same three-dimensional space. It solves the problems of data isolation, disjointed display, and time-consuming post-processing in existing systems, and realizes a rapid remote sensing mission closed loop of "collecting, integrating, analyzing, and displaying simultaneously", which improves the integration degree and visualization expression capability of remote sensing data.

[0055] (3) Lightweight Real-Time Video Stitching and Quick View Generation Mechanism in the Front-End Environment: Addressing the limitations of backhaul bandwidth, asynchronous video transmission, and terminal computing power in a vehicle-mounted dual-airport operation environment, this invention designs a lightweight real-time video stream stitching mechanism in the front-end environment. Instead of directly calling existing vision libraries for offline stitching, this invention migrates the data flow of feature detection, robust matching, homography matrix estimation, and perspective projection to the front-end. Lightweight browser-side deployment is achieved using WebAssembly and WebWorker, enabling real-time front-end execution of asynchronous UAV video frames and pose data synchronization, lightweight feature matching and error removal, and real-time 3D surface texturing driven by UAV attitude parameters. This method can complete real-time video frame stitching and dynamic 3D geospatial fusion display without requiring back-end GPU / server computation. Compared with existing technologies, this invention enables real-time video stitching under low-bandwidth / no dedicated network conditions, real-time binding and coordinate consistency between stitched images and 3D space, and is no longer limited to the traditional "video window mode."

[0056] (4) Spatiotemporal synchronization and registration mechanism of video stream and attitude data: To address the issues of asynchronous transmission between multiple machines and inconsistent sampling frequencies of different payloads, this invention establishes a temporal synchronization and spatial calibration mechanism for video frames and UAV attitude data. Through unified timestamp calibration and attitude angle correction, precise alignment between video images and the spatial position of the UAV is achieved. This mechanism solves problems such as inconsistent sampling frequencies and asynchronous transmission delays between video and sensor data, enabling the stitched image to maintain consistency with the real-time trajectory and heading of the UAV in the 3D scene, providing high-precision support for subsequent geographic projection and 3D spatial fusion. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0058] Figure 1 This is a structural diagram of the Cesium-based multi-machine multimodal remote sensing monitoring and 3D visualization system in an embodiment of the present invention;

[0059] Figure 2 This is a schematic diagram of multi-source heterogeneous data types in the system of this embodiment of the invention;

[0060] Figure 3 This is a flowchart of the real-time video stitching and fusion process of the UAV in an embodiment of the present invention;

[0061] Figure 4This is a flowchart illustrating the multimodal remote sensing data management and publishing process in an embodiment of the present invention. Detailed Implementation

[0062] The relevant terminology involved in this invention is explained as follows:

[0063] Cesium is an open-source JavaScript library for creating 3D globes and map applications, supporting WebGL (Web Graphics Library) rendering technology.

[0064] MQTT (Message Queuing Telemetry Transport) is a lightweight, publish / subscribe-based message transport protocol.

[0065] PostgreSQL: Object-Relational Database Management System.

[0066] WebSocket: A TCP-based full-duplex communication protocol that allows clients to establish persistent connections with servers and perform bidirectional real-time data transmission.

[0067] Spring Boot 3 is a major version of the Spring Boot framework, built on Spring Framework 6, and supports Java 17 and above. It features automatic configuration, starter dependency management, and out-of-the-box functionality.

[0068] MyBatis: A Java-based persistence layer framework.

[0069] GeoServer is a J2EE implementation of the OpenGIS Web Server specification. GeoServer allows users to publish map data and perform update, delete, and insert operations on feature data.

[0070] CesiumLab: A data processing tool used to convert various spatial data into Cesium-compatible formats.

[0071] WMS: OGC (Open Geospatial Information Consortium) Web Map Service Specification, which uses data with geospatial location information to create maps and defines maps as a visual representation of geographic data.

[0072] WMTS: OGC's map tile service.

[0073] WFS: OGC's Web Vector Feature Service, returns vector-level GeoMarketing Language (GML) encoding.

[0074] 3D Tiles: A data format.

[0075] WFS-T is a geographic information service specification defined by the OGC, which supports editing operations (insertion, deletion, and update) on geographic features.

[0076] PostGLS: An object-relational database extension module based on PostgreSQL.

[0077] GeoServer API: GeoServer provides an application programming interface (REST API) for accessing and manipulating its functions, supporting geographic data management, service configuration, and interactive operations.

[0078] ORB: Oriented BRIEF, uses the FAST (features from accelerated segment test) algorithm to detect feature points, and the feature point description is an improvement on the BRIEF feature description algorithm.

[0079] Brute-Force is an image matching method that selects a keypoint in the first image and then performs distance tests on each keypoint in the second image in turn, finally returning the keypoint with the closest distance.

[0080] RANSAC: Random Sample Consensus, is a robust parameter estimation method.

[0081] WebAssembly is a virtual instruction set architecture.

[0082] WebWorker: A mechanism for implementing multithreading in JavaScript.

[0083] Vite: A front-end build tool / framework.

[0084] Vue: A JavaScript framework built on standard HTML, CSS, and JavaScript.

[0085] This embodiment provides a Cesium-based multi-machine, multi-modal remote sensing monitoring and 3D visualization system. Please refer to [link to relevant documentation]. Figure 1 ,include:

[0086] The data acquisition and communication module is used to collect real-time flight data, multimodal remote sensing data, and mission metadata of the UAV. The real-time flight data of the UAV includes flight status data and video stream data.

[0087] In practice, this module collects flight data from multiple drones in real time and transmits it to the backend for processing via the MQTT protocol. After parsing and cleaning, the data is stored in a PostgreSQL database for subsequent analysis and visualization. The flight data is updated in real time and can be dynamically displayed on the front end.

[0088] Specifically, this module is located at the very front of the system's data chain. It can establish communication links with multiple UAVs controlled by the vehicle-mounted dual airports using the MQTT protocol. It collects real-time flight data, including flight status data (such as latitude, longitude, altitude, attitude, and speed) and video stream data, as well as mission metadata. It also acquires multimodal remote sensing data from various payloads. The mission metadata includes information such as mission ID, mission type, mission area, and the number of the UAV to which it belongs. The multimodal remote sensing data includes visible light imagery, multispectral imagery, infrared imagery, and lidar point cloud data. Please see [link / reference]. Figure 2 This is the type of multi-source heterogeneous data in this invention.

[0089] The data acquisition and communication module is also used for:

[0090] A thematic data distribution strategy is introduced, and independent message channels are established according to the drone number and data type to transmit data from multiple drones in parallel.

[0091] The collected data undergoes preliminary analysis, time alignment, and compression before being pushed to the backend data processing module.

[0092] Specifically, a thematic data distribution strategy is introduced, establishing independent message channels based on UAV number and data type. This enables parallel data transmission from multiple UAVs, differentiated subscriptions, and latency minimization. During data transmission, this module performs integrity verification and compression on the collected data to ensure data quality, and then pushes the processed flight data, mission metadata, and multimodal remote sensing data to the backend data processing module.

[0093] The backend data processing module is used to preprocess real-time flight data, multimodal remote sensing data, and mission metadata of UAVs, and provides data interfaces.

[0094] The backend data processing module includes:

[0095] The preprocessing unit is used to clean, standardize, and perform temporal and spatial correction on real-time flight data, multimodal remote sensing data, and mission metadata.

[0096] The association unit establishes a relationship between flight status data and mission metadata in real-time flight data and then stores them in the database in a unified manner.

[0097] The data forwarding and synchronization management unit is used to establish a bridge channel with the MQTT message queue and push processed data in conjunction with WebSocket.

[0098] The interface support unit provides interface support for the automated publishing of multimodal remote sensing data.

[0099] In the specific implementation, the backend uses the Spring Boot 3 framework combined with MyBatis for data operations, and a PostgreSQL database is used to centrally store real-time flight data from multiple UAVs, multimodal remote sensing data, mission metadata, and system-related business information. For multimodal remote sensing data, this module does not directly perform remote sensing image tiling and 3D model conversion. Instead, it pushes remote sensing files and metadata to the multimodal remote sensing data management and publishing module through event triggering and API calls, enabling automatic publishing and service generation of heterogeneous spatial data.

[0100] Specifically, the backend data processing module receives real-time flight data, mission metadata, and multimodal remote sensing data from the data acquisition and communication module. It performs cleaning, format standardization, and temporal and spatial correction on the received data. This module associates flight status data with mission metadata to ensure data consistency during mission execution and stores all the data in a PostgreSQL database. This module also handles real-time data forwarding and synchronization management. By establishing a bridge channel with an MQTT message queue, it achieves low-latency data storage and high-concurrency forwarding of flight status data. Furthermore, it uses WebSocket to push processed real-time flight data to the frontend framework and video stream synchronization module, and pushes processed real-time flight data and mission metadata to the multi-aircraft flight path and mission status monitoring module.

[0101] In addition, this module provides interface support for the data management and publishing module, enabling "one-click" automated publishing of multimodal remote sensing data. This module also serves as the data entry point for the spatial analysis and mission auxiliary decision-making modules, providing them with uniformly spatiotemporally aligned flight data, mission metadata, and pre-processed terrain data from the database. It links flight status data from the flight data with mission metadata and stores them uniformly in the database. Through message distribution and event push mechanisms, it provides the spatial analysis module with uniformly spatiotemporally aligned flight data, mission metadata, and pre-processed and stored terrain data from the database; pushes processed real-time flight data and mission metadata to the monitoring module; pushes processed real-time flight data to the video stream synchronization module and the front-end display module; and pushes multimodal remote sensing data to the data management and publishing module to achieve "one-click" automated service publishing.

[0102] The data management and publishing module is used to process multimodal remote sensing data based on the data interface, generate corresponding map services and 3D data services according to the data type, and publish them.

[0103] The data management and publishing module is also used for:

[0104] Automatic generation of time-series indexes and pyramid slice structures for multi-temporal UAV imagery;

[0105] An interface was established with the video stream synchronization module to enable the real-time generated stitched images to be mounted to the GeoServer server.

[0106] Specifically, the data management and publishing module is used for unified management, fusion processing, and automated publishing of multimodal remote sensing data transmitted from the backend data processing module. Specifically, based on the data interface provided by the backend data processing module, it performs standardization processing, band combination, and format conversion on the received multimodal remote sensing data. It can automatically select the optimal strategy based on the data type; for example, it can automatically select GeoServer or CesiumLab for service publishing based on the data type (raster, vector, point cloud, or 3D model), generating corresponding WMS, WMTS, or WFS standardized map services or 3D Tiles 3D data services, and storing the publishing results (service address, layer name, rendering style) in the database of the backend data processing and management module, thus achieving automated service publishing and unified invocation.

[0107] The module further supports "time-series remote sensing data rasterization management," which automatically generates time-series indexes and pyramid tile structures for multi-temporal UAV imagery, improving the efficiency of backend data retrieval and publishing. The module also establishes an interface with the video stream synchronization module, allowing real-time generated stitched images to be directly mounted to the GeoServer server within this module, achieving "automated service publishing of real-time stitched images." The standardized map and 3D data services output after processing by this module can be called by the front-end display and monitoring modules.

[0108] In practical implementation, the data management and publishing module is responsible for the full lifecycle management of multimodal heterogeneous remote sensing data collected by multiple UAVs and their various payloads (such as RGB cameras, multispectral cameras, infrared thermal imagers, and lidar) under the jurisdiction of the dual vehicle-mounted airports. This enables fully automated processes from data upload, processing, and publishing to visualization and access. Please refer to [link to relevant documentation]. Figure 4 The implementation steps of the module are as follows:

[0109] (1) Multi-source data upload and intelligent parsing

[0110] Users can upload multimodal remote sensing data collected by multiple drones via the front-end page, including vector data (mission flight path, mission area boundary, acquisition point vectors, etc.), image data (color imagery, multispectral, hyperspectral, etc.), and point cloud and model data (LiDAR point cloud, oblique photogrammetry 3D model, etc.). Upon receiving the data file, the system automatically performs file parsing and metadata extraction.

[0111] 1) Verify the integrity of the data file, the uniformity of the coordinate system, and the compatibility of the format;

[0112] 2) Automatically extract and associate multi-dimensional metadata, including: drone ID, payload type, sensor model, spectral band range (for multispectral / hyperspectral), spatial resolution, acquisition timestamp, geographical location, etc.

[0113] 3) Based on the data type and collection task, store the files in the backend data directory and record the path information and metadata information.

[0114] (2) Automatic publishing and service generation for multimodal data

[0115] After the upload is complete, the backend program interface is intelligently invoked based on the data type to achieve automated data publishing and service generation:

[0116] 1) Creates coverage layers for multispectral, hyperspectral, infrared and other raster image data, and supports band combination configuration to generate image services that can be used for different analysis purposes (such as vegetation index, thermal anomaly detection);

[0117] 2) Publish vector feature layers for vector data such as vector routes and mission areas, supporting WFS-T transaction operations;

[0118] 3) Process and publish LiDAR point cloud and 3D model data as 3D Tiles services for efficient 3D visualization;

[0119] 4) Intelligent layer style setting (SLD) automatically adapts rendering rules according to data type. For example, multispectral data uses false color synthesis by default, and thermal infrared data uses temperature gradient color band rendering.

[0120] The automated publishing process can be formally described as follows:

[0121] (1)

[0122] Where f represents the function that generates the GeoServer layer via the REST API. The input file path is specified, CRS is the coordinate reference system, Style is the layer style, and Workspace is the workspace.

[0123] (3) Multimodal metadata management and data association

[0124] The system stores the published layer information and extracted metadata in a PostgreSQL / PostGLOS database, including:

[0125] 1) Basic layer information: layer name, data type (raster / vector), publication status (published / unpublished), service address URL;

[0126] 2) Collect metadata: UAV ID, payload type, band information, resolution, acquisition time, etc.;

[0127] 3) Task-related information: Task ID, task area, executing drone number, operator information, etc.;

[0128] 4) Supports fast retrieval and related queries based on multi-dimensional metadata, such as "query all mission data collected by UAV MH350 in multispectral mode".

[0129] (4) CRUD operations

[0130] The system provides a complete data management interface:

[0131] 1) Added: Supports batch uploading and automatic publishing of multiple data sets;

[0132] 2) Delete: Delete the database record and delete the corresponding layer via GeoServerAPI;

[0133] 3) Editing: Supports operations such as renaming layers, adjusting styles, and transforming coordinates;

[0134] 4) Query: Supports retrieval based on layer name, type, task association, or spatial location, and also supports multi-condition combination retrieval based on UAV ID, payload type, acquisition time, task area, and data mode;

[0135] CRUD operations are synchronized with GeoServer, and data status updates can be described using formulas:

[0136] (2)

[0137] in, This represents the state of the database at time t+1. The database represents the state at time t. For layer changes caused by add, delete, and modify operations, function g is the synchronization mapping function from the database state to the GeoServer service. This is the updated state of Geoserver.

[0138] (5) Multimodal data fusion visualization and linkage analysis

[0139] The front-end Cesium calls published multimodal data through standardized service interfaces (WMS, WMTS, WFS, 3D Tiles) and implements the following functions in the 3D scene:

[0140] 1) Multi-source data overlay display: Visible light orthophotos, thermal infrared distribution maps, multispectral vegetation index maps, and real-time UAV flight paths can be overlaid and displayed in the same view;

[0141] 2) Interactive query and comparison: Users can click to query the attribute information of any data, and can switch and compare data based on dimensions such as time axis, drone, and payload type;

[0142] 3) Collaborative spatial analysis: Utilize the spatial analysis and task-assisted decision-making modules to perform comprehensive analysis on multimodal data, such as locating abnormal areas based on thermal infrared data and then switching to high-resolution visible light images for detailed confirmation.

[0143] (6) Automation and real-time performance

[0144] The system of this invention supports the automatic uploading and publishing of data after task completion, without manual intervention. After new data is published, the front-end visualization scene is updated in real time, realizing a closed-loop linkage of UAV data collection, data processing, and 3D visualization, significantly improving the efficiency of multi-UAV collaborative tasks and the timeliness of data analysis.

[0145] The spatial analysis module is used to perform spatial analysis by calling map services or 3D data services based on preprocessed real-time flight data, mission metadata, and pre-saved terrain data, to assess mission feasibility, and to generate mission feasibility reports and optimized execution plans.

[0146] Specifically, this module calls map services or 3D data services to perform buffer analysis, visibility analysis, optimal path calculation, and area measurement. Buffer analysis includes calculating the safety boundary range of the flight path or mission area; visibility analysis includes assessing the monitoring coverage of the UAV in the mission area; optimal path calculation includes generating and optimizing the flight path based on 3D terrain and obstacles; and area measurement includes measuring the length and area of ​​the flight mission area.

[0147] Specifically, this module drives task decision-making logic through pre-emptive spatial analysis to determine whether a task should be executed or replanned. Based on flight data, task metadata, and pre-saved terrain data in the backend data processing module's database, it calls standardized map services or 3D data services published by the data management and publishing module to perform buffer analysis, visibility analysis, and path optimization calculations. Combining energy consumption models and safety boundary conditions, it assesses task feasibility, generates task feasibility reports and optimized execution plans (such as flight path avoidance suggestions and task adjustment instructions), and achieves comprehensive analysis of multi-UAV flight path safety, monitoring coverage, and operational efficiency. This is used to adjust flight path planning and task priorities in real time, helping users optimize flight paths, and feeding the analysis results back to the monitoring module.

[0148] The monitoring module performs multi-dimensional spatial calculations and analysis on the areas involved in UAV flight missions, improving the scientific nature of mission planning and the accuracy of flight scheduling. This module not only supports traditional functions such as buffer zones and measurement, but also introduces visual field modeling and path optimization algorithms based on 3D terrain, thereby meeting the flight planning needs in complex scenarios. Specifically, it includes:

[0149] (1) Buffer analysis: Calculate the safety boundary range of the flight path or mission area.

[0150] Buffer analysis is used to calculate the safety boundary range of a flight path or mission area, and is commonly used for obstacle avoidance and safety warning zone setting. For a flight path point set P = {p1, p2, ..., p...} n}, with each point A buffer zone is generated with the center of the circle and the flight safety radius r as the radius. The overall flight path buffer zone is then generated by merging line segment buffers.

[0151] (3)

[0152] Where (x, y) is any point in space, and n is the number of route points. Each route point Generate a circular safe zone with radius r, and combine all these circles using a union operation. These buffer zones are connected to form a continuous flight path buffer. The resulting buffer zones can be visually displayed in a 3D scene, alerting users to potential risk areas.

[0153] (2) Visibility analysis: Evaluate the surveillance coverage of the UAV in the mission area.

[0154] Field of view analysis is used to evaluate the surveillance coverage of a UAV in the mission area. Let the UAV's observation point be O(x0, y0, h0), and the field of view angle be... The maximum observation distance is Then any point Q(x, y, z) within the cone of view must satisfy:

[0155] (4)

[0156] in Let this be the drone's orientation vector. From the perspective of the viewpoint. By combining three-dimensional terrain data and occlusion calculation, the visible field of view of the UAV within the mission area can be analyzed. Specifically, the terrain model is first divided into regular grids, and the visibility of the line connecting each grid point Q(x, y, z) and the UAV observation point O(x0, y0, h0) is determined. In the occlusion judgment process, discrete sampling is performed in the direction of the line connecting O and Q, and the terrain elevation value of each sampling point is extracted from the three-dimensional terrain data. If the terrain elevation of any point is higher than the height of the line of sight at the same position, the line of sight is considered to be occluded; otherwise, the point is visible. The set of visible points that satisfy the field of view constraint formula (10) constitutes the actual visible area of ​​the UAV at a specified height. The analysis results can be used to evaluate the spatial coverage of the UAV mission and provide data support for flight path planning and mission scheduling.

[0157] (3) Optimal path calculation: Optimize the flight path based on 3D terrain and obstacles

[0158] In environments with complex terrain and obstacles, path optimization is performed based on a 3D raster map. Assuming the 3D space is discretized into a voxel mesh, the cost function for each voxel is:

[0159] (5)

[0160] Where (x,y,z) represents the coordinate position of a three-dimensional voxel, , , ) represents the weighting coefficients for different cost items. Indicates the cost of terrain elevation difference. Indicates the cost of obstacle collision. Indicates energy consumption cost. (Through three dimensions) Alternatively, Dijkstra's algorithm can be used to search for the optimal path in the cost graph, resulting in an optimized flight path that considers terrain and energy consumption. Specifically, the UAV mission area is first discretized into a three-dimensional voxel grid. Each voxel stores information such as terrain height, obstacle occupancy status, and weather data, constructing a set of cost functions to comprehensively reflect terrain undulation, obstacle distribution, and energy consumption characteristics. By setting weight parameters ω1, ω2, and ω3, the system can flexibly adjust the influence of different factors (cost terms) on path planning. Subsequently, a three-dimensional... Alternatively, Dijkstra's algorithm can be used for heuristic search in the cost graph to obtain the optimal route with the minimum cumulative cost while ensuring path feasibility. This method can effectively avoid obstacles, reduce energy consumption, and adapt to the operational needs of UAVs in complex terrain environments.

[0161] (4) Length and area calculation: Supports task area division and farmland area measurement.

[0162] The system supports length and area calculations for flight mission areas. For a spatial curve route L, its total length is:

[0163] (6)

[0164] Where t is a parameter variable used to parameterize the curve; , represents the lower and upper limits of the curve parameters, corresponding to the positions of the starting and ending points of the route in the parameter space, and (x(t),y(t),z(t)) represents the coordinate components of the curve in the three-dimensional Cartesian coordinate system.

[0165] For a polygonal task area, its area can be calculated using the formula for the area of ​​a spherical polygon:

[0166] (7)

[0167] in, Let m be the Earth's radius and m be the number of vertices of the polygon. Let be the interior angle of the j-th vertex in the polygon. This function can be used in applications such as farmland measurement and task area division.

[0168] The above analysis methods can quickly complete the setting of safety boundaries, coverage assessment, path optimization, and area calculation of the mission area, thereby improving efficiency and reducing risks in the planning and execution of UAV missions, and providing scientific basis for scenarios such as agricultural monitoring and emergency rescue.

[0169] The monitoring module is used to perform continuity verification and interval sampling of waypoints based on real-time UAV flight data, mission metadata, mission feasibility reports, and optimized execution plans, construct three-dimensional waypoint curves, and generate real-time waypoint layers.

[0170] Specifically, this module is responsible for constructing trajectories, identifying status, and tracking the mission execution process based on real-time operational data from multiple UAVs. First, based on the mission feasibility assessment report and optimized execution plan obtained from the spatial analysis module, the module adjusts the mission. If the report includes route avoidance suggestions, no-fly zone intrusion warnings, energy consumption risk assessments, etc., the module adjusts the mission status and trajectory based on these reports and real-time flight and mission data received from the backend data processing module. After adjustment, the module performs continuity verification and error removal based on the adjusted mission trajectory points. It then filters redundant points and retains key nodes from the planned mission route points, generates smooth trajectory curves based on a cubic spline interpolation model, and uses Cesium dynamic streaming rendering technology to generate visualization effects, making the flight path display more continuous and aesthetically pleasing. The module further determines the mission status (e.g., takeoff, cruise, anomaly) of multiple UAVs based on the adjusted mission data, generates mission status identifiers and alarm information, and creates structured trajectory layers and mission table information, which are pushed to the front-end display module via events to achieve a visual display of mission progress.

[0171] In practice, this module uses Cesium to display the flight paths and mission progress of multiple drones in a 3D Earth scene in real time. The system can dynamically display the start and end times, execution progress, and flight status of the mission, and continuously update the attitude parameters, speed, battery level, and other information of multiple drones on the front end. To ensure the smoothness and accuracy of the flight paths in the 3D environment, the mission waypoints are first sampled at intervals to reduce redundant data, and then a cubic spline interpolation method is used to generate continuous flight path curves to ensure the continuity of the position function and its first and second derivatives.

[0172] First, for the pre-defined flight path points of the UAV, the system reduces redundant point sets through interval sampling, improving the efficiency of subsequent interpolation and rendering. Let the flight path point set be:

[0173] (8)

[0174] The system then operates at fixed intervals. The waypoints are sampled to generate an optimized waypoint sequence.

[0175] Secondly, cubic spline interpolation is used to smooth the sampled route points to ensure the continuity of the position coordinates with the first and second derivatives. The cubic spline curve satisfies the following conditions:

[0176] (9)

[0177] Where x is the sampling number of the route point. Let the i-th spline function be represented in three-dimensional space. This represents the coordinates of the flight path at that location. (Coefficient) , , , Determined by node positions and continuity conditions, the flight path ensures a smooth transition in space. The interpolated flight path is rendered as a smooth curve in Cesium, avoiding jagged jumps and making the spatial display of the flight path more consistent with actual flight conditions.

[0178] To enhance the visualization, this invention overlays flowing light effects along the flight path. This method is based on a custom Cesium material and utilizes a time parameter t to control the dynamic offset of the texture coordinates.

[0179] (10)

[0180] Where u and v are the original texture coordinates. and These are the texture coordinates after time offset, where V represents the flow velocity and t is the time parameter in seconds. Periodic offsets are used to achieve a dynamic flowing light effect along the flight path, allowing users to intuitively perceive the drone's mission progress.

[0181] In addition, the system provides a tabular display of flight missions, including fields such as mission progress percentage, operators, and planned and actual execution times, and links these fields with the flight status in the 3D scene. The executing flight path and the drone's position are highlighted in the 3D environment, allowing users to grasp the overall and local execution status of the mission in real time, thereby achieving intuitive monitoring and scheduling management of drone missions.

[0182] The video stream synchronization module is used to receive flight status data and video stream data, and to stitch and fuse the UAV video frames using a front-end lightweight algorithm.

[0183] The video stream synchronization module is specifically used for:

[0184] For two adjacent video frames, a preset algorithm is used to detect feature points;

[0185] Feature matching is performed on feature points of two adjacent video frames;

[0186] A homography matrix is ​​constructed using a random sampling consistency method to eliminate erroneous matching points;

[0187] By using the homography matrix, the coordinate system of the next frame image is mapped to that of the previous frame image. A weighted fusion algorithm is then used to smooth the overlapping areas to obtain the stitched image.

[0188] The stitched images are projected onto the Cesium 3D Earth scene based on the UAV's position information and attitude parameters, forming a real-time video overlay layer synchronized with the flight path and mission status.

[0189] For specific implementation details, please refer to [link / reference]. Figure 3 The implementation process of the video stream synchronization module is as follows:

[0190] (1) ORB feature detection and descriptor generation

[0191] For two adjacent video frames, the ORB algorithm is first used to detect feature points. The ORB algorithm introduces orientation information based on FAST corner detection, ensuring the stability of features under rotation. Let the image grayscale function be... The response at the corner is determined by the intensity change. ORB calculates the principal direction θ for each corner:

[0192] (11)

[0193] Where P is the pixel neighborhood centered at the corner point. Subsequently, a binary feature descriptor is generated in the direction θ using the BRIEF descriptor.

[0194] (2) Feature matching (Brute-Force Matcher)

[0195] For two frames I1 and I2, the extracted ORB descriptor subsets are as follows: , Calculate the Hamming distance between descriptors using a brute-force matcher:

[0196] (12)

[0197] in and This indicates a pair of matched descriptors in the set. and For the first in the descriptor Bit binary value, This represents the XOR operation, which selects the point pair with the smallest distance as the candidate match.

[0198] (3) RANSAC algorithm for estimating homography matrix H

[0199] To eliminate erroneous matching pairs, RANSAC (Random Sample Consensus) is used to iteratively estimate the homography matrix. The homography matrix satisfies:

[0200] (13)

[0201] Where (x, y) are the coordinates of the feature points in the previous frame image, , Let ) represent the coordinates of the matching point in the next frame, s be the scale factor, and H be the homography matrix. The elements in the homography matrix represent the linear transformation parameters. RANSAC calculates H by repeatedly sampling the minimum point set and counts the number of interior points to select the optimal model.

[0202] (4) Image perspective transformation and splicing fusion

[0203] The homography matrix H maps the subsequent frame image I2 to the coordinate system of the previous frame image I1:

[0204] (14)

[0205] in =[x,y,1]T, = , ,1]T, This is the homogeneous coordinate vector of the key points or pixels in the previous frame image I1. This maps the next frame image I2 to the homogeneous coordinate system of I1. After mapping, the system uses a weighted fusion algorithm to smooth the overlapping areas, for example:

[0206] (15)

[0207] in For weight fusion.

[0208] (5) Three-dimensional projection and linkage of the splicing result

[0209] The stitched imagery is projected onto the Cesium 3D Earth scene based on the drone's position and attitude parameters, forming a real-time video overlay synchronized with the flight path and mission status. Users can not only view the drone's real-time flight video but also intuitively observe the stitched wide-area imagery, achieving multi-dimensional interactive display of the flight path and video footage.

[0210] In summary, this module proposes a browser-based method for real-time video frame stitching and synchronized 3D visualization. It receives real-time flight data from the backend data processing module, including real-time transmitted video stream data and synchronously acquired flight status data (latitude, longitude, altitude, attitude, speed, and timestamps). Spatiotemporal alignment of video data and UAV flight data is achieved through timestamp matching and attitude angle correction mechanisms. The module lightweightly integrates algorithms such as ORB feature detection, Brute-Force matching, and RANSAC homography estimation into the front-end environment. It utilizes the multi-threaded separation mechanism of WebAssembly and WebWorker to improve the efficiency of feature extraction and matching operations, enabling real-time stitching and quick view generation of UAV video streams without server-side computation. To ensure real-time performance and robustness, the module optimizes feature matching accuracy in the front-end environment through adaptive threshold matching and intelligent fault-tolerant processing strategies. The stitched quick view image generated based on the above collaborative algorithms is projected onto the Cesium 3D scene surface in the front-end framework using a perspective transformation algorithm. Combined with UAV trajectory information such as latitude, longitude, altitude, and attitude from the flight status data, dynamic consistency visualization of the image and spatial location is achieved. This module also works in conjunction with the data management and publishing module to automatically trigger image stitching archiving and service publishing tasks, forming a closed-loop data flow of "acquisition - stitching - service publishing".

[0211] The front-end display module is used to build a 3D geographic scene based on the Cesium engine and comprehensively display the analysis results of each module.

[0212] Specifically, this module is based on the Cesium 3D Earth engine and comprehensively displays data and analysis results from the backend data processing module and the video stream synchronization module. This includes the real-time flight status of multiple UAVs from the backend data processing module, the multi-source remote sensing data published by the data management and publishing module, the spatial analysis results from the spatial analysis module, the task information from the monitoring module, and the video stream stitching results completed by the video stream synchronization module.

[0213] In the specific implementation, Cesium was used for 3D Earth visualization, and Vite and Vue were used for front-end page development. Users can view real-time flight data, flight paths, and mission status of multiple drones through the web interface. The front-end interface supports dynamic updates, providing real-time feedback on drone flight data and displaying the drone's flight status on a 3D map. Users can view flight paths from different perspectives on the Cesium map and interact with the system, such as adjusting flight paths and monitoring mission progress. Users can click on the flight path in the 3D globe to view drone mission details and flight status, and watch real-time flight videos captured by the drones.

[0214] Based on a multi-data rendering pipeline, the module supports the dynamic display of the aforementioned UAV entity trajectory, flight direction, UAV attitude, and status indicators, and enables linked annotation with mission information. Simultaneously, it renders the analysis results output by the spatial analysis module, including buffer zones, safe flight corridors, visible area volume, and path optimization results. These results are presented through semi-transparent area overlays, color coding, luminous flight paths, risk icons, and clickable pop-up prompts, allowing users to directly observe strategy analysis results and flight interference factors within the scene. The front-end display framework also serves as a feedback mechanism for system interaction commands. Users can perform operations such as task selection, flight path start / stop, layer switching, spatial queries, and risk alarm confirmation in the 3D interface. The front-end transmits these interaction commands through a communication interface established with the back-end data processing module, and the back-end data processing module can synchronously notify the monitoring and spatial analysis modules. This allows the system to update the rendered content and mission view based on user operations, forming a linked mechanism of "front-end operation—back-end response—3D scene update."

[0215] In summary, this invention establishes a multi-UAV remote sensing closed-loop collaborative system encompassing "acquisition—fusion—analysis—display—return control." Acquisition involves accessing multi-source flight and payload sensing data; fusion involves real-time backend analysis and automated service publishing; analysis involves mission risk assessment and spatial analysis and deduction; display involves synchronous rendering of 3D flight paths, stitched images, and remote sensing scenes; and return control involves interactive task adjustment commands based on analysis results. This forms a full-link intelligent monitoring system with autonomous analysis and dynamic task optimization capabilities. The system supports reliable synchronization of asynchronous data streams from multiple UAVs in a real-time communication link, achieves field-of-view analysis, path evaluation, and dynamic adjustment of task priorities in the spatial analysis link, and realizes video stitching and image geographic alignment rendering on the browser side. This overcomes the technical bottlenecks of fragmented data processing, difficulties in spatiotemporal alignment, and insufficient 3D display capabilities in existing UAV remote sensing systems, offering advantages such as strong real-time performance, high fusion degree, and convenient deployment.

[0216] Example 2

[0217] Based on the same inventive concept, this embodiment discloses a Cesium-based multi-machine multimodal remote sensing monitoring and 3D visualization method, including:

[0218] S1: Use the data acquisition and communication module to collect real-time flight data, multimodal remote sensing data, and mission metadata of the UAV. The real-time flight data of the UAV includes flight status data and video stream data.

[0219] S2: The backend data processing module preprocesses the real-time flight data, multimodal remote sensing data, and mission metadata of the UAV, and provides a data interface;

[0220] S3: Utilize the data management and publishing module to process multimodal remote sensing data based on the data interface, generate corresponding map services and 3D data services according to the data type, and publish them;

[0221] S4: Using the spatial analysis module, based on the pre-processed real-time flight data, mission metadata, and pre-saved terrain data, call map services or 3D data services to perform spatial analysis, assess mission feasibility, and generate a mission feasibility report and optimized execution plan.

[0222] S5: Using the monitoring module, based on the UAV's real-time flight data, mission metadata, mission feasibility report, and optimized execution plan, the track points are continuously verified and sampled at intervals to construct a three-dimensional track curve and generate a real-time track layer.

[0223] S6: Utilizes the video stream synchronization module to receive flight status data and video stream data, and uses a front-end lightweight algorithm to stitch and fuse the UAV video frames;

[0224] S7: Utilize the front-end display module to construct a 3D geographic scene based on the Cesium engine and comprehensively display the analysis results of each module.

[0225] Since the method described in Embodiment 2 of this invention is based on the Cesium-based multi-machine multimodal remote sensing monitoring and 3D visualization system in Embodiment 1 of this invention, those skilled in the art can understand the specific implementation process of this method based on the system described in Embodiment 1 of this invention, and therefore will not be repeated here. All methods implemented based on the system in Embodiment 1 of this invention fall within the scope of protection of this invention.

[0226] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0227] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0228] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various modifications and variations to the embodiments of the invention without departing from the spirit and scope of the invention. Thus, if these modifications and variations of the embodiments of the invention fall within the scope of the claims of the invention and their equivalents, the invention also intends to include these modifications and variations.

Claims

1. A Cesium-based multi-machine, multi-modal remote sensing monitoring and 3D visualization system, characterized in that, include: The data acquisition and communication module is used to collect real-time flight data and mission metadata of the UAV and acquire multimodal remote sensing data. The real-time flight data of the UAV includes flight status data and video stream data. The backend data processing module is used to preprocess real-time flight data, multimodal remote sensing data, and mission metadata of UAVs, and provides data interfaces; The data management and publishing module is used to process multimodal remote sensing data based on the data interface, generate corresponding map services and 3D data services according to the data type, and publish them. The spatial analysis module is used to perform spatial analysis by calling map services or 3D data services based on preprocessed real-time flight data, mission metadata and pre-saved terrain data, to assess mission feasibility and generate mission feasibility reports and optimized execution plans. The monitoring module is used to perform continuity verification and interval sampling of waypoints based on real-time UAV flight data, mission metadata, mission feasibility reports, and optimized execution plans, construct three-dimensional waypoint curves, and generate real-time waypoint layers. The video stream synchronization module is used to receive flight status data and video stream data, and to stitch and fuse the UAV video frames through a front-end lightweight algorithm; The front-end display module is used to build a 3D geographic scene based on the Cesium engine and comprehensively display the analysis results of each module; Specifically, the spatial analysis module is used for: Call map services or 3D data services to perform buffer analysis, visibility analysis, optimal path calculation and area measurement. Among them, buffer analysis includes calculating the safety boundary range of the flight path or mission area, visibility analysis includes assessing the monitoring coverage range of the UAV in the mission area, optimal path calculation includes generating and optimizing the flight path based on 3D terrain and obstacles, and area measurement includes measuring the length and area of ​​the flight mission area. Assess the surveillance coverage of the drone in the mission area, including: The terrain model is divided into regular grids. Visibility is determined for the line connecting each grid point Q and the UAV observation point O. Specifically, discrete sampling is performed along the line connecting O and Q, and the terrain elevation value of each sampling point is extracted from the 3D terrain data. If the terrain elevation of any sampling point is higher than the height of the line of sight at the same location, the line of sight is considered to be blocked; otherwise, the grid point is visible. Among all visible points, those that satisfy the field-of-view constraint constitute the actual visible area of ​​the UAV at a specified altitude. The field-of-view constraint is: in, For the field of view, For maximum observation distance, From the perspective of This is the drone's orientation vector; The video stream synchronization module is specifically used for: For two adjacent video frames, a preset algorithm is used to detect feature points; Feature matching is performed on feature points of two adjacent video frames; A homography matrix is ​​constructed using a random sampling consistency method to eliminate erroneous matching points; By using the homography matrix, the coordinate system of the next frame image is mapped to that of the previous frame image. A weighted fusion algorithm is then used to smooth the overlapping areas to obtain the stitched image. The stitched images are projected onto the Cesium 3D Earth scene based on the UAV's position information and attitude parameters, forming a real-time video overlay layer synchronized with the flight path and mission status.

2. The Cesium-based multi-machine multimodal remote sensing monitoring and 3D visualization system as described in claim 1, characterized in that, The data acquisition and communication module is also used for: A thematic data distribution strategy is introduced, and independent message channels are established according to the drone number and data type to transmit data from multiple drones in parallel. The collected data undergoes preliminary analysis, time alignment, and compression before being pushed to the backend data processing module.

3. The Cesium-based multi-machine multimodal remote sensing monitoring and 3D visualization system as described in claim 1, characterized in that, The backend data processing module includes: The preprocessing unit is used to clean, standardize, and perform temporal and spatial correction on real-time flight data, multimodal remote sensing data, and mission metadata. The association unit establishes a relationship between flight status data and mission metadata in real-time flight data and then stores them in the database in a unified manner. The data forwarding and synchronization management unit is used to establish a bridge channel with the MQTT message queue and push processed data in conjunction with WebSocket. The interface support unit provides interface support for the automated publishing of multimodal remote sensing data.

4. The Cesium-based multi-machine multimodal remote sensing monitoring and 3D visualization system as described in claim 1, characterized in that, The data management and publishing module is also used for: Automatic generation of time-series indexes and pyramid slice structures for multi-temporal UAV imagery; An interface was established with the video stream synchronization module to enable the real-time generated stitched images to be mounted to the GeoServer server.

5. The Cesium-based multi-machine multimodal remote sensing monitoring and 3D visualization system as described in claim 1, characterized in that, Calculate the safety boundary range of the flight path or mission area, specifically as follows: Where (x, y) are the coordinates of any point in space, and n is the number of route points, with the i-th route point as the starting point. Generate a circular safe zone with radius r centered at the center. All circular safe zones are then processed by union operation. Connect them to form a continuous route buffer zone. , , For waypoints The coordinates.

6. The Cesium-based multi-machine multimodal remote sensing monitoring and 3D visualization system as described in claim 5, characterized in that, Optimized flight paths are generated based on 3D terrain and obstacles, including: The UAV mission area is discretized into a three-dimensional voxel grid, where each voxel stores terrain height, obstacle occupancy status, and meteorological data, forming a set of cost functions to comprehensively reflect terrain characteristics; The impact of different factors on path planning is adjusted by setting weight parameters. These factors include terrain elevation difference cost, obstacle collision cost, and energy consumption cost. Using three-dimensional Alternatively, the Dijkstra algorithm can perform a heuristic search in the cost graph to obtain the route with the minimum cumulative cost, which can then be used as the optimized route.

7. A Cesium-based multi-machine multimodal remote sensing monitoring and 3D visualization method, characterized in that, The system implementation based on claim 1 includes: The data acquisition and communication module is used to collect real-time flight data and mission metadata of the UAV to obtain multimodal remote sensing data. The real-time flight data of the UAV includes flight status data and video stream data. The backend data processing module is used to preprocess real-time flight data, multimodal remote sensing data, and mission metadata of UAVs, and provides a data interface. The data management and publishing module processes multimodal remote sensing data based on the data interface, generates corresponding map services and 3D data services according to the data type, and publishes them. The spatial analysis module utilizes preprocessed real-time flight data, mission metadata, and pre-saved terrain data to call map services or 3D data services for spatial analysis, assess mission feasibility, and generate mission feasibility reports and optimized execution plans. The monitoring module uses real-time UAV flight data, mission metadata, mission feasibility reports, and optimized execution plans to perform continuity verification and interval sampling of waypoints, construct a three-dimensional waypoint curve, and generate a real-time waypoint layer. The video stream synchronization module receives flight status data and video stream data, and the drone video frames are stitched and fused using a front-end lightweight algorithm; The front-end display module is used to construct a 3D geographic scene based on the Cesium engine and comprehensively display the analysis results of each module.

Images