Data management system based on aerial photogrammetry

Abstract

The invention discloses a data management system based on aerial photogrammetry, and relates to the technical field of aerial photogrammetry and geographic information, and the system comprises a multi-source data fusion access module which is used for receiving heterogeneous data generated by various sensors in aerial photogrammetry operation; the distributed parallel processing engine is used for realizing parallelization and streamline of the whole aerial survey data processing flow by adopting a micro-service architecture; the intelligent quality control and optimization module is constructed based on a machine learning algorithm, continuously monitors the output quality of each processing node in the distributed parallel processing engine, and generates a quality evaluation report and an optimization adjustment suggestion; the elastic resource scheduling platform dynamically allocates hardware resources according to the real-time task load; and the full-process visual monitoring module provides full-chain visual display and interaction control functions from data acquisition to product distribution. The invention aims to solve the problems of high data isomerism, long processing period and lack of flexibility in system resource allocation.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of aerial photogrammetry and geographic information technology, and in particular to a data management system based on aerial photogrammetry. BACKGROUND

[0002] In the field of spatial information technology, aerial photogrammetry, as a key means of obtaining high-precision geographic spatial data, has been widely applied in urban planning, resource investigation and environmental monitoring, and other aspects. This technology collects surface images through sensors mounted on aerial platforms, and generates diversified geographic information products such as digital elevation models, orthophoto maps and three-dimensional models based on photogrammetry principles.

[0003] Among them, the data management system based on aerial photogrammetry undertakes the core task of efficiently organizing, storing, processing and distributing massive aerial survey data. Its basic goal is to ensure the integrity and reliability of the entire data processing chain from raw images to final products through systematic data management processes, and to provide high-quality data support for various geographic spatial applications.

[0004] In the prior art, the data management system of aerial photogrammetry generally faces the severe challenges brought by strong data heterogeneity and complex processing flow. The system needs to manage image data, flight attitude parameters, ground control point information and intermediate processing results obtained by multiple source sensors, but the existing architecture cannot realize the unified organization and efficient indexing of these heterogeneous data. In the data processing link, the traditional system adopts a serial processing mode, which leads to a long conversion period from raw images to final products, and cannot meet the timeliness requirements in emergency mapping and dynamic monitoring scenarios. In addition, the existing system lacks flexibility in storage and computing resource allocation for massive aerial survey data, and is prone to performance bottlenecks when dealing with large-scale aerial photography projects, and the data isolation and security management mechanism between different aerial photography tasks is not perfect. SUMMARY

[0005] The purpose of the present application is to provide a data management system based on aerial photogrammetry to solve the problems of strong data heterogeneity leading to integration difficulties, serial processing mode causing long conversion period, and lack of flexibility in system resource allocation causing performance bottlenecks in the prior art.

[0006] To solve the above technical problems, the present application provides the following technical solutions: A data management system based on aerial photogrammetry, comprising: a multi-source data fusion access module, a distributed parallel processing engine, an intelligent quality control and optimization module, an elastic resource scheduling platform, and a full-process visual monitoring module.

[0007] The multi-source data fusion access module is responsible for receiving heterogeneous data generated by various sensors in aerial photogrammetry operations, performing data parsing, format unification, and spatio-temporal reference alignment operations. It has a data integrity verification unit and a multi-dimensional index construction unit inside to ensure that the original aerial survey data is standardized before being stored and an efficient retrieval mechanism is established.

[0008] The distributed parallel processing engine uses a micro-service architecture to achieve parallelization and pipelining of the entire aerial survey data processing flow. It includes a task decomposition sub-engine and a process coordination sub-engine, which work together through a unified message communication mechanism and state management service to achieve efficient scheduling and fault-tolerant execution of processing tasks, significantly shortening the data processing cycle from raw images to final products.

[0009] The intelligent quality control and optimization module is based on machine learning algorithms and continuously monitors the output quality of each processing node in the distributed parallel processing engine, automatically detects data anomalies and precision deviations, and generates quality evaluation reports and optimization adjustment suggestions. The module integrates a quality indicator calculation unit and an adaptive parameter adjustment unit, which can dynamically optimize processing algorithm parameters and quality control thresholds based on data characteristics.

[0010] The elastic resource scheduling platform is responsible for managing computing, storage, and network resources in the system, dynamically allocating hardware resources based on real-time task load. It implements a resource monitoring sub-platform and a strategy execution sub-platform, ensuring stable performance in high-concurrency task scenarios through predictive scaling mechanisms.

[0011] The full-process visual monitoring module provides visual display and interactive control functions from data acquisition to product distribution, supporting real-time tracking of processing progress, dynamic display of quality indicators, and immediate push of abnormal alarms. Its rendering engine uses hierarchical progression and level-of-detail techniques to ensure smooth visual experience in large-scale aerial survey data scenarios.

[0012] Further, the specific implementation of the multi-source data fusion access module is as follows. The module receives raw data collected by various sensors such as aerial cameras, laser radars, global navigation satellite system receivers, and inertial measurement units through a high-speed data interface. The data analysis unit uses special analysis algorithms for different sensor data formats. The aerial image data analysis extracts camera parameters and image metadata, the laser radar data analysis extracts point cloud coordinates and intensity information, the global navigation satellite system data analysis extracts positioning coordinates and timestamps, and the inertial measurement unit data analysis extracts attitude angles and acceleration values. The format unification unit converts the parsed heterogeneous data into a standardized data exchange format and adds a unified space-time reference system identifier to all data. The space-time reference alignment unit unifies the spatial coordinates of all sensor data to the national geodetic coordinate system based on precise point positioning technology and coordinate conversion models, and unifies the time information to coordinated universal time. The data integrity verification unit verifies that there is no loss or damage to the data during transmission by calculating the data packet integrity index and checksum. The multi-dimensional index construction unit establishes spatial index, time index, and sensor type index for all data based on space-filling curve and time segmentation technology, supporting multi-dimensional conditional combination query.

[0013] Further, the specific implementation of the distributed parallel processing engine is as follows. The task decomposition sub-engine decomposes the aerial survey data processing flow into multiple independent and parallel executable atomic tasks, including aerial triangulation tasks, dense matching tasks, digital elevation model generation tasks, orthographic correction tasks, and three-dimensional modeling tasks. Each atomic task is encapsulated as a standardized microservice and managed through a service registry center. The flow coordination sub-engine builds a processing flow dependency relationship based on a directed acyclic graph and executes atomic tasks in the order of dependency through a workflow engine. The message communication mechanism uses a high-reliability message queue to realize asynchronous communication between microservices, ensuring reliable transmission of task status and processing results. The state management service records the execution status, progress percentage, and resource consumption of each processing task, supporting task breakpoint resume and failure retry. The fault-tolerant execution mechanism realizes task migration to a healthy node for re-execution when an abnormal processing node is detected through task replication and heartbeat detection.

[0014] Furthermore, the specific implementation of the intelligent quality control and optimization module is as follows: The machine learning algorithm employs an ensemble learning framework, integrating multiple base learners to comprehensively evaluate the quality of aerial survey data. The quality index calculation unit calculates key quality indicators in real time, such as aerial triangulation accuracy, image matching success rate, digital elevation model accuracy, and 3D model integrity. Data anomaly detection, based on the isolated forest algorithm and the local anomaly factor algorithm, automatically identifies outliers and gross errors in the aerial survey data. Accuracy deviation analysis calculates the statistical characteristics of planar accuracy and elevation accuracy by comparing the processed results with high-precision reference data. The quality assessment report generation unit integrates various quality indicators and anomaly detection results to generate a detailed report including quality level assessment and problem localization. The adaptive parameter adjustment unit dynamically optimizes the processing algorithm parameters based on the quality assessment results. When the image matching success rate is detected to be below 95%, the matching window size and similarity threshold are automatically adjusted; when the digital elevation model accuracy deviation exceeds the limit, the filtering parameters and interpolation algorithm are automatically optimized.

[0015] Furthermore, the specific implementation of the elastic resource scheduling platform is as follows: The resource monitoring sub-platform collects real-time resource indicators such as CPU utilization, memory usage, disk I / O performance, and network bandwidth utilization of each computing node. The policy execution sub-platform dynamically adjusts resource allocation strategies based on monitoring data and task queue status. The predictive scaling mechanism, based on time series analysis and machine learning prediction models, predicts changes in resource demand over a future period. When it is predicted that the task load will increase by more than 50% within one hour, the platform automatically initiates the resource scaling process, pre-allocating additional computing nodes to join the cluster. When it is detected that resource utilization has remained below 30% for more than two hours, the platform automatically performs resource scaling down, releasing idle nodes to conserve resources. The resource isolation mechanism uses containerization technology to achieve resource and security isolation between different aerial photography tasks, ensuring the service quality of critical tasks.

[0016] Furthermore, the specific implementation of the full-process visualization monitoring module is as follows. This module uses a browser and server architecture to provide visualization services. The front-end uses vector slicing technology and a spatial data engine to achieve smooth rendering of large-scale aerial survey data. The real-time processing progress tracking function displays the current status and completion percentage of each processing task through progress bars and Gantt charts, supporting drill-down to view detailed execution logs of any task. The dynamic quality indicator display function visualizes key quality indicators in dashboard form, supporting historical trend comparison and threshold alarms. The instant anomaly alarm push function pushes system and quality anomalies to relevant personnel in real time through multiple channels such as message centers and email notifications. The hierarchical progressive technology dynamically loads data of different levels of detail according to the view zoom level, displaying an overview of task distribution in the global view and detailed processing parameters and quality data in the local view. The level of detail technology optimizes the rendering performance of large-scale 3D scenes through frustum culling and level of detail model switching.

[0017] Compared with the prior art, the beneficial technical effects of the present invention are as follows: This invention achieves a comprehensive improvement in aerial survey data management capabilities by constructing a multi-source data fusion access module and a distributed parallel processing engine. The system can efficiently integrate multi-source heterogeneous sensor data and establish a unified indexing mechanism. At the same time, through microservice architecture and parallel processing technology, the traditional serial processing mode is transformed into a high-efficiency pipeline, significantly shortening the data processing cycle.

[0018] This invention achieves automated monitoring and optimization of processing quality through an intelligent quality control and optimization module based on machine learning algorithms. It can promptly detect data anomalies and accuracy deviations and automatically adjust processing parameters, significantly improving the quality stability and reliability of aerial survey products.

[0019] This invention achieves intelligent allocation and efficient utilization of system resources through a flexible resource scheduling platform using predictive scaling and resource isolation technology, ensuring stable system performance and secure isolation between different tasks under high load scenarios. The full-process visualization monitoring module provides a comprehensive monitoring perspective from macro to micro through hierarchical and detailed technologies, enabling managers to grasp the operational status and quality of the entire data processing chain in real time. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the data management system architecture based on aerial photogrammetry proposed in this invention; Figure 2 This is a schematic diagram of the core principle framework of the distributed parallel processing engine proposed in this invention. Detailed Implementation

[0021] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present invention and not to limit the present invention. For those skilled in the art, the present invention can be practiced without some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present invention by illustrating examples of the invention.

[0022] Example 1 Please refer to the attached document. Figure 1 This embodiment details the specific technical implementation of a data management system based on aerial photogrammetry. The system aims to solve the technical challenges of existing technologies, such as difficulties in data integration due to strong data heterogeneity, excessively long conversion cycles caused by serial processing, and performance bottlenecks resulting from a lack of flexible system resource allocation. The core system architecture consists of five modules: a multi-source data fusion and access module, a distributed parallel processing engine, an intelligent quality control and optimization module, an elastic resource scheduling platform, and a full-process visual monitoring module. These modules work together efficiently through strictly defined data interfaces and communication protocols.

[0023] As the system's data entry point, the multi-source data fusion access module undertakes the key tasks of receiving heterogeneous data generated by various sensors in aerial photogrammetry operations, performing data parsing, format unification, and spatiotemporal benchmark alignment.

[0024] This module receives raw data streams from various sensors, such as aerial cameras, lidar, global navigation satellite system receivers, and inertial measurement units, via a high-speed data interface. The data parsing unit is the primary processing stage of this module, and it uses dedicated parsing algorithms to process data from different sensors.

[0025] For aerial imagery data, the parsing algorithm extracts interior and exterior orientation parameters of the camera, including focal length, principal point coordinates, spatial coordinates of the shooting center, and attitude angle parameters. At the same time, it extracts metadata information such as image acquisition time, resolution, and depth.

[0026] For lidar data, the analytical algorithm extracts three-dimensional point cloud coordinates and echo intensity information from the original waveform data. The point cloud coordinates include the geodetic coordinates of each laser footprint, while the intensity information reflects the reflection characteristics of ground objects to the laser signal.

[0027] For Global Navigation Satellite System (GNSS) data, the analytical algorithm extracts carrier phase and pseudorange observations from the raw observation data, and calculates the precise coordinates of the photography center and timestamp information using precise point positioning technology.

[0028] For inertial measurement unit data, the analytical algorithm extracts the three attitude angle parameters (roll angle, pitch angle, and yaw angle) and the three-axis acceleration values ​​of the flight platform from the raw gyroscope and accelerometer outputs.

[0029] The format unification unit operates immediately following the data parsing unit. Its core function is to convert the parsed heterogeneous data into a standardized data exchange format. All sensor data is converted into a hybrid format based on binary large objects and Extensible Markup Language (XML). Binary large objects store raw observation data and large-scale point cloud data, while XML stores structured metadata and parameter information. Each successfully converted data record is automatically appended with a unified spatiotemporal reference system identifier, using the spatial reference system identifier format defined by the International Organization for Standardization (ISO), ensuring a consistent reference benchmark for subsequent data processing. The spatiotemporal reference alignment unit, based on precise point positioning technology and a seven-parameter coordinate transformation model, unifies the spatial coordinates of all sensor data to the national 2000 geodetic coordinate system. Precise point positioning technology utilizes Global Navigation Satellite System (GNSS) observation data and precise ephemeris products to calculate the coordinates of the photography center to centimeter-level accuracy. The seven-parameter coordinate transformation model uses three translation parameters, three rotation parameters, and one scale parameter to transform the lidar point cloud coordinates and image exterior orientation elements from the original coordinate system to the target coordinate system. Time information is entirely unified to Coordinated Universal Time (UTC) format and synchronized at the millisecond level through a precise time protocol. The data integrity verification unit runs throughout the entire converged access process. It ensures no data loss or corruption during transmission by calculating data packet integrity metrics and performing cyclic redundancy checksum verification. The data packet integrity metric calculates the percentage of valid data fields in each data packet; any percentage below 99.5% triggers a data retransmission mechanism. Cyclic redundancy checksum verification calculates a 32-bit checksum for each data block and compares it with the checksum calculated at the transmitting end; inconsistencies indicate data corruption. The multi-dimensional index building unit, as the module's output, establishes an efficient retrieval mechanism for all imported data based on spatial filling curves and time segmentation technology. The spatial index uses a hierarchical quadtree structure, dividing the entire survey area into multiple levels of geographic grids. Each grid records pointers to all data records within that area. The time index is segmented according to data acquisition time, supporting fast data retrieval by time range. The sensor type index is categorized according to four sensor types: aerial cameras, lidar, GPS receivers, and inertial measurement units. These three indexes provide services through a combined query interface, supporting conditional queries of any dimension, with a typical query response time controlled within 200 milliseconds.

[0030] Please refer to the attached document. Figure 2The distributed parallel processing engine is the core of the entire system's computation, employing a microservice architecture to achieve parallelization and pipelined processing of the entire aerial survey data process. This engine consists of five core sub-engines: a task decomposition sub-engine, a process coordination sub-engine, a message communication mechanism, a state management service, and a fault-tolerant execution mechanism. The task decomposition sub-engine breaks down the traditional serial aerial survey data processing flow into multiple independent, parallelizable atomic tasks. These atomic tasks include aerial triangulation, dense matching, digital elevation model (DEM) generation, orthorectification, and 3D modeling. The aerial triangulation task is responsible for solving high-precision exterior orientation elements of all images through image connection point matching and adjustment calculations; the dense matching task generates high-density 3D point clouds based on multi-view images; the DEM generation task extracts terrain elevation information from point cloud data and generates a regular grid elevation model; the orthorectification task uses the DEM and image exterior orientation elements to eliminate projection differences in images and generate orthorectified images; and the 3D modeling task generates a 3D reality model through point cloud segmentation and surface reconstruction algorithms. Each atomic task is encapsulated as a standardized microservice, containing complete input / output interface definitions, processing logic implementation, and performance monitoring metrics. All microservices are managed uniformly through a service registry, which records the network address, version number, load status, and health status of each microservice. The workflow coordination sub-engine constructs the dependencies of the processing flow based on a directed acyclic graph (DAG), driving the atomic tasks to execute in the correct order through a workflow engine. Nodes in the DAG represent atomic tasks, and edges represent dependencies between tasks. For example, a dense matching task must wait for the aerial triangulation task to complete before it can begin, as dense matching requires precise image exterior orientation elements as input. The workflow engine parses the DAG structure, dynamically schedules the execution order of each microservice, and manages data transfer between tasks. The message communication mechanism uses a highly reliable message queue to achieve asynchronous communication between microservices, ensuring reliable transmission of task status and processing results. Each microservice publishes its processing results to a specified message topic, and downstream microservices subscribe to relevant topics to obtain input data. The message queue implements at-least-once semantics, ensuring that messages are not lost even in the event of network fluctuations or service restarts. The state management service records the execution status, progress percentage, and resource consumption of each processing task. Execution status includes four states: waiting, executing, completed, and failed. Progress percentage is updated in 5% increments. Resource consumption records metrics such as CPU time, memory usage, and disk I / O. The state management service supports task resumption from interrupted points, allowing execution to resume from the most recent state checkpoint if a task is interrupted, avoiding redundant calculations. Fault-tolerant execution mechanisms achieve high availability through task replicas and heartbeat detection. Each running task maintains a hot backup copy on another computing node; if the primary node fails, the backup copy can take over and continue execution within 300 milliseconds.The heartbeat detection mechanism sends a health check request to all tasks every five seconds. Any task that does not respond within fifteen seconds will be marked as abnormal and reassigned by the scheduler to a healthy node for execution.

[0031] The intelligent quality control and optimization module is responsible for continuously monitoring the output quality of each processing node in the distributed parallel processing engine, automatically detecting data anomalies and accuracy deviations, and generating quality assessment reports and optimization suggestions. The core of this module is a quality assessment and optimization system built on machine learning algorithms. The machine learning algorithms employ an ensemble learning framework, integrating multiple base learners to comprehensively evaluate the quality of aerial survey data. The ensemble learning framework combines the prediction results of multiple base learners through a weighted voting mechanism. The base learners include three algorithms: Support Vector Machine (SVM), Random Forest, and Gradient Boosting Decision Tree. SVM excels at handling nonlinear classification problems in high-dimensional feature spaces; Random Forest, by constructing multiple decision trees and integrating their prediction results, has excellent anti-overfitting capabilities; Gradient Boosting Decision Tree iteratively trains a series of weak learners to gradually improve the model's prediction accuracy. These three base learners are trained using different feature subsets, and the final comprehensive quality score is obtained by weighted averaging of their output probabilities. The quality index calculation unit calculates key quality indicators in real time, such as aerial triangulation accuracy, image matching success rate, digital elevation model accuracy, and 3D model integrity. Aerial triangulation accuracy is calculated through checkpoint residual statistics, including two indicators: horizontal and vertical mean square errors. Image matching success rate is calculated as the proportion of successfully matched feature points out of the total number of feature points. Digital elevation model accuracy is calculated by comparing the vertical mean square error with lidar checkpoints. 3D model integrity is assessed by evaluating the number and area ratio of holes on the model surface. Data anomaly detection is based on the isolated forest algorithm and the local anomaly factor algorithm to automatically identify outliers and gross errors in aerial survey data. The isolated forest algorithm constructs multiple isolated trees by randomly selecting features and segmentation values; any easily isolated sample point is judged as an anomaly. The local anomaly factor algorithm calculates the local density deviation between each sample point and its neighbors; points with significantly lower density than their neighbors are marked as anomalies. These two algorithms run in parallel, and any data point judged as anomaly by either algorithm is entered into a manual review queue. Accuracy deviation analysis calculates the statistical characteristics of horizontal and vertical accuracy by comparing the processed results with high-precision reference data. The reference data typically comes from ground survey control points or high-precision lidar data; the system calculates the mean, standard deviation, root mean square error, and maximum deviation between the processed results and the reference data. The quality assessment report generation unit integrates various quality indicators and anomaly detection results to generate a detailed report including quality level assessment and problem location. Quality levels are assessed as Excellent, Good, Pass, and Fail, with red, yellow, and green colors used to indicate problem areas of varying severity in the report. The adaptive parameter adjustment unit dynamically optimizes the processing algorithm parameters based on the quality assessment results. Its adjustment logic can be expressed in the following mathematical form. The parameter adjustment amount is calculated based on the negative feedback mechanism of quality deviation, and the adjustment formula is: in, Indicates the parameter adjustment amount. The learning rate parameter controls the adjustment step size. This indicates the loss function with respect to the parameters. The system automatically adjusts the matching window size and similarity threshold when the image matching success rate is detected to be below 95%. The matching window size gradually increases from the default 21 pixels x 21 pixels to 31 pixels x 31 pixels; the similarity threshold gradually decreases from 0.7 to 0.6. When the accuracy deviation of the digital elevation model exceeds the 0.15-meter limit, the system automatically optimizes the filtering parameters and interpolation algorithm. The standard deviation factor in the filtering parameters is adjusted from 2.0 to 1.5 to retain more details; the interpolation algorithm is switched from inverse distance weighted interpolation to Kriging interpolation to improve interpolation accuracy.

[0032] The elastic resource scheduling platform manages the computing, storage, and network resources in the system, dynamically allocating hardware resources based on real-time task load. This platform consists of four core components: a resource monitoring sub-platform, a policy execution sub-platform, a predictive scaling mechanism, and a resource isolation mechanism. The resource monitoring sub-platform collects real-time resource metrics data from each computing node, including CPU utilization, memory usage, disk I / O performance, and network bandwidth utilization. CPU utilization is sampled at a granularity of 5%, with a moving average calculated every 30 seconds over a one-minute period; memory usage records the ratio of used memory to total memory; disk I / O performance statistics include the number of read / write operations per second and data transfer rate; and network bandwidth utilization records the proportion of network interface I / O traffic to the theoretical bandwidth. All these metrics are collected by an agent program and aggregated at the monitoring center to form a global resource view. The policy execution sub-platform dynamically adjusts resource allocation strategies based on monitoring data and task queue status. The strategy engine makes decisions based on a rule base and reinforcement learning algorithms. The rule base contains one hundred predefined resource allocation rules. For example, when the CPU utilization exceeds 85% for five consecutive minutes, more computing resources are automatically allocated to that node; when the number of tasks waiting in the task queue exceeds fifty, a new computing instance is automatically launched. The reinforcement learning algorithm continuously tries different resource allocation strategies and evaluates their effects to gradually optimize long-term resource utilization. The predictive scaling mechanism is based on time series analysis and machine learning prediction models to predict changes in resource demand over a future period. Time series analysis uses an autoregressive integral moving average model to analyze the periodic patterns and trends in historical resource usage data. The machine learning prediction model uses a long short-term memory network to make predictions considering multi-dimensional features such as task type, data volume, and processing progress. When it is predicted that the task load will increase by more than 50% within two hours, the platform automatically initiates a resource scaling process, pre-allocating additional computing nodes to join the cluster. The scaling process includes four steps: resource application, system initialization, service deployment, and load balancing adjustment, and the entire process is completed within ten minutes. When resource utilization is detected to remain below 30% for more than three hours, the platform automatically performs resource scaling down, releasing idle nodes to conserve resources. Before scaling down, tasks on the nodes are migrated to other nodes to ensure uninterrupted task execution. The resource isolation mechanism uses containerization technology to achieve resource and security isolation between different aerial photography tasks. Each aerial photography task runs in an independent container environment, and the container uses control group technology to limit the number of CPU cores, memory size, disk space, and network bandwidth it can use. Security isolation is achieved through kernel namespaces, ensuring complete isolation of processes, file systems, and user accounts for different tasks. This isolation mechanism prevents resource contention between tasks while ensuring data security and privacy protection.

[0033] The end-to-end visualization and monitoring module provides system users with full-chain visualization and interactive control functions from data acquisition to product distribution. This module uses a browser and server architecture to provide visualization services. The front-end uses vector tiling technology and a spatial data engine to achieve smooth rendering of large-scale aerial survey data. Vector tiling technology preprocesses large-scale geographic data into a multi-level tile pyramid, dynamically loading data at the corresponding level based on the view zoom level. The spatial data engine is responsible for spatial query and analysis functions, supporting interactive operations such as point selection, bounding selection, and buffer analysis. The real-time processing progress tracking function displays the current status and completion percentage of each processing task through progress bars and Gantt charts. The progress bar uses different colors to distinguish four states: waiting, executing, completed, and failed, and displays the completion percentage accurate to one percent. The Gantt chart displays the timeline horizontally and lists all processing tasks vertically, clearly showing the dependencies and time overlaps between tasks. Users can drill down to view detailed execution logs for any task, including start time, end time, time consumed, resource consumption, and possible error messages. The dynamic quality indicator display function visualizes key quality indicators in the form of a dashboard. The dashboard uses various chart types to display data. Aerial triangulation accuracy uses a scatter plot to show the residual distribution of checkpoints; image matching success rate uses a line chart to show the success rate changes in different areas; digital elevation model accuracy uses a bar chart to compare the accuracy indicators of different algorithms; and 3D model integrity uses a heatmap to show the integrity distribution of the model surface. All charts support historical trend comparison, allowing users to slide the timeline to view the quality status at any point in time. When any quality indicator exceeds a preset threshold, the system automatically triggers a threshold alarm, which is displayed as a flashing icon on the dashboard. An instant alarm push function pushes system and quality anomalies to relevant personnel in real time through multiple channels, including the message center and email notifications. The message center is integrated in the upper right corner of the system interface, displaying the number of unread alarms as digital badges; users can click to view alarm details. Email notifications include alarm summaries and detailed descriptions, along with relevant data links for further analysis. Alarms are divided into three levels: urgent, important, and general. Urgent alarms will also receive an SMS notification to ensure timely response. A hierarchical loading technology dynamically loads data of different levels of detail based on the view zoom level. In the global view, the system displays an overview of the task distribution and overall progress status across the entire survey area. When the user zooms to the region level, the system loads detailed parameters and quality data for each processing task within that region. Zooming further to the single image level displays the image's metadata and processing logs. This hierarchical display method ensures both a grasp of the macro-level situation and the ability to explore micro-details. The level-of-detail technology optimizes the rendering performance of large-scale 3D scenes through frustum culling and level-of-detail model switching. The frustum culling algorithm renders only the 3D model portion within the camera's field of view, significantly reducing the number of triangles that need to be rendered.The level-of-detail model prepares multiple model versions of the same 3D object with different levels of precision. It automatically switches between these versions based on the distance between the object and the camera, using a lower-precision model for greater distances and a higher-precision model for closer proximity. The combination of these two techniques ensures that even in large-scale 3D scenes containing hundreds of millions of triangles, the system maintains a smooth rendering performance of 30 frames per second.

[0034] The entire system is deployed on a distributed cluster consisting of over one hundred proxy servers, with a load balancer distributing external requests to various computing nodes. The system employs a microservice architecture, with each core module running as an independent microservice. Services communicate with each other via remote procedure calls and message queues. All microservices are horizontally scalable, dynamically adjusting the number of instances based on business load. The system monitoring system covers all levels from hardware resources to application performance, including key metrics such as CPU utilization, memory usage, disk I / O, network bandwidth, service response time, and error rate. When any metric exceeds a preset threshold, the monitoring system immediately triggers an alarm and notifies the operations team. Regarding data security, the system implements end-to-end encrypted data transmission and storage. All sensitive data is encrypted before being stored. The access control system is based on a role-based access control model, ensuring that different users can only access data within their authorized scope. The system performs a full data backup every 24 hours, retains the backup data for 30 days, and supports data recovery operations at any point in time.

[0035] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape, and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A data management system based on aerial photogrammetry, characterized in that, include: The multi-source data fusion access module is used to receive heterogeneous data generated by various sensors in aerial photogrammetry operations, and perform data parsing, format unification and spatiotemporal benchmark alignment operations. It is internally deployed with a data integrity verification unit and a multi-dimensional index building unit to ensure that the original aerial survey data is standardized before being stored and to establish an efficient retrieval mechanism. The distributed parallel processing engine adopts a microservice architecture to achieve parallelization and pipelined processing of the entire aerial survey data processing process. The engine includes a task decomposition sub-engine and a process coordination sub-engine, which achieve efficient scheduling and fault-tolerant execution of processing tasks through a unified message communication mechanism and state management service. The intelligent quality control and optimization module, built on machine learning algorithms, continuously monitors the output quality of each processing node in the distributed parallel processing engine, automatically detects data anomalies and accuracy deviations, and generates quality assessment reports and optimization adjustment suggestions. This module integrates a quality index calculation unit and an adaptive parameter adjustment unit; The elastic resource scheduling platform is responsible for managing the computing, storage, and network resources in the system. It dynamically allocates hardware resources based on real-time task load and implements a resource monitoring sub-platform and a policy execution sub-platform within it. It ensures the system's stable performance in high-concurrency task scenarios through a predictive scaling mechanism. The end-to-end visualization monitoring module provides visualization and interactive control functions for the entire chain from data collection to product distribution. It supports real-time tracking of processing progress, dynamic display of quality indicators, and instant push of abnormal alarms. Its rendering engine adopts hierarchical and detail level technology.

2. The data management system based on aerial photogrammetry according to claim 1, characterized in that: The multi-source data fusion access module receives raw data from various sensors, including aerial cameras, lidar, global navigation satellite system receivers, and inertial measurement units, via a high-speed data interface. The data parsing unit employs specialized parsing algorithms for different sensor data formats. It extracts camera parameters and image metadata from aerial image data, extracts point cloud coordinates and intensity information from lidar data, extracts positioning coordinates and timestamps from global navigation satellite system data, and extracts attitude angles and acceleration values ​​from inertial measurement unit data. The format unification unit converts the parsed heterogeneous data into a standardized data exchange format and adds a unified spatiotemporal reference frame identifier to all data.

3. The data management system based on aerial photogrammetry according to claim 2, characterized in that: The spatiotemporal reference alignment is based on precise single-point positioning technology and coordinate transformation model, which unifies the spatial coordinates of all sensor data to the national geodetic coordinate system and coordinates the time information to the universal time. The data integrity verification unit verifies data packet integrity by calculating data packet integrity indicators and checksums; The multidimensional index building unit is based on spatial filling curve and time segmentation technology to build spatial index, time index and sensor type index for all imported data, and supports multidimensional condition combination queries.

4. The data management system based on aerial photogrammetry according to claim 1, characterized in that: The task decomposition sub-engine breaks down the aerial survey data processing flow into multiple independent and parallel-executable atomic tasks, including aerial triangulation, dense matching, digital elevation model generation, orthorectification, and 3D modeling. Each atomic task is encapsulated as a standardized microservice and managed uniformly through a service registry. The process collaboration sub-engine constructs the process dependency relationship based on the directed acyclic graph, and drives atomic tasks to be executed in the dependency order through the workflow engine.

5. The data management system based on aerial photogrammetry according to claim 1, characterized in that: The message communication mechanism uses a highly reliable message queue to achieve asynchronous communication between microservices; The status management service records the execution status, progress percentage, and resource consumption of each processing task, and supports task resumption and retry upon failure. The fault-tolerant execution is achieved through task replication and heartbeat detection. When an abnormality is detected in a processing node, the task is automatically migrated to a healthy node for re-execution.

6. The data management system based on aerial photogrammetry according to claim 1, characterized in that: The machine learning algorithm adopts an ensemble learning framework, which integrates multiple base learners to comprehensively evaluate the quality of aerial survey data; the quality index calculation unit calculates key quality indicators such as aerial triangulation accuracy, image matching success rate, digital elevation model accuracy, and 3D model integrity in real time; the data anomaly detection is based on the isolated forest algorithm and the local anomaly factor algorithm to automatically identify outliers and gross errors in the aerial survey data.

7. The data management system based on aerial photogrammetry according to claim 1, characterized in that: The intelligent quality control and optimization module compares the processing results with high-precision reference data, calculates the statistical characteristics of plane accuracy and elevation accuracy, and generates a detailed report including quality level assessment and problem location by integrating various quality indicators and anomaly detection results. The adaptive parameter adjustment unit dynamically optimizes the processing algorithm parameters based on the quality assessment results. When the image matching success rate is detected to be lower than 95%, it automatically adjusts the matching window size and similarity threshold. When the accuracy deviation of the digital elevation model exceeds the limit, it automatically optimizes the filtering parameters and interpolation algorithm.

8. The data management system based on aerial photogrammetry according to claim 1, characterized in that: The resource monitoring sub-platform collects real-time resource indicators such as CPU utilization, memory usage, disk I / O performance, and network bandwidth utilization of each computing node. The strategy execution sub-platform dynamically adjusts the resource allocation strategy based on monitoring data and task queue status; the predictive scaling mechanism is based on time series analysis and machine learning prediction models to predict changes in resource demand in the future.

9. The data management system based on aerial photogrammetry according to claim 8, characterized in that: When it is predicted that the task load will increase by more than 50% within 1 hour, the platform automatically starts the resource expansion process and pre-allocates additional computing nodes to join the cluster; When the resource utilization rate is detected to be below 30% for more than 2 hours, the platform automatically performs a resource reduction operation to release idle nodes and save resources; the resource isolation mechanism achieves resource isolation and security isolation between different aerial photography missions through containerization technology.

10. The data management system based on aerial photogrammetry according to claim 1, characterized in that: The full-process visualization monitoring module uses a browser and server architecture to provide visualization services. The front end uses vector slicing technology and a spatial data engine to achieve smooth rendering of large-scale aerial survey data. The real-time processing progress tracking function displays the current status and completion percentage of each processing task through progress bars and Gantt charts, and supports drill-down to view detailed execution logs of any task; The dynamic display function of quality indicators visualizes key quality indicators in the form of a dashboard, and supports historical trend comparison and threshold alarms; the real-time push function of anomaly alarms pushes system anomalies and quality anomalies to relevant personnel in real time through multiple channels such as message center and email notification.

Images