Multi-source data processing system for geographic information big data

Abstract

The invention discloses a geographic information big data-oriented multi-source data processing system, and relates to the technical field of data acquisition and sensors, and the system comprises a data acquisition module which accesses a remote sensing satellite, an unmanned aerial vehicle, an Internet of Things sensor and social media in real time through a multi-source heterogeneous interface, and carries out the adaptive analysis of a data format and metadata marking; the distributed storage module is used for performing partition storage on the geographic information data based on a space-time database and an object storage architecture, and establishing dynamic space-time index and version control; the data fusion module is used for realizing coordinate system conversion, time sequence calibration and semantic knowledge graph matching based on a multi-source data alignment method of dynamic weight distribution; and an intelligent analysis module and a security control module. According to the method, core pain points such as data splitting, low storage efficiency, extensive analysis and compliance risks in the geographic information field are systematically solved, and a full-stack type technical base is provided for scenes such as smart cities, emergency disaster relief and environment monitoring.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition and sensors, and more particularly to a multi-source data processing system for geographic information big data. Background Art

[0002] The global satellite data volume grows at an annual rate of over 30%, and the demand in fields such as smart cities and environmental monitoring drives the integrated application of multi-source data. Disaster early warning requires the integration of geological, meteorological, and population data; autonomous driving relies on the fusion of high-precision maps and real-time road condition data, but the centralized architecture is difficult to process PB-level data; single-source data analysis cannot meet complex scenarios, such as urban traffic congestion, which requires the integration of satellite images, GPS trajectories, and meteorological data.

[0003] There are still problems to be solved in the prior art: the spatio-temporal resolution, coordinate system, and semantic description of multi-source data vary greatly. For example, the registration error between drone images and satellite data exceeds 10%, resulting in low fusion efficiency; traditional GIS tools take several hours to process TB-level data, unable to meet the emergency response requirements; the processing delay of streaming data is generally higher than the second level; the continuous update of spatio-temporal data, such as urban building changes, causes the invalidation of traditional database indexes and an increase in storage costs by more than 30%; sensor noise (such as temperature and humidity drift in weather stations) and data loss (such as insufficient remote sensing coverage in remote areas) affect the analysis accuracy; the desensitization of geographical data uses static generalization (such as fixed-area blurring), which is easily cracked by reverse engineering, and the operation logs are stored in a centralized database and are easily tampered with. Summary of the Invention

[0004] To solve the above technical problems, a multi-source data processing system for geographic information big data is provided. The technical solution of the present invention solves the problems of large differences in spatio-temporal resolution, coordinate system, and semantic description of multi-source data, resulting in low fusion efficiency; traditional GIS tools taking several hours to process TB-level data and being unable to meet the emergency response requirements; the processing delay of streaming data generally being higher than the second level; the continuous update of spatio-temporal data causing the invalidation of traditional database indexes and an increase in storage costs; sensor noise and data loss affecting the analysis accuracy; the desensitization of geographical data using the static method being easily cracked by reverse engineering, and the operation logs being stored in a centralized database and being easily tampered with.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A multi-source data processing system for geographic information big data, comprising:

[0007] A data acquisition module configured to access remote sensing satellites, drones, Internet of Things sensors, and social media in real time through a multi-source heterogeneous interface, and support adaptive parsing of data formats and metadata marking;

[0008] The distributed storage module, based on the spatio-temporal database and object storage architecture, partitions and stores geographic information data, and establishes a dynamic spatio-temporal index and version control mechanism;

[0009] The data fusion module uses a multi-source data alignment method based on dynamic weight assignment to achieve coordinate system conversion, time series calibration, and semantic knowledge graph matching;

[0010] The intelligent analysis module integrates a parallel computing framework and machine learning models for tasks such as ground object classification, spatial interpolation, and anomaly detection, and supports CPU-GPU hybrid accelerated computing;

[0011] The security control module deploys differential privacy algorithms and blockchain traceability mechanisms to desensitize sensitive geographic data and record operation logs.

[0012] Preferably, the data acquisition module specifically includes:

[0013] The multi-source heterogeneous interface adaptation unit: satellite remote sensing data interface, compatible with the push / pull mode of mainstream satellite data sources, accessing high-timeliness data through Apache Kafka and RabbitMQ; UAV and aerial image interface, integrating the MAVLink protocol parser, supporting the synchronous acquisition of real-time video streams and metadata from UAV manufacturers, and deploying a lightweight SDK on the UAV side; Internet of Things sensor interface, compatible with the heterogeneous data access of sensors such as temperature and humidity, air quality, and traffic flow, performing time alignment and outlier filtering on sensor data based on InfluxDB and TimescaleDB; social media data interface, calling the Geotagging APIs of various social media platforms to extract text, pictures, and video data with geographical coordinates, identifying geographical location keywords in the text through BERT and GPT models, and associating and mapping them with OpenStreetMap;

[0014] The data format adaptive parsing unit: remote sensing image parsing, supporting pixel-level parsing of multiple formats, automatically extracting band information and spatial reference parameters, using the GDAL / OGR library to achieve unified coordinate system conversion, and generating thumbnails for quick preview; stream data and structured data parsing, extracting key frames and geographical registration for UAV video streams; for Internet of Things sensor data, extracting key fields and converting them into a standardized data model; unstructured data extraction, parsing GPS tags, shooting device models, and timestamps of social media pictures through ExifTool, and combining with the GeoNames database to convert fuzzy location descriptions into longitude and latitude coordinates;

[0015] Metadata Tagging and Management Unit: Automatically generate metadata, automatically append timestamps and spatial extents according to the data source, record the data source type, the serial number of the acquisition device, and the transmission path log; adopt the ISO 19115-1 Geographical Information Metadata Standard to define the core fields; describe sensor parameters through a custom Schema; when the data is cleaned and fused, record the operation history and establish the spatio-temporal association between multi-source data.

[0016] Preferably, the distributed storage module specifically includes:

[0017] Spatio-Temporal Database Core Unit: The storage engine layer adopts a vector-raster-stream data integration model, supports the hybrid storage of points, lines, surfaces, volumes, and time series. For the raster data of remote sensing images, it uses the Apache Parquet format for compressed storage; the distributed computing layer extends Spark SQL to support spatial extent queries, and optimizes the collaborative computing of R-tree indexes and KD-Tree partitions. The correlation analysis between real-time sensor data and the historical database is realized through Flink Stateful Functions.

[0018] Object Storage Architecture Design Unit: Based on Ceph RBD to provide low-latency access, store high-frequency access data within one week, and use AWS Glacier Deep Archive and Alibaba Cloud OSS for archival storage of historical remote sensing data; divide the data into a 6+3 redundancy strategy and use CRDT to achieve data consistency across multiple data centers.

[0019] Spatio-Temporal Partitioned Storage Unit: Divide the globe into grid cells with adjustable precision, support fast spatial extent retrieval, and perform Z-order curve encoding on massive point cloud data; divide the data by hour / day granularity, combine the Watermark mechanism to process out-of-order data, and automatically migrate expired data to the cold storage layer.

[0020] Dynamic Spatio-Temporal Index Unit: Based on the PostGIS GiST index, support spatial extent queries and topological relationship judgments, and construct an inverted index to accelerate attribute filtering; when new data is written, only update the index blocks of the affected grids, and dynamically select the index type according to the query mode.

[0021] Preferably, the data fusion module specifically includes:

[0022] Dynamic Weight Allocation Unit: Dynamically adjust the weights according to the accuracy, timeliness, and confidence of the data source.

[0023] Multi-source data alignment unit: Use the seven-parameter Bursa model to convert the WGS84 coordinate system of UAV data into the local engineering coordinate system; For social media data, through the reverse geocoding API, map the text location description to the GeoHash grid code; Adopt the sliding window dynamic alignment technology to interpolate and match the satellite transit time and the sensor acquisition timestamp; Perform dynamic time warping processing on low-frequency data and high-frequency data to eliminate the sampling interval difference.

[0024] Preferably, the dynamic weight assignment unit specifically includes:

[0025] Dynamic weight assignment formula:

[0026] W i = α·Norm(Accuracy i ) + β·Norm(Recency i ) + γ·Norm(Confidence i )

[0027] Its constraint conditions: α + β + γ = 1, and α, β, γ ∈ [0, 1]; In the formula, W i is the dynamic fusion weight of the i-th data source, and the value range is 0 ≤ W i ≤ 1. The higher the weight, the greater the contribution of the data source to the fusion result; Accuracy i is the accuracy index of the i-th data source; Recency i is the timeliness index of the i-th data source; Confidence i is the credibility index of the i-th data source; α, β, γ are the global weight coefficients of accuracy, timeliness, and credibility; Norm(.) is the normalization function that maps indicators with different dimensions to the [0, 1] interval.

[0028] Preferably, the data fusion module specifically includes:

[0029] Geographic ontology library construction unit: Define standardized geographic entity types and attributes, and describe semantic associations through RDF triples;

[0030] Multi-source semantic mapping unit: Associate the pixel-level annotations in UAV images with the point data of IoT sensors through spatial overlay analysis; Extract geographic entities from the fuzzy descriptions in social media texts through the BERT-Geo model and match them with the POI database of OpenStreetMap; Use graph neural networks to automatically discover implicit relationships in new data;

[0031] Conflict Detection and Resolution Unit: It uses a Bayesian network to evaluate the conflict probability of multi-source data, conducts majority voting on conflict data according to weights, and records abnormal cases with confidence levels lower than the threshold for manual review.

[0032] Preferably, the intelligent analysis module specifically includes:

[0033] Parallel Computing Framework and Task Scheduling Unit: It dynamically scales the computing nodes through Kubernetes, supports batch processing of hundreds of billions of raster data, and real-time processes the correlation analysis of UAV video streams and historical satellite images; automatically generates a task flow chart according to the data dependency relationship, preferentially schedules high-timeliness tasks, and separates CPU-intensive tasks from GPU-intensive tasks;

[0034] Machine Learning Model and Algorithm Library Unit: Based on the improved Swin-Transformer + U-Net architecture, it supports the classification of buildings, roads, and vegetation in high-resolution remote sensing images. Combining lidar point clouds and multi-spectral data, it improves the classification accuracy of complex scenes through graph attention networks; integrates the Kriging algorithm and deep neural processes, and uses spatio-temporal graph convolutional networks to complete the second-level complementation of missing data from IoT sensors in seconds; uses isolation forests to discover regional-level anomalies, identifies pixel-level anomalies through the Transformer-XL model, and combines Bayesian structural equations to distinguish the root causes of anomalies;

[0035] CPU-GPU Hybrid Accelerated Computing Unit: It uses the AVX-512 instruction set to accelerate data normalization and coordinate transformation tasks, and adopts NVIDIA H100 GPUs and CUDA-X libraries to achieve millisecond-level inference of models with tens of billions of parameters; parallelizes the iterative calculation part of the spatial interpolation algorithm on the GPU, and converts the PyTorch model into a FP16 precision engine.

[0036] Preferably, the intelligent analysis module specifically includes:

[0037] Model Optimization and Lightweight Technology: Automatically switches between FP32 / INT8 precision according to task requirements, and trains lightweight models with large models; splits the deep learning model into edge-side feature extraction + cloud-side decision output, and runs lightweight models on the browser side to achieve AR real-scene object annotation.

[0038] Preferably, the security control module specifically includes:

[0039] Differential Privacy Protection Unit: Dynamically sets the differential privacy budget according to the data sensitivity level; adds Laplace noise to numerical data and uses the exponential mechanism for categorical data; in the transmission of UAV video streams, it blurs sensitive areas in real time and performs k-anonymization on the results returned by public map APIs;

[0040] Blockchain Traceability Unit: The government regulatory agency, data provider, and user form a consortium blockchain. Hyperledger Fabric is used to implement a distributed ledger, automatically verifying the compliance of data operations and triggering violation alerts. The SHA-3 hash value of the data operation is uploaded to the blockchain, supporting reverse tracing to the original data version by timestamp without revealing sensitive information when verifying the operator's identity and permissions.

[0041] Preferably, the security control module specifically includes:

[0042] Sensitive Data Desensitization Unit: Randomly perturb and regionally aggregate coordinates, abstract the user's movement path into a topological structure, and eliminate individual identification features. Desensitization is turned off in the disaster emergency mode to allow rescue agencies to obtain accurate locations. In the business analysis scenario, the administrative region level accuracy is retained, and the details of the house number are masked.

[0043] Security Enhancement Unit: Use CRYSTALS-Kyber to encrypt the data transmission channel, and automate key rotation and destruction based on HSM. Dynamically authorize according to user roles, data sensitivity, and operation types. Force high-risk operations to be executed in a containerized environment to block potential data leakage paths.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] The present invention proposes to access multi-source data such as remote sensing satellites, drones, Internet of Things sensors, and social media in real time through multi-source heterogeneous interfaces, support adaptive parsing of data formats and metadata tagging, solve the problems of single data source and low access efficiency in traditional systems, and can quickly obtain dynamically changing geographical information data, providing real-time and comprehensive data support for subsequent analysis; based on a spatio-temporal database and an object storage architecture, the system stores geographical information data in partitions, and establishes a dynamic spatio-temporal index and version control mechanism, improving the storage efficiency of data, and also supporting efficient query and historical data traceability. Through the dynamic index, users can quickly locate data within a specific time and space range, while the version control mechanism ensures the integrity and consistency of data; the data fusion module adopts a multi-source data alignment method based on dynamic weight allocation, solves the problems of inconsistency of multi-source data in coordinate systems, time series, and semantics, and through coordinate system conversion, time series calibration, and semantic knowledge graph matching, the system can seamlessly integrate data from different sources to generate a unified geographical information data set; the intelligent analysis module integrates a parallel computing framework and machine learning models, supports CPU-GPU hybrid accelerated computing, can efficiently complete complex tasks such as ground object classification, spatial interpolation, and anomaly detection, improves the analysis efficiency and accuracy, and is especially suitable for real-time processing of large-scale data; the security control module deploys differential privacy algorithms and blockchain traceability mechanisms, performs desensitization processing on sensitive geographical data, and records operation logs, effectively protecting data privacy, preventing data leakage and abuse, and at the same time ensuring the traceability of data operations through blockchain technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is an internal framework diagram of a multi-source data processing system for geographical information big data;

[0047] Figure 2 is an internal composition diagram of the data acquisition module;

[0048] Figure 3 is an internal composition diagram of the distributed storage module;

[0049] Figure 4 is an internal composition diagram of the data fusion module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments in the following description are only examples, and those skilled in the art can think of other obvious variations.

[0051] Referring to Figure 1 as shown, a multi-source data processing system for geographical information big data includes:

[0052] The data acquisition module is configured to access remote sensing satellites, drones, Internet of Things sensors, and social media in real time through multi-source heterogeneous interfaces, and supports adaptive parsing of data formats and metadata tagging.

[0053] The distributed storage module, based on a spatio-temporal database and an object storage architecture, stores geographical information data in partitions and establishes a dynamic spatio-temporal index and version control mechanism.

[0054] The data fusion module uses a multi-source data alignment method based on dynamic weight allocation to achieve coordinate system conversion, time series calibration, and semantic knowledge graph matching.

[0055] The intelligent analysis module integrates a parallel computing framework and machine learning models for tasks such as ground object classification, spatial interpolation, and anomaly detection, and supports CPU-GPU hybrid accelerated computing.

[0056] The security control module deploys differential privacy algorithms and blockchain traceability mechanisms to desensitize sensitive geographical data and record operation logs.

[0057] It should be noted that cross-module data flow control includes:

[0058] In the stage from data acquisition to storage, resources are allocated in real time through a priority queue (such as disaster emergency data being processed first) to ensure that the transmission delay of high-timeliness data (drone video stream) is <50ms; according to the historical query pattern, the intelligent analysis module preloads frequently accessed data in the distributed storage (such as urban heat maps), and the query response time is shortened by 30%.

[0059] Closed-loop feedback optimization: The anomaly detection results output by the intelligent analysis module, such as illegal building areas, automatically trigger the weight adjustment of the data fusion module to reduce the weight of low-confidence data sources.

[0060] Dynamic adaptation of security policies includes: The operation logs recorded by the blockchain are used to train differential privacy parameters to achieve a dynamic balance between privacy protection intensity and data availability.

[0061] Cross-domain scalability design includes:

[0062] Through Geohash hierarchical compression technology, EB-level remote sensing data is compressed into a TB-level summary model to support global climate change simulation (1km resolution); integrating a digital twin engine to support the fusion of 3D building models with centimeter-level accuracy and real-time traffic data for smart city path planning.

[0063] Preset NDVI vegetation index and PM2.5 diffusion models to support the rapid generation of ecological assessment reports; fuse seismic wave propagation models and social media distress signals to achieve second-level prediction of disaster impact ranges.

[0064] Referring to Figure 2 as shown, the data acquisition module specifically includes:

[0065] Multi-source heterogeneous interface adaptation unit: Satellite remote sensing data interface, compatible with the push / pull mode of mainstream satellite data sources, accessing high-timeliness data through Apache Kafka and RabbitMQ; UAV and aerial image interface, integrating the MAVLink protocol parser, supporting the synchronous acquisition of real-time video streams and metadata of UAV manufacturers, and deploying a lightweight SDK on the UAV side; Internet of Things sensor interface, compatible with the access of heterogeneous data from sensors such as temperature and humidity, air quality, and traffic flow, aligning sensor data in time and filtering outlier values based on InfluxDB and TimescaleDB; Social media data interface, calling the Geotagging API of multiple social media platforms, extracting text, pictures, and video data with geographical coordinates, identifying geographical location keywords in the text through BERT and GPT models, and associating and mapping them with OpenStreetMap;

[0066] Data format adaptive parsing unit: Remote sensing image parsing, supporting pixel-level parsing of multiple formats, automatically extracting band information and spatial reference parameters, realizing unified coordinate system conversion using the GDAL / OGR library, and generating thumbnails for quick preview; Streaming data and structured data parsing, extracting key frames and georegistration from UAV video streams; For Internet of Things sensor data, extracting key fields and converting them into a standardized data model; Unstructured data extraction, parsing GPS tags, shooting device models, and timestamps of social media pictures through ExifTool, and combining with the GeoNames database to convert fuzzy location descriptions into latitude and longitude coordinates;

[0067] Metadata tagging and management unit: Automatic metadata generation, automatically attaching timestamps and spatial ranges according to the data source, recording data source types, collection device serial numbers, and transmission path logs; Adopting the ISO 19115-1 geographic information metadata standard to define core fields; Describing sensor parameters through a custom Schema; When the data is cleaned and fused, recording the operation history and establishing spatio-temporal associations between multi-source data.

[0068] It should be noted that the satellite remote sensing data interface in the multi-source heterogeneous interface adaptation unit includes:

[0069] Quantum encryption transmission protocol, protecting highly sensitive satellite data through QKD (Quantum Key Distribution) technology, with a key refresh period ≤ 10 seconds to prevent quantum computing attacks;

[0070] Dynamic bandwidth allocation, according to the satellite transit time and data priority (such as disaster monitoring data having higher priority), automatically adjusts the Kafka partition bandwidth to ensure processing a data stream of ≥10GB per second.

[0071] The interfaces for drones and aerial images include:

[0072] In addition to MAVLink, integrate private protocol converters from manufacturers such as DJI SkyPort and Parrot ANAFI, supporting more than 90% of commercial drone models; the lightweight SDK incorporates an H.265 encoder accelerated by FPGA, improving the video stream compression ratio to 1:50 in weak network environments while retaining key geographical metadata.

[0073] The interfaces for IoT sensors include:

[0074] For sensors with different sampling frequencies (such as 1Hz for temperature and humidity vs. 10Hz for traffic flow), adopt a timestamp alignment interpolation algorithm with an error rate <0.1%; deploy a TinyML model, such as LSTM anomaly detection, at the sensor end to filter out noise data in real time and reduce the processing pressure on the cloud.

[0075] The interfaces for social media data include:

[0076] Multi - language geographical location parsing, uses the GPT - 5 multimodal model to identify fuzzy location descriptions in unstructured text and associates them with a 500 - meter accuracy grid based on context; uses a generative adversarial network to detect forged geographical tags.

[0077] In the data format adaptive parsing unit, the parsing of remote sensing images includes:

[0078] Super - resolution reconstruction, for low - resolution satellite images, such as 30m / pixel, performs ESRGAN enhancement to generate 5m - resolution images for emergency scenarios; according to the task type, such as water body identification and vegetation monitoring, automatically selects the best band combination, such as SWIR + NIR, reducing manual intervention.

[0079] The parsing of stream data includes:

[0080] For drone video streams, adopt the SLAM (Simultaneous Localization and Mapping) technology to dynamically correct the image offset caused by wind speed, with a registration error <1 pixel; separate voice (identifying location keywords), text (bullet screen coordinates), and visual (landmark buildings) information from social media videos and fuse them to generate high - confidence geographical tags.

[0081] Refer to Figure 3 As shown, the distributed storage module specifically includes:

[0082] Core Unit of Spatiotemporal Database: The storage engine layer adopts a vector-raster-stream data integration model, supports the hybrid storage of points, lines, surfaces, volumes, and time series, and uses the Apache Parquet format for compressed storage of raster data of remote sensing images; the distributed computing layer extends Spark SQL to support spatial range queries, optimizes the collaborative computing of R-tree indexes and KD-Tree partitions, and realizes the correlation analysis of real-time sensor data and historical databases through Flink Stateful Functions;

[0083] Object Storage Architecture Design Unit: Based on Ceph RBD, it provides low-latency access and stores high-frequency access data within a week. It uses AWS Glacier Deep Archive and Alibaba Cloud OSS for archival storage of historical remote sensing data; the data is chunked with a 6+3 redundancy strategy, and CRDT is used to achieve data consistency across multiple data centers;

[0084] Spatiotemporal Partition Storage Unit: The globe is divided into grid cells with adjustable precision, which supports fast spatial range retrieval and performs Z-order curve encoding on massive point cloud data; the data is partitioned by hour / day granularity, combines the Watermark mechanism to process out-of-order data, and automatically migrates expired data to the cold storage layer;

[0085] Dynamic Spatiotemporal Index Unit: Based on the PostGIS GiST index, it supports spatial range queries and topological relationship judgments, constructs an inverted index to accelerate attribute filtering; when new data is written, only the index blocks of the affected grids are updated, and the index type is dynamically selected according to the query mode.

[0086] It should be noted that the core unit of the spatiotemporal database includes:

[0087] A vector-raster-stream data integration model. Vector data (points, lines, surfaces) is stored in the GeoJSON format, which supports fast verification of topological relationships; raster data (remote sensing images) is compressed column by column in Apache Parquet and combined with the COG chunking technology to improve storage efficiency and query speed; stream data (sensor time series) is stored in the Apache Avro format, retaining timestamps, device IDs, and raw values, and supports millisecond-level time window aggregation.

[0088] The object storage architecture design unit includes:

[0089] Multi - level storage architecture, hot data layer, stores high - frequency access data within 7 days, such as real - time disaster images, with latency < 10ms and supports concurrent access volume ≥ 100,000 QPS; warm data layer: stores data within 1 year, adopts Erasure Coding (6 + 3) redundancy strategy, and reduces storage cost by 70%; cold data layer: stores historical remote sensing data such as satellite images 10 years ago, and supports pre - loading for glacier thawing (triggered 12 hours in advance).

[0090] The spatio - temporal partition storage unit includes:

[0091] H3 grid system, divides the globe into hexagonal grids, with adjustable resolution up to 0.5 meters, supports fast spatial range retrieval, such as "retrieve all grid data in Yuhang District, Hangzhou City"; Z - order curve coding, performs spatial filling curve coding on massive point cloud data, making adjacent spatial points continuous in storage and improving query efficiency;

[0092] Dynamic time slicing, partitions by hour / day granularity, combines with the Watermark mechanism to process delayed data such as data disorder caused by sensor network interruption, and tolerates a maximum disorder time window of 5 minutes; identifies low - frequency data based on the LRU - K strategy and migrates it to the cold storage layer to save costs while maintaining access transparency.

[0093] The dynamic spatio - temporal index unit includes:

[0094] Multi - modal index collaboration, GiST index is used for spatial range query and topological relationship judgment, such as "find all hydrological stations intersecting with the Yangtze River Basin"; inverted index accelerates attribute filtering, such as "air quality index > 150 and sensor type = government - certified"; vectorized index constructs an HNSW graph index for high - dimensional spatio - temporal features such as satellite image spectral features, and supports similarity search, such as "find historical images with spectral features similar to a certain plot".

[0095] Analyzes historical query patterns through a reinforcement learning model, automatically selects the optimal index type, such as mainly using R - tree and supplemented by HNSW; only updates the index blocks of the affected grids, for example, when new data in Hangzhou is written, only updates the corresponding partition of the H3 grid index, and shortens the index reconstruction time by 90%.

[0096] Refer to Figure 4 As shown, the data fusion module specifically includes:

[0097] Dynamic weight assignment unit: dynamically adjusts weights according to the accuracy, timeliness, and confidence of data sources;

[0098] Multi-source data alignment unit: Use the seven-parameter Bursa model to convert the WGS84 coordinate system of UAV data into the local engineering coordinate system; for social media data, map the text location description to the GeoHash grid code through the reverse geocoding API; adopt the sliding window dynamic alignment technology to interpolate and match the satellite transit time and the sensor acquisition timestamp; perform dynamic time warping processing on low-frequency data and high-frequency data to eliminate the sampling interval difference.

[0099] Dynamic weight allocation formula:

[0100] W i = α·Norm(Accuracy i ) + β·Norm(Recency i ) + γ·Norm(Confidence i )

[0101] Its constraint conditions: α + β + γ = 1, and α, β, γ ∈ [0, 1]; in the formula, W i is the dynamic fusion weight of the i-th data source, and the value range is 0 ≤ W i ≤ 1. The higher the weight, the greater the contribution of the data source to the fusion result; Accuracy i is the accuracy index of the i-th data source; Recency i is the timeliness index of the i-th data source; Confidence i is the credibility index of the i-th data source; α, β, γ are the global weight coefficients of accuracy, timeliness, and credibility; Norm(.) is the normalization function that maps indicators with different dimensions to the [0, 1] interval.

[0102] The data fusion module specifically includes:

[0103] Geographic ontology library construction unit: Define standardized geographic entity types and attributes, and describe semantic associations through RDF triples;

[0104] Multi-source semantic mapping unit: Associate the pixel-level annotations in UAV images with the point data of IoT sensors through spatial overlay analysis; extract geographic entities from the fuzzy descriptions in social media texts through the BERT-Geo model and match them with the POI database of OpenStreetMap; use graph neural networks to automatically discover implicit relationships in new data;

[0105] Conflict detection and resolution unit: Use Bayesian networks to evaluate the conflict probability of multi-source data, perform majority voting on conflict data according to weights, and record abnormal cases with confidence levels lower than the threshold for manual review.

[0106] It should be noted that the weight factors and algorithm logic in the dynamic weight allocation unit are as follows:

[0107] Accuracy weight (α): Satellite data (0.8 - 0.95) > UAV (0.7 - 0.85) > Sensor (0.6 - 0.75) > Social media (0.3 - 0.5). The positioning error is calculated through the covariance matrix, and the weight is dynamically adjusted.

[0108] Timeliness weight (β): For real-time data such as sensor updates in seconds, β = 1.0; for historical data such as 12 hours before satellite overpass, β = 0.3. It is dynamically updated using an exponential decay function.

[0109] Reliability weight (γ): Based on the device calibration records of the data source, such as γ + 0.2 for ISO-certified sensors, and historical accuracy, such as γ + 0.1 if the location misreport rate of a certain social media account in the past 90 days is < 5%. It is continuously optimized through a reinforcement learning model.

[0110] In the multi-source data alignment unit, the coordinate system conversion and spatial alignment include:

[0111] The seven-parameter Bursa model is used to convert WGS84 coordinates to the local coordinate system through translation (ΔX, ΔY, ΔZ), rotation (θ_X, θ_Y, θ_Z), and scale factor (μ).

[0112] Inverse geocoding optimization: Social media text is parsed through the GPT-5 geographic enhancement model and matched to a GeoHash grid with an accuracy of 30 meters, such as wx4g08, and verified in combination with street view images.

[0113] Sliding window dynamic calibration: The satellite overpass time (10:00:00 UTC) and the sensor acquisition time (10:00:03 local time) are aligned through cubic spline interpolation; Dynamic Time Warping (DTW) is used to align non-uniformly sampled low-frequency data such as daily weather station data and high-frequency data such as traffic flow per minute, and key event points are retained.

[0114] In the geographic ontology library construction unit, the standardized entity definitions include:

[0115] The core entity types include natural entities (rivers, mountains), artificial entities (roads, buildings), and event entities (earthquakes, traffic jams). The attributes include spatial range, time validity period, and confidence labels.

[0116] Semantic association rules: A geographic knowledge graph is constructed through RDF triples (such as <Yangtze River, flows through, Chongqing City>) to support inferences such as "bridges in the Yangtze River basin within Chongqing City".

[0117] In the multi-source semantic mapping unit, the spatial overlay and feature fusion include:

[0118] In the UAV images, the "farmland" pixels are spatially connected and matched with the data from soil moisture sensors to generate a heat map of irrigation suggestions; the Graph Neural Network (GNN) is applied to extract spatio-temporal features from social media texts and traffic camera videos, construct a heterogeneous graph model, and predict the congestion diffusion path in the next 30 minutes;

[0119] For fuzzy semantic parsing, the BERT-Geo model parses unstructured texts such as "the snack bar opposite the east gate of the hospital" into POI coordinates such as "the east gate of the Third Hospital of Peking University, longitude 116.35, latitude 40.05"; by combining the Exif data of the picture and the text description, the positioning accuracy is improved to within 10 meters through the CLIP-Geo model.

[0120] The conflict detection and resolution unit includes:

[0121] Bayesian network modeling, where the nodes include data source type, historical accuracy, and device status, and the conflict probability is calculated through Markov chain Monte Carlo sampling. For example, when the difference between the data of a certain sensor and satellite data is > 20 meters, the conflict probability P = 0.85;

[0122] The majority voting mechanism weights and votes on the conflict data, and the threshold is set to a confidence level ≥ 0.7, otherwise it is marked as an abnormal case;

[0123] Low-confidence case library: Records conflict data with a confidence level < 0.5, such as false fire location reports on social media, which are annotated by experts and then fed back to the weight model for training; the source of the conflict data is traced through blockchain logs to locate hardware or transmission failures.

[0124] In summary, the advantages of the present invention are:

[0125] Through multi-source heterogeneous interfaces, real-time access to multiple data sources such as remote sensing satellites, UAVs, Internet of Things sensors, and social media is achieved, and it supports adaptive parsing of data formats and metadata tagging, improving the efficiency and flexibility of data collection and enabling rapid acquisition of dynamically changing geographic information data;

[0126] Based on a spatio-temporal database and object storage architecture, the system stores geographic information data in partitions and establishes a dynamic spatio-temporal index and version control mechanism. This storage method not only improves the storage efficiency of data but also supports efficient querying and historical data tracing, ensuring the integrity and consistency of the data;

[0127] The data fusion module adopts a multi-source data alignment method based on dynamic weight allocation to solve the inconsistency problems of multi-source data in coordinate systems, time series, and semantics. Through coordinate system transformation, time series calibration, and semantic knowledge graph matching, data from different sources are seamlessly integrated to generate a unified geographic information dataset, improving the accuracy and efficiency of data fusion;

[0128] The intelligent analysis module integrates a parallel computing framework and a machine learning model, supports CPU-GPU hybrid accelerated computing, efficiently completes complex tasks such as ground object classification, spatial interpolation, and anomaly detection, improves the analysis efficiency and accuracy, and is especially suitable for real-time processing of large-scale data;

[0129] The security control module deploys a differential privacy algorithm and a blockchain traceability mechanism, performs desensitization processing on sensitive geographical data, and records operation logs, protecting data privacy, preventing data leakage and abuse, and at the same time ensuring the traceability of data operations through blockchain technology;

[0130] Through efficient acquisition, intelligent storage, precise fusion, rapid analysis, and comprehensive security guarantee, the efficiency and quality of geographical information big data processing have been significantly improved, and it can be widely applied to multiple fields such as urban planning, disaster monitoring, environmental assessment, and agricultural management, providing strong support for intelligent decision-making.

[0131] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and what is described in the above embodiments and the specification is only the principle of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements all fall within the scope of the present invention claimed. The scope of protection claimed by the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-source data processing system for geographic information big data, characterized in that, Including: A data acquisition module, configured to access remote sensing satellites, drones, Internet of Things sensors, and social media in real time through multi-source heterogeneous interfaces, and support adaptive parsing of data formats and metadata tagging; A distributed storage module, based on a spatio-temporal database and an object storage architecture, stores geographic information data in partitions, and establishes a dynamic spatio-temporal index and version control mechanism; A data fusion module, adopting a multi-source data alignment method based on dynamic weight allocation, to achieve coordinate system conversion, time series calibration, and semantic knowledge graph matching; An intelligent analysis module, integrating a parallel computing framework and machine learning models, for tasks such as ground object classification, spatial interpolation, and anomaly detection, and supporting CPU-GPU hybrid accelerated computing; A security control module, deploying differential privacy algorithms and blockchain traceability mechanisms to desensitize sensitive geographic data and record operation logs.

2. The multi-source data processing system for geographic information big data according to claim 1, wherein The data acquisition module specifically includes: A multi-source heterogeneous interface adaptation unit: a satellite remote sensing data interface, compatible with the push / pull mode of mainstream satellite data sources, accessing high-timeliness data through Apache Kafka and RabbitMQ; a drone and aerial image interface, integrating a MAVLink protocol parser, supporting real-time video stream and metadata synchronization acquisition of drone manufacturers, and deploying a lightweight SDK on the drone side; an Internet of Things sensor interface, compatible with heterogeneous data access of sensors such as temperature and humidity, air quality, and traffic flow, performing time alignment and outlier filtering on sensor data based on InfluxDB and TimescaleDB; a social media data interface, calling the Geotagging API of multiple social media platforms, extracting text, pictures, and video data with geographic coordinates, identifying geographic location keywords in the text through BERT and GPT models, and performing associated mapping with OpenStreetMap; A data format adaptive parsing unit: remote sensing image parsing, supporting pixel-level parsing of multiple formats, automatically extracting band information and spatial reference parameters, using the GDAL / OGR library to achieve unified coordinate system conversion, and generating thumbnails for quick preview; streaming data and structured data parsing, extracting key frames and geographic registration for drone video streams; for Internet of Things sensor data, extracting key fields and converting them into a standardized data model; unstructured data extraction, parsing GPS tags, shooting device models, and timestamps of social media pictures through ExifTool, and combining with the GeoNames database to convert fuzzy location descriptions into latitude and longitude coordinates; A metadata tagging and management unit: automatic metadata generation, automatically attaching timestamps and spatial ranges according to data sources, recording data source types, collection device serial numbers, and transmission path logs; adopting the ISO 19115-1 geographic information metadata standard to define core fields; describing sensor parameters through a custom Schema; when the data is cleaned and fused, recording the operation history and establishing spatio-temporal associations between multi-source data.

3. A multi-source data processing system for geographic information big data according to claim 2, characterized in that, The distributed storage module specifically includes: Core Unit of Spatiotemporal Database: The storage engine layer adopts a vector-raster-stream data integration model, supports the hybrid storage of points, lines, surfaces, volumes, and time series, and uses the Apache Parquet format for compressed storage of raster data of remote sensing images; the distributed computing layer extends Spark SQL to support spatial range queries and optimizes the collaborative computing of R-tree indexes and KD-Tree partitions, and realizes the correlation analysis of real-time sensor data and historical databases through Flink Stateful Functions; Object Storage Architecture Design Unit: Based on Ceph RBD, it provides low-latency access and stores high-frequency access data within a week. It uses AWS Glacier Deep Archive and Alibaba Cloud OSS for archival storage of historical remote sensing data; divides data into a 6+3 redundancy strategy and uses CRDT to achieve data consistency across multiple data centers; Spatiotemporal Partition Storage Unit: Divides the globe into grid cells with adjustable precision, supports fast spatial range retrieval, and performs Z-order curve encoding on massive point cloud data; divides data by hour / day granularity, combines the Watermark mechanism to process out-of-order data, and automatically migrates expired data to the cold storage layer; Dynamic Spatiotemporal Index Unit: Based on the PostGIS GiST index, it supports spatial range queries and topological relationship judgments, and constructs an inverted index to accelerate attribute filtering; when new data is written, only the index blocks of the affected grids are updated, and the index type is dynamically selected according to the query mode.

4. A multi-source data processing system for geographic information big data according to claim 3, characterized in that The data fusion module specifically includes: Dynamic Weight Allocation Unit: Dynamically adjusts weights according to the accuracy, timeliness, and confidence of data sources; Multi-source Data Alignment Unit: Converts the WGS84 coordinate system of UAV data to the local engineering coordinate system using the seven-parameter Bursa model; for social media data, maps the text location description to the GeoHash grid encoding through the inverse geocoding API; adopts a sliding window dynamic alignment technique to interpolate and match the satellite transit time and the sensor acquisition timestamp; performs dynamic time warping on low-frequency and high-frequency data to eliminate sampling interval differences.

5. A multi-source data processing system for geographic information big data according to claim 4, characterized in that The dynamic weight allocation unit specifically includes: Dynamic Weight Allocation Formula: W i = α·Norm(Accuracy i ) + β·Norm(Recency i ) + γ·Norm(Confidence i ) Its constraints are: α + β + γ = 1, and α, β, γ ∈ [0, 1]; where, W i is the dynamic fusion weight of the i-th data source, and its value range is 0 ≤ W i ≤ 1. The higher the weight, the greater the contribution of the data source to the fusion result; Accuracy i is the accuracy index of the i-th data source; Recency i is the timeliness index of the i-th data source; Confidence i is the credibility index of the i-th data source; α, β, γ are the global weight coefficients of accuracy, timeliness, and credibility; Norm(.) is the normalization function that maps indicators with different dimensions to the [0, 1] interval.

6. The multi-source data processing system for geographic information big data according to claim 5, characterized in that, The data fusion module specifically includes: Geographic Ontology Library Construction Unit: Defines standardized geographic entity types and attributes, and describes semantic associations through RDF triples; Multi-source Semantic Mapping Unit: Associates pixel-level annotations in UAV images with the point data of IoT sensors through spatial overlay analysis; extracts geographic entities from fuzzy descriptions in social media texts through the BERT-Geo model and matches them with the POI database of OpenStreetMap; uses graph neural networks to automatically discover implicit relationships in new data; Conflict Detection and Resolution Unit: Uses Bayesian networks to evaluate the conflict probability of multi-source data, performs majority voting on conflict data according to weights, and records abnormal cases with confidence levels lower than the threshold for manual review.

7. A multi-source data processing system for geographic information big data according to claim 6, characterized in that The intelligent analysis module specifically includes: Parallel Computing Framework and Task Scheduling Unit: Dynamically scale computing nodes through Kubernetes, support batch processing of hundreds of billions of grid data, and perform real-time association analysis of UAV video streams and historical satellite images; automatically generate a task flow chart according to data dependencies, prioritize high-timeliness tasks, and separate CPU-intensive tasks from GPU-intensive tasks; Machine Learning Model and Algorithm Library Unit: Based on the improved Swin-Transformer + U-Net architecture, support building, road, and vegetation classification of high-resolution remote sensing images, combine lidar point clouds with multi-spectral data, and improve the classification accuracy of complex scenes through graph attention networks; integrate Kriging algorithm and deep neural processes, and use spatio-temporal graph convolutional networks to complete missing data of IoT sensors in seconds; use Isolation Forest to discover regional-level anomalies, identify pixel-level anomalies through the Transformer-XL model, and combine Bayesian structural equations to distinguish the root causes of anomalies; CPU-GPU Hybrid Accelerated Computing Unit: Use the AVX-512 instruction set to accelerate data normalization and coordinate conversion tasks, and use NVIDIA H100 GPUs and CUDA-X libraries to achieve millisecond-level inference of 10 billion parameter models; parallelize the iterative calculation part of the spatial interpolation algorithm on the GPU, and convert the PyTorch model into a FP16 precision engine.

8. A multi-source data processing system for geographic information big data according to claim 7, characterized in that The intelligent analysis module specifically includes: Model Optimization and Lightweight Technology: Automatically switch between FP32 / INT8 precision according to task requirements, and train lightweight models with large models; split the deep learning model into edge feature extraction + cloud decision output, and run lightweight models on the browser side to achieve AR real-scene object annotation.

9. A multi-source data processing system for geographic information big data according to claim 8, characterized in that, The security control module specifically includes: Differential Privacy Protection Unit: Dynamically set differential privacy budgets according to data sensitivity levels; add Laplace noise to numerical data and use the exponential mechanism for categorical data; in the transmission of UAV video streams, perform real-time blurring of sensitive areas and k-anonymize the results returned by public map APIs; Blockchain Traceability Unit: Government regulatory agencies, data providers, and users form a consortium chain, use Hyperledger Fabric to implement a distributed ledger, automatically verify the compliance of data operations, and trigger violation alerts; upload the SHA-3 hash value of data operations to the chain, support reverse tracing to the original data version according to the timestamp, and do not disclose sensitive information when verifying the identity and permissions of operators.

10. A multi-source data processing system for geographic information big data according to claim 9, characterized in that, The security control module specifically includes: Sensitive Data Desensitization Unit: Randomly perturb and regionally aggregate coordinates, abstract the user's movement path into a topological structure, and eliminate individual identification features; turn off desensitization in the disaster emergency mode to allow rescue agencies to obtain accurate locations; retain administrative region-level accuracy in commercial analysis scenarios and mask the details of house numbers; Security Enhancement Unit: It uses CRYSTALS-Kyber to encrypt the data transmission channel, realizes the automation of key rotation and destruction based on HSM; dynamically authorizes according to user roles, data sensitivity, and operation types; enforces high-risk operations to be executed in a containerized environment to block potential data leakage paths.

Images