A data bus based on multi-source remote sensing data processing

Abstract

The present invention relates to the field of data processing technology, and in particular to a data bus based on multi-source remote sensing data processing, comprising a unified data organization and management module and a unified data service module. The present invention achieves seamless integration of the entire process from original remote sensing data acquisition to refined processing through a refined hierarchical storage system and dynamic ETL processing, ensuring that data is effectively preserved and independently managed in the detail layer, intermediate processing layer, service layer, and application layer, thereby significantly improving data integrity and high traceability. The data state tree constructed through the state information synchronization component can monitor the update time and state changes of data at each layer in real time, and dynamically adjust the data state using version control and negative feedback mechanisms to achieve threshold adaptive correction, effectively solving the problems of a small range of data processing application scenarios and low multi-source remote sensing data integration due to over-reliance on static data processing and low data sharing.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data bus based on multi-source remote sensing data processing. Background Art

[0002] With the rapid development of Earth observation technology, the scale of multi-source remote sensing data processing operations is continuously expanding, and the resulting business systems are becoming increasingly complex and diverse. Due to historical reasons and technical limitations, remote sensing satellite ground systems have traditionally been divided by business, with different departments carrying out independent, "siloed" construction. This has resulted in inconsistent data production standards, independent data storage and management, and difficulties in cross-system interconnection and interoperability. This has led to numerous data management and application challenges. To address the need for data standardization in multi-source remote sensing data processing, a more efficient and flexible data integration and interaction solution is urgently needed.

[0003] Patent document No. CN116737854A discloses a spatiotemporal data lake management system based on multi-source remote sensing data and its security protection method. The system includes: a data identification layer, which identifies multi-source remote sensing data, monitors abnormal data during identification, and extracts and disinfects virus data separately; after disinfection, the data is monitored again to ensure that the data is normal, and then the data is restored to its original location; a data collection layer, which organizes the identified data, eliminates redundant data, and obtains digital remote sensing data from different historical periods in a time-series order; and collects The data is analyzed and converted to analytical data that can be directly used; the data storage layer builds a data lake architecture, first caching the spatiotemporal data in the Kafka queue and storing them in categories; then the spatiotemporal data is stored from the Kafka queue to the data lake through the storage component, and the data lake is partitioned; the data fusion layer converts digital remote sensing data into spatial resolution images based on multi-source digital remote sensing data; then the medium spatial resolution images and high spatial resolution images are fused to generate composite medium and high resolution images, thereby obtaining high-quality remote sensing images.

[0004] It can be seen that the spatiotemporal data lake management system based on multi-source remote sensing data has the following problems: although the spatiotemporal data lake architecture uses Kafka queue caching and partition management for data storage, storage management and data migration face performance bottlenecks when processing large-scale remote sensing data; the system fuses images of different resolutions in the data fusion layer, with the goal of generating high-quality remote sensing images, but the details of data quality and collection methods are not fully optimized during the data collection process; the system relies on static data for data identification, storage and processing, and lacks a real-time adjustment and optimization mechanism when faced with changes in different data quality and storage requirements, and cannot quickly respond to different application requirements. Summary of the Invention

[0005] To this end, the present invention provides a data bus based on multi-source remote sensing data processing, which is used to overcome the problems in the prior art of small application scenarios of multi-source remote sensing data processing and low fusion of multi-source remote sensing data due to over-reliance on static data processing and low data sharing through a data management mechanism of unified organizational management and data status tree.

[0006] To achieve the above object, the present invention provides a data bus based on multi-source remote sensing data processing, comprising:

[0007] The unified data organization and management module includes an archiving and aggregation component, which is used to perceive the product data updated by each remote sensing business plug-in set in real time, and archive and aggregate the product data into the remote sensing data global resource pool to form an archive data set;

[0008] an association integration component connected to the archive aggregation component for performing ETL processing on the archive data set to form a standardized data set;

[0009] a hierarchical storage component connected to the association integration component for storing the standardized data set in the detailed data layer, the data intermediate layer, the data service layer, and the data application layer according to preset business requirements to form a hierarchical storage data set;

[0010] a state information synchronization component connected to the hierarchical storage component, configured to construct a data state tree based on the hierarchical storage data set, and mark the data state in the data state tree and assign a version number to form a versioned data set;

[0011] A data unified service module, connected to the data unified organization and management module, includes a service construction component for constructing a unified service interface based on preset multi-source data processing requirements and the original data, standardized data, and version control information in the versioned data set, thereby forming a data service interface;

[0012] A data service component, which is connected to the service construction component and is used to provide query, browsing, statistics, download, subscription and customization services of remote sensing data through the data service interface to form a unified data service;

[0013] The hierarchical storage data set includes a detailed data set, an intermediate processing data set, a data service set and an application data set.

[0014] Furthermore, the association integration component includes:

[0015] A quality verification unit, configured to perform integrity, spatiotemporal consistency, and data accuracy quality verification on the archived data set to generate a quality verification result;

[0016] a preprocessing unit, configured to perform format conversion, outlier removal, and data completion on the archived data set according to the quality verification result to form a preprocessed data set;

[0017] a feature matching unit connected to the preprocessing unit, configured to perform feature matching based on a feature vector of the preprocessing data set and a preset threshold value group to form a matching data set;

[0018] a threshold adjustment unit connected to the feature matching unit, configured to adjust the preset threshold group according to the number of matching data sets formed within a preset time period to form an adjusted threshold group;

[0019] An ETL conversion unit is connected to the feature matching unit and is used to perform ETL conversion on the matching data set formed based on the adjustment threshold group to form a standardized data set.

[0020] Furthermore, the feature matching unit includes:

[0021] A similarity calculation subunit, configured to calculate the similarity between each feature according to the feature vector and a preset similarity measurement method to form a plurality of similarities;

[0022] a similar data capturing subunit, connected to the similarity calculating subunit, for capturing corresponding pre-processed data in the pre-processed data set when the similarity is greater than a preset similarity threshold in the preset threshold group, to form a similar data set;

[0023] a time series matching subunit, connected to the similar data capture subunit, for performing temporal data matching based on the similar data set, and determining that the match is successful when the time offset of the data match is less than a preset time offset threshold in the preset threshold group, thereby forming a time series data set;

[0024] a spatial matching subunit, connected to the temporal matching subunit, for performing spatial data matching based on the temporal data set, and determining that the match is successful when the spatial overlap of the data matching is greater than a preset overlap threshold in the preset threshold group, thereby forming a spatial data set;

[0025] The feature fusion subunit is connected to the spatial matching subunit and is used to fuse the spatial features in the spatial data set to form the matching data set.

[0026] Furthermore, the threshold adjustment unit includes:

[0027] a quantity fluctuation calculation subunit, for calculating the standard deviation of the formed quantity to form a quantity fluctuation value;

[0028] An adjustment subunit is connected to the quantity fluctuation calculation subunit and is used to reduce the preset similarity threshold according to the relative deviation between the quantity fluctuation value and the maximum value of the preset quantity fluctuation range when the quantity fluctuation value is greater than the maximum value of the preset quantity fluctuation range to form an adjustment threshold group, or to increase the preset time offset threshold when the quantity fluctuation value is greater than the midpoint value within the preset quantity fluctuation range to form an adjustment threshold group, or to reduce the preset overlap threshold when the quantity fluctuation value is between the minimum value and the midpoint value of the preset quantity fluctuation range to form an adjustment threshold group.

[0029] Furthermore, the hierarchical storage component includes:

[0030] a detailed data storage unit, used for storing the standardized data that has not been aggregated and processed in the standardized data set to form the detailed data set;

[0031] a data intermediate layer storage unit connected to the detailed data storage unit, for cleaning, converting, multi-dimensionally aggregating and storing the detailed data set to form the intermediate processed data set;

[0032] A data service layer storage unit connected to the data intermediate layer storage unit for performing index optimization, partition storage, query acceleration, and storage on the intermediate processing data set to form the data service set;

[0033] A data application layer storage unit is connected to the data service layer storage unit and is used to extract and store visual analysis data and decision support data from the data service set based on the preset business requirements to form the application data set.

[0034] Furthermore, the state information synchronization component includes:

[0035] A state tree construction unit, configured to initialize each node in a preset state tree based on the data update time and storage level in the hierarchically stored data set to form the data state tree;

[0036] A status marking unit, configured to classify and mark the data according to the data update time to form a data status marking set;

[0037] A version control unit connected to the status marking unit, configured to assign a unique version number to each data state in the status marking data set, and to maintain version change records to form version management information;

[0038] A state synchronization unit is connected to the version control unit and is used to synchronize a number of version management information into the data state tree according to the version management information and a preset time window to form a versioned data set.

[0039] Furthermore, the state synchronization unit includes:

[0040] a change value calculation subunit, configured to calculate a difference between timestamps in the data state of a current version and a previous adjacent version in the version management information to form a data state change value;

[0041] The state synchronization subunit is connected to the change value calculation subunit and is used to synchronize the version management information to the data state tree to form a versioned data set when the data state change value is greater than the preset time window.

[0042] Furthermore, the service building component includes:

[0043] A requirement analysis unit, configured to analyze the preset multi-source data processing requirements to obtain functional requirements and interface requirements;

[0044] A data mapping unit, configured to uniformly map and convert the format of the original data, the standardized data, and the version control information in the versioned data set to form a data mapping table;

[0045] an interface design unit, connected to the demand analysis unit and the data mapping unit, respectively, for designing the unified service interface according to the functional requirements, the interface requirements, the data mapping table, and the preset business requirements to form a service interface design document;

[0046] An interface encapsulation unit is connected to the interface design unit and is used to perform encapsulation according to the service interface design document to construct the unified service interface and form the data service interface.

[0047] Furthermore, the data service component includes:

[0048] Demand identification unit, used to identify users' query, browsing, statistics, downloading, subscription and customization needs, and form a user demand set based on the data request input by the user;

[0049] a demand processing unit connected to the demand identification unit, for processing and filtering the original data, standardized data and version control information to form a processed data set;

[0050] A data interaction unit is connected to the demand processing unit and is used to provide the processed data set through the constructed service interface to form a unified data service.

[0051] Furthermore, the unified data service includes general data services and dedicated data services:

[0052] The general data services include metadata services, remote sensing data publishing services, remote sensing data subscription services and remote sensing data retrieval services;

[0053] The dedicated data services include auxiliary data services, calibration data services, optical remote sensing image data services and multispectral remote sensing image services.

[0054] Compared with existing technologies, the present invention achieves a beneficial effect by achieving seamless integration from the initial acquisition of remote sensing data to refined processing through a refined hierarchical storage system and dynamic ETL processing, ensuring that data is effectively stored and independently managed at the detail layer, intermediate processing layer, service layer, and application layer, thereby significantly improving data integrity and traceability. The data state tree constructed through the state information synchronization component can monitor the update time and state changes of data at each layer in real time, and dynamically adjust the data state using version control and negative feedback mechanisms, accurately calculating the change value between adjacent versions and achieving adaptive threshold correction. This not only improves the accuracy of data matching and fusion, but also significantly shortens the data update cycle, ensuring that the processing process meets actual business needs. At the same time, multiple data services based on a unified service interface enable efficient and convenient data access in different application scenarios, meeting the requirements for accurate, real-time, and efficient remote sensing data processing. This effectively solves the problems of limited application scenarios and low multi-source remote sensing data fusion caused by over-reliance on static data processing and low data sharing.

[0055] Furthermore, the basic quality of the data is ensured through quality verification and preprocessing. Combined with feature matching and dynamic threshold adjustment mechanisms, multi-level processing from archived data to standardized data is achieved, which significantly improves the accuracy and consistency of data matching and effectively reduces the error rate. Finally, the standardized data set generated by ETL conversion provides a high-quality, standardized data foundation for subsequent data storage and application, greatly improving the operational efficiency of data processing and data utilization.

[0056] Furthermore, through multi-level similarity, temporal, and spatial matching and fusion processing, the matching accuracy and consistency of remote sensing data from different sources are improved. Similarity calculation and similar data capture ensure the similarity of data features, temporal matching ensures the temporal correlation of data, and spatial matching enhances the spatial consistency of data. Furthermore, feature fusion further improves data integrity and usability, enabling the resulting matching dataset to better support subsequent data analysis and applications, thereby enhancing the service value and reliability of remote sensing data.

[0057] Furthermore, by dynamically adjusting the matching threshold, the adaptability of the matching dataset is enhanced, allowing the matching rules to be optimized as data fluctuates, avoiding data matching failures or over-selection issues caused by fixed parameters. At the same time, this method improves the flexibility and applicability of data integration while ensuring data quality, making data matching more stable and accurate, and suitable for a variety of complex application scenarios.

[0058] Furthermore, the tiered storage approach not only optimizes data storage efficiency but also increases the flexibility and speed of data processing and access. The detailed data storage unit ensures data integrity, the middle-tier storage unit enhances data availability, the data service layer storage unit ensures efficient querying and rapid response, and the application layer storage unit meets specific business needs for rapid data extraction and analysis. This effectively improves scalability, flexibility, and efficiency, allowing data at different levels to be managed and used appropriately according to specific needs.

[0059] Furthermore, through real-time tracking of data status and version control, data consistency, accuracy, and traceability are effectively guaranteed. Precise version management and synchronization mechanisms ensure data consistency and stability at all stages, improving reliability and transparency in data processing.

[0060] Furthermore, by calculating the timestamp difference between the current version and the previous version, changes in the data status can be effectively determined, ensuring that data is synchronized to the data status tree only when there is a significant change in the data status. This not only avoids unnecessary frequent updates but also ensures efficient system operation. When the data status change value is greater than the preset time window, the system immediately synchronizes the version management information to the data status tree, ensuring that the data status promptly and accurately reflects the actual changes, thereby improving data consistency and accuracy. In this way, data management becomes more flexible and intelligent, able to dynamically adapt to different data update needs, reduce unnecessary storage and computing overhead, and improve the system's response speed and stability.

[0061] Furthermore, requirements analysis and data mapping ensure the accuracy and compatibility of interface design, avoiding issues with inconsistent data formats. Through standardized interface design, the system provides users with consistent and flexible data access interfaces, improving the scalability and maintainability of data services. Furthermore, interface encapsulation simplifies service integration, enabling rapid response to changing business requirements and contributing to improved overall system performance and user experience.

[0062] Furthermore, high-quality and high-availability data services are provided through clear screening and processing processes, which improves the flexibility and response efficiency of the system. The unified data service interface simplifies the user's operation process, reduces dependence on different data sources and service interfaces, and makes data access more efficient and convenient.

[0063] Furthermore, general data services can meet the basic needs of most users for remote sensing data, ensuring efficient and widely applicable data access and management. Dedicated data services provide deeply customized support for specific fields and needs, enhancing the application value and accuracy of remote sensing data. This service classification structure enhances the system's flexibility and adaptability, providing optimized service solutions for different user needs and application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 Schematic diagram of the architecture of the data bus based on multi-source remote sensing data processing in this embodiment;

[0065] Figure 2 A flow chart showing how the feature matching unit of this embodiment forms a matching data set;

[0066] Figure 3 A flowchart of forming a hierarchical storage data set for the hierarchical storage component of this embodiment;

[0067] Figure 4 A decision logic diagram for forming a versioned data set for the state synchronization unit of this embodiment. DETAILED DESCRIPTION

[0068] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are merely used to explain the present invention and are not intended to limit the present invention.

[0069] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0070] See also Figure 1 , which is a schematic diagram of a data bus based on multi-source remote sensing data processing in this embodiment;

[0071] This embodiment provides a data bus based on multi-source remote sensing data processing, including:

[0072] The unified data organization and management module includes an archiving and aggregation component, which is used to perceive the product data updated by each remote sensing business plug-in set in real time, and archive and aggregate the product data into the remote sensing data global resource pool to form an archive data set;

[0073] an association integration component connected to the archive aggregation component for performing ETL processing on the archive data set to form a standardized data set;

[0074] a hierarchical storage component connected to the association integration component for storing the standardized data set in the detailed data layer, the data intermediate layer, the data service layer, and the data application layer according to preset business requirements to form a hierarchical storage data set;

[0075] a state information synchronization component connected to the hierarchical storage component, configured to construct a data state tree based on the hierarchical storage data set, and mark the data state in the data state tree and assign a version number to form a versioned data set;

[0076] a data unified service module, connected to the data unified organization and management module, comprising a service construction component for constructing a unified service interface based on preset multi-source data processing requirements and the original data, standardized data, and version control information in the versioned data set, thereby forming a data service interface;

[0077] A data service component, which is connected to the service construction component and is used to provide query, browsing, statistics, download, subscription and customization services of remote sensing data through the data service interface to form a unified data service;

[0078] The hierarchical storage data set includes a detailed data set, an intermediate processing data set, a data service set and an application data set.

[0079] Product data refers to the remote sensing data results obtained and updated in real time by different remote sensing business plug-in sets (i.e., business modules designed for different remote sensing data types or application scenarios) during data collection, preprocessing, analysis or product generation. It usually includes remote sensing images that have been preliminarily processed or corrected, feature extraction results, data indicators or other data information related to remote sensing information, providing a basic basis for subsequent data integration, storage and application services.

[0080] Preset business requirements are a series of goals and performance indicators that are pre-established at the beginning of system design based on specific business scenarios and user needs to guide the construction of data storage and service interfaces. Their formulation depends on the actual scenarios of the application field, user expectations, and the quality and quantity of existing data resources.

[0081] First, the archive aggregation component in the data unified organization and management module perceives the product data updated by each remote sensing business plug-in set in real time, and archives and aggregates it into the remote sensing data global resource pool to form an archive data set; then, the association integration component performs ETL processing on the archive data set to generate a standardized data set; then, the hierarchical storage component stores the standardized data set in the detailed data layer, intermediate processing data layer, data service layer and data application layer according to preset business requirements, forming a complete hierarchical storage data set; then, the status information synchronization component builds a data status tree based on the hierarchical storage data set, marks the status of each node and assigns a version number, thus forming a versioned data set; finally, the service construction component in the data unified service module builds a unified service interface based on the preset multi-source data processing requirements and the original data, standardized data and version control information in the versioned data set, and the data service component uses this interface to provide remote sensing data query, browsing, statistics, download, subscription and customization services to form a unified data service.

[0082] Through a refined tiered storage system and dynamic ETL processing, the entire remote sensing data process, from raw acquisition to refined processing, is seamlessly integrated. This ensures that data is effectively stored and independently managed across the detail, intermediate processing, service, and application layers, significantly enhancing data integrity and traceability. A data state tree, constructed using the state information synchronization component, monitors the update time and state changes of data at each layer in real time. Using version control and negative feedback mechanisms, the data state is dynamically adjusted, accurately calculating the change between adjacent versions and enabling adaptive threshold adjustments. This not only improves the accuracy of data matching and fusion, but also significantly shortens the data update cycle, ensuring that the processing process meets actual business needs. Furthermore, multiple data services based on a unified service interface enable efficient and convenient data access in diverse application scenarios, meeting the requirements for accurate, real-time, and efficient remote sensing data processing. This effectively addresses the limitations of multi-source remote sensing data processing applications and the low degree of multi-source data fusion caused by overreliance on static data processing and low data sharing.

[0083] Specifically, the association integration component includes:

[0084] A quality verification unit, configured to perform integrity, spatiotemporal consistency, and data accuracy quality verification on the archived data set to generate a quality verification result;

[0085] a preprocessing unit, configured to perform format conversion, outlier removal, and data completion on the archived data set according to the quality verification result to form a preprocessed data set;

[0086] a feature matching unit connected to the preprocessing unit, configured to perform feature matching based on a feature vector of the preprocessing data set and a preset threshold value group to form a matching data set;

[0087] a threshold adjustment unit connected to the feature matching unit, configured to adjust the preset threshold group according to the number of matching data sets formed within a preset time period to form an adjusted threshold group;

[0088] An ETL conversion unit is connected to the feature matching unit and is used to perform ETL conversion on the matching data set formed based on the adjustment threshold group to form a standardized data set.

[0089] The preset threshold group refers to a series of threshold parameters set to ensure the accuracy and consistency of data matching during the processing of multi-source remote sensing data, including the preset similarity threshold, the preset time offset threshold and the preset overlap threshold. These thresholds are used to determine the degree of matching, time synchronization accuracy and spatial overlap ratio between data, so as to improve the quality of data fusion, reduce mismatching and information loss, and ensure the reliability and stability of data processing results.

[0090] First, the archived data set is verified for integrity, spatiotemporal consistency, and data accuracy through the quality verification unit to form a quality verification result; then, the preprocessing unit converts the format of the archived data set, removes outliers, and completes the data based on the verification result to form a preprocessed data set; then, the feature matching unit uses the feature vectors in the preprocessed data set and the preset threshold group to perform feature matching, extract similar data and form a matching data set; next, the threshold adjustment unit dynamically adjusts the preset threshold group according to the number of matching data sets formed within the preset time length to form an adjusted threshold group; finally, the ETL conversion unit performs ETL conversion on the matching data set based on the adjusted threshold group to form a standardized data set.

[0091] The basic quality of the data is ensured through quality verification and preprocessing. Combined with feature matching and dynamic threshold adjustment mechanisms, multi-level processing from archived data to standardized data is achieved, which significantly improves the accuracy and consistency of data matching and effectively reduces the error rate. Finally, the standardized data set generated by ETL conversion provides a high-quality, standardized data foundation for subsequent data storage and application, greatly improving the operational efficiency of data processing and data utilization.

[0092] Please continue reading Figure 2 As shown, it is a flow chart of the feature matching unit forming a matching data set in this embodiment;

[0093] The feature matching unit includes:

[0094] A similarity calculation subunit, configured to calculate the similarity between each feature according to the feature vector and a preset similarity measurement method to form a plurality of similarities;

[0095] a similar data capturing subunit, connected to the similarity calculating subunit, for capturing corresponding pre-processed data in the pre-processed data set when the similarity is greater than a preset similarity threshold in the preset threshold group, to form a similar data set;

[0096] a time series matching subunit, connected to the similar data capture subunit, for performing temporal data matching based on the similar data set, and determining that the match is successful when the time offset of the data match is less than a preset time offset threshold in the preset threshold group, thereby forming a time series data set;

[0097] a spatial matching subunit, connected to the temporal matching subunit, for performing spatial data matching based on the temporal data set, and determining that the match is successful when the spatial overlap of the data matching is greater than a preset overlap threshold in the preset threshold group, thereby forming a spatial data set;

[0098] The feature fusion subunit is connected to the spatial matching subunit and is used to fuse the spatial features in the spatial data set to form the matching data set.

[0099] The preset similarity threshold is the minimum standard for measuring the similarity between data feature vectors. It depends on the data type, the calculation method of the feature vector and the business needs. It is usually set between 0.7 and 0.95. In this embodiment, it is set to 0.85 to ensure high accuracy of matching data while taking into account data coverage.

[0100] The preset time offset threshold is the maximum allowable deviation used to determine whether the time attributes of two data items match. It depends on the acquisition frequency of remote sensing data, the time sensitivity of the application scenario, and the data alignment requirements. It is usually set between 5 seconds and 30 minutes to adapt to the time matching requirements of different business needs. In this embodiment, it is set to 10 minutes to ensure data timeliness while providing sufficient time redundancy to match asynchronously collected data.

[0101] The preset overlap threshold is the minimum standard for measuring the degree of data overlap during spatial matching. It depends on the resolution of the remote sensing image, the business requirements for data overlap, and the spatial distribution characteristics of the target area. It is usually set between 50% and 90%. In this embodiment, it is set to 75% to ensure a high degree of spatial matching of the data while taking into account the availability of the data.

[0102] The preset similarity measurement method is an algorithm for calculating the similarity between data feature vectors. In this embodiment, cosine similarity is used, which can effectively measure the directional similarity between feature vectors while avoiding the influence of numerical scale on the calculation results.

[0103] The feature matching unit first calculates the similarity between each feature vector in the preprocessed dataset through the similarity calculation subunit and selects data with similarity exceeding the preset similarity threshold to form a similarity dataset. Subsequently, the time series matching subunit performs time matching on the similar dataset, ensuring that the time offset is within the preset time offset threshold to form a time series dataset. Next, the spatial matching subunit performs spatial matching on the time series dataset, selecting data with spatial overlap exceeding the preset overlap threshold to form a spatial dataset. Finally, the feature fusion subunit performs feature fusion on the spatial dataset to generate a matching dataset, ensuring the feature consistency and integrity of the multi-source data.

[0104] Through multi-level similarity, temporal, and spatial matching and fusion processing, the matching accuracy and consistency of remote sensing data from different sources are improved. Similarity calculation and similar data capture ensure the similarity of data features, temporal matching ensures the temporal relevance of data, and spatial matching enhances the spatial consistency of data. Furthermore, feature fusion further improves data integrity and usability, enabling the resulting matching dataset to better support subsequent data analysis and applications, enhancing the service value and reliability of remote sensing data.

[0105] Specifically, the threshold adjustment unit includes:

[0106] a quantity fluctuation calculation subunit, for calculating the standard deviation of the formed quantity to form a quantity fluctuation value;

[0107] An adjustment subunit is connected to the quantity fluctuation calculation subunit and is used to reduce the preset similarity threshold according to the relative deviation between the quantity fluctuation value and the maximum value of the preset quantity fluctuation range when the quantity fluctuation value is greater than the maximum value of the preset quantity fluctuation range to form an adjustment threshold group, or to increase the preset time offset threshold when the quantity fluctuation value is greater than the midpoint value within the preset quantity fluctuation range to form an adjustment threshold group, or to reduce the preset overlap threshold when the quantity fluctuation value is between the minimum value and the midpoint value of the preset quantity fluctuation range to form an adjustment threshold group.

[0108] The preset number fluctuation range is a predefined range used to measure fluctuations in the number of matching datasets. It is set based on the historical performance of the dataset and expected data fluctuations, and is typically set between 0.05 and 0.2. In this embodiment, it is set to 0.1 to 0.15, which balances tolerance for normal fluctuations while avoiding excessive threshold adjustments caused by excessive fluctuations, ensuring the stability and adaptability of the data matching process.

[0109] First, the quantity fluctuation calculation subunit calculates the standard deviation of the number of matching datasets to obtain the quantity fluctuation value. Then, the adjustment subunit dynamically adjusts the quantity fluctuation value based on its range: when the quantity fluctuation value exceeds the maximum value of the preset quantity fluctuation range, the preset similarity threshold is lowered to expand the matching range; when the quantity fluctuation value is above the midpoint value of the range but within the maximum value, the preset time offset threshold is increased to allow for greater temporal matching errors; when the quantity fluctuation value is between the minimum and midpoint values, the preset overlap threshold is lowered to relax the spatial matching requirements, ultimately forming a new set of adjustment thresholds.

[0110] By dynamically adjusting the matching threshold, the adaptability of matching datasets is enhanced, allowing matching rules to be optimized as data fluctuates, avoiding data matching failures or over-selection issues caused by fixed parameters. Furthermore, this method improves the flexibility and applicability of data integration while ensuring data quality, making data matching more stable and accurate, and applicable to a variety of complex application scenarios.

[0111] Please continue reading Figure 3 As shown, it is a flow chart of the hierarchical storage component forming a hierarchical storage data set in this embodiment;

[0112] The tiered storage component includes:

[0113] a detailed data storage unit, used for storing the standardized data that has not been aggregated and processed in the standardized data set to form the detailed data set;

[0114] a data intermediate layer storage unit connected to the detailed data storage unit, for cleaning, converting, multi-dimensionally aggregating and storing the detailed data set to form the intermediate processed data set;

[0115] A data service layer storage unit connected to the data intermediate layer storage unit for performing index optimization, partition storage, query acceleration, and storage on the intermediate processing data set to form the data service set;

[0116] A data application layer storage unit is connected to the data service layer storage unit and is used to extract and store visual analysis data and decision support data from the data service set based on the preset business requirements to form the application data set.

[0117] First, the detailed data storage unit stores the original standardized data that has not been aggregated to form a detailed data set; then, the data intermediate layer storage unit cleans, converts and multi-dimensionally aggregates this data to form an intermediate processing data set; then, the data service layer storage unit optimizes the index and accelerates the query of the intermediate processing data to form a data service set; finally, the data application layer storage unit extracts visual analysis data and decision support data from the data service set according to preset business needs to form an application data set.

[0118] The tiered storage approach not only optimizes data storage efficiency but also increases the flexibility and speed of data processing and access. The detailed data storage unit ensures data integrity, the intermediate storage unit enhances data availability, the data service layer storage unit ensures efficient querying and rapid response, and the application layer storage unit meets specific business needs for rapid data extraction and analysis. This effectively improves scalability, flexibility, and efficiency, allowing data at different levels to be managed and used appropriately according to specific needs.

[0119] Specifically, the status information synchronization component includes:

[0120] A state tree construction unit, configured to initialize each node in a preset state tree based on the data update time and storage level in the hierarchically stored data set to form the data state tree;

[0121] A status marking unit, configured to classify and mark the data according to the data update time to form a data status marking set;

[0122] A version control unit connected to the status marking unit, configured to assign a unique version number to each data state in the status marking data set, and to maintain version change records to form version management information;

[0123] A state synchronization unit is connected to the version control unit and is used to synchronize a number of version management information into the data state tree according to the version management information and a preset time window to form a versioned data set.

[0124] Data state refers to the information used to describe data at different points in time and during processing stages, including timestamps, version numbers, processing status, quality flags, storage hierarchy, and update times. Timestamps mark the time when data is generated or updated, version numbers distinguish different data versions, processing status reflects whether the data has undergone steps such as cleansing and conversion, quality flags indicate data quality, storage hierarchy displays the data's location in different storage tiers, and update times record the last time the data was updated. Together, these elements help manage and track changes in data throughout its lifecycle, ensuring data validity and consistency.

[0125] The state information synchronization component coordinates multiple units to ensure effective management of data status and version information. First, the state tree construction unit initializes each node based on the update time and storage hierarchy of the hierarchically stored dataset, forming a data state tree. Next, the state marking unit categorizes and marks the data based on its update time, forming a data state marking set. The version control unit assigns version numbers to the state marking dataset and maintains a record of version changes, forming version management information. Finally, the state synchronization unit synchronizes the version management information to the data state tree, forming a versioned dataset.

[0126] Through real-time tracking of data status and version control, data consistency, accuracy, and traceability are effectively guaranteed. Through precise version management and synchronization mechanisms, data consistency and stability can be ensured at all stages, improving reliability and transparency of data processing.

[0127] Please continue reading Figure 4 As shown, it is a decision logic diagram of the state synchronization unit in this embodiment to form a versioned data set;

[0128] Specifically, the state synchronization unit includes:

[0129] a change value calculation subunit, configured to calculate a difference between timestamps in the data state of a current version and a previous adjacent version in the version management information to form a data state change value;

[0130] The state synchronization subunit is connected to the change value calculation subunit and is used to synchronize the version management information to the data state tree to form a versioned data set when the data state change value is greater than the preset time window.

[0131] The preset time window refers to a pre-set time range or period during the data processing or synchronization process, which is used to determine and control the frequency of data status changes, synchronization or updates. It depends on the real-time requirements of the system, the frequency of data updates, and the needs of specific business scenarios. It is set between 10 seconds and 2 hours. In this embodiment, it is set to 1 hour. It can balance the real-time nature of data synchronization and the system load, avoid too frequent updates or synchronization operations, and ensure that the system can process data efficiently and stably.

[0132] The state synchronization unit uses the change value calculation subunit to calculate the timestamp difference between the current and previous versions of the version management information, forming a data state change value. When the data state change value exceeds a preset time window, the state synchronization subunit synchronizes the version management information to the data state tree, ensuring the update of the versioned dataset.

[0133] By calculating the timestamp difference between the current version and the previous version, changes in the data state can be effectively determined, ensuring that data status is only synchronized to the data state tree when there is a significant change. This not only avoids unnecessary frequent updates but also ensures efficient system operation. When the data state change value is greater than the preset time window, the system immediately synchronizes the version management information to the data state tree, ensuring that the data state promptly and accurately reflects the actual changes, thereby improving data consistency and accuracy. In this way, data management becomes more flexible and intelligent, able to dynamically adapt to different data update needs, reduce unnecessary storage and computing overhead, and improve the system's response speed and stability.

[0134] Specifically, the service building components include:

[0135] A requirement analysis unit, configured to analyze the preset multi-source data processing requirements to obtain functional requirements and interface requirements;

[0136] A data mapping unit, configured to uniformly map and convert the format of the original data, the standardized data, and the version control information in the versioned data set to form a data mapping table;

[0137] an interface design unit, connected to the demand analysis unit and the data mapping unit, respectively, for designing the unified service interface according to the functional requirements, the interface requirements, the data mapping table, and the preset business requirements to form a service interface design document;

[0138] An interface encapsulation unit is connected to the interface design unit and is used to perform encapsulation according to the service interface design document to construct the unified service interface and form the data service interface.

[0139] Preset multi-source data processing requirements refer to pre-set data processing requirements based on the specific needs and goals of multiple data sources. These requirements cover the rules and standards for how to acquire, process, fuse and present data from multiple different data sources (such as sensors, remote sensing equipment, databases, etc.) to ensure the quality, availability and consistency of the data. They depend on factors such as the system's business objectives, the type of data source, the complexity of data processing and real-time requirements.

[0140] First, the requirements analysis unit analyzes the preset multi-source data processing requirements to determine the functional requirements and interface requirements; then, the data mapping unit uniformly maps and formats the original data, standardized data, and version control information in the versioned data set according to the requirements to form a data mapping table; the interface design unit designs a unified service interface based on the functional requirements, interface requirements, data mapping table, and preset business requirements, and generates a service interface design document; finally, the interface encapsulation unit encapsulates the interface according to the service interface design document to construct a data service interface.

[0141] Requirements analysis and data mapping ensure the accuracy and compatibility of interface design, avoiding issues with inconsistent data formats. Through standardized interface design, the system provides users with consistent and flexible data access interfaces, improving the scalability and maintainability of data services. Furthermore, interface encapsulation simplifies service integration, enabling rapid response to changing business requirements and improving overall system performance and user experience.

[0142] Specifically, the data service components include:

[0143] Demand identification unit, used to identify users' query, browsing, statistics, downloading, subscription and customization needs, and form a user demand set based on the data request input by the user;

[0144] a demand processing unit connected to the demand identification unit, for processing and filtering the original data, standardized data and version control information to form a processed data set;

[0145] A data interaction unit is connected to the demand processing unit and is used to provide the processed data set through the constructed service interface to form a unified data service.

[0146] First, the demand identification unit identifies user data needs, such as query, browsing, statistics, downloading, subscription, and customization, generating a user demand set. The demand processing unit then filters and processes the raw data, standardized data, and version control information to form a processed data set. Finally, the data interaction unit provides the processed data set to users through a pre-built service interface, forming a unified data service.

[0147] Providing high-quality and high-availability data services through a clear screening and processing process improves the system's flexibility and response efficiency. The unified data service interface simplifies the user's operation process, reduces dependence on different data sources and service interfaces, and makes data access more efficient and convenient.

[0148] Specifically, the unified data service includes general data services and dedicated data services:

[0149] The general data services include metadata services, remote sensing data publishing services, remote sensing data subscription services and remote sensing data retrieval services;

[0150] The dedicated data services include auxiliary data services, calibration data services, optical remote sensing image data services and multispectral remote sensing image services.

[0151] General data services provide metadata services, remote sensing data publishing services, remote sensing data subscription services, and remote sensing data retrieval services, meeting users' needs for basic data query, publishing, subscription, and retrieval. Specialized data services offer more specialized services, including auxiliary data services, calibration data services, optical remote sensing image data services, and multispectral remote sensing image data services, focusing on data processing and application needs in specific fields.

[0152] General data services meet the basic needs of most users for remote sensing data, ensuring efficient and broad applicability of data access and management. Dedicated data services provide deeply customized support for specific fields and needs, enhancing the application value and accuracy of remote sensing data. This service classification structure enhances the system's flexibility and adaptability, providing optimized service solutions for diverse user needs and application scenarios.

[0153] Thus far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art may make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will fall within the scope of protection of the present invention.

Claims

1. A data bus based on multi-source remote sensing data processing, characterized in that: include: The unified data organization and management module includes an archiving and aggregation component, which is used to perceive the product data updated by each remote sensing business plug-in set in real time, and archive and aggregate the product data into the remote sensing data global resource pool to form an archive data set; an association integration component connected to the archive aggregation component for performing ETL processing on the archive data set to form a standardized data set; a hierarchical storage component connected to the association integration component for storing the standardized data set in the detailed data layer, the data intermediate layer, the data service layer, and the data application layer according to preset business requirements to form a hierarchical storage data set; a state information synchronization component connected to the hierarchical storage component, configured to construct a data state tree based on the hierarchical storage data set, and mark the data state in the data state tree and assign a version number to form a versioned data set; A data unified service module, connected to the data unified organization and management module, includes a service construction component for constructing a unified service interface based on preset multi-source data processing requirements and the original data, standardized data, and version control information in the versioned data set, thereby forming a data service interface; The hierarchical storage data set includes a detailed data set, an intermediate processing data set, a data service set and an application data set; The association integration component includes: A feature matching unit, configured to perform feature matching based on a feature vector of a preprocessed data set and a preset threshold value group to form a matching data set; a threshold adjustment unit connected to the feature matching unit, configured to adjust the preset threshold group according to the number of matching data sets formed within a preset time period to form an adjusted threshold group; The feature matching unit includes: a similar data capturing subunit, configured to capture corresponding pre-processed data in the pre-processed data set when the similarity between features is greater than a preset similarity threshold in the preset threshold group, to form a similar data set; a time series matching subunit, configured to perform temporal data matching based on the similar data set, and determine that the match is successful when the time offset of the data match is less than a preset time offset threshold in the preset threshold group, thereby forming a time series data set; a spatial matching subunit, configured to perform spatial data matching based on the time series data set, and determine that the matching is successful when the spatial overlap of the data matching is greater than a preset overlap threshold in the preset threshold group, thereby forming a spatial data set; a feature fusion subunit, configured to fuse the spatial features in the spatial dataset to form the matching dataset; The threshold adjustment unit includes: a quantity fluctuation calculation subunit, for calculating the standard deviation of the formed quantity to form a quantity fluctuation value; An adjustment subunit is connected to the quantity fluctuation calculation subunit and is used to reduce the preset similarity threshold according to the relative deviation between the quantity fluctuation value and the maximum value of the preset quantity fluctuation range when the quantity fluctuation value is greater than the maximum value of the preset quantity fluctuation range to form an adjustment threshold group, or to increase the preset time offset threshold when the quantity fluctuation value is greater than the midpoint value within the preset quantity fluctuation range to form an adjustment threshold group, or to reduce the preset overlap threshold when the quantity fluctuation value is between the minimum value and the midpoint value of the preset quantity fluctuation range to form an adjustment threshold group.

2. The data bus based on multi-source remote sensing data processing according to claim 1, characterized in that: The tiered storage component includes: a detailed data storage unit, used for storing the standardized data that has not been aggregated and processed in the standardized data set to form the detailed data set; a data intermediate layer storage unit connected to the detailed data storage unit, for cleaning, converting, multi-dimensionally aggregating and storing the detailed data set to form the intermediate processed data set; A data service layer storage unit connected to the data intermediate layer storage unit for performing index optimization, partition storage, query acceleration, and storage on the intermediate processing data set to form the data service set; A data application layer storage unit is connected to the data service layer storage unit and is used to extract and store visual analysis data and decision support data from the data service set based on the preset business requirements to form the application data set.

3. The data bus based on multi-source remote sensing data processing according to claim 2, characterized in that: The state information synchronization component includes: a state tree construction unit, configured to initialize each node in a preset state tree based on the data update time and storage level in the hierarchically stored data set to form the data state tree; A status marking unit, configured to classify and mark the data according to the data update time to form a data status marking set; A version control unit connected to the state marking unit, configured to assign a unique version number to each data state in the data state marking set, and maintain version change records to form version management information; A state synchronization unit is connected to the version control unit and is used to synchronize a number of version management information into the data state tree according to the version management information and a preset time window to form a versioned data set.

4. The data bus based on multi-source remote sensing data processing according to claim 3, characterized in that: The state synchronization unit includes: a change value calculation subunit, configured to calculate a difference between timestamps in the data state of a current version and a previous adjacent version in the version management information to form a data state change value; The state synchronization subunit is connected to the change value calculation subunit and is used to synchronize the version management information to the data state tree to form a versioned data set when the data state change value is greater than the preset time window.

5. The data bus based on multi-source remote sensing data processing according to claim 4, characterized in that: The service building components include: A requirement analysis unit, configured to analyze the preset multi-source data processing requirements to obtain functional requirements and interface requirements; A data mapping unit, configured to uniformly map and convert the format of the original data, the standardized data, and the version control information in the versioned data set to form a data mapping table; an interface design unit, connected to the demand analysis unit and the data mapping unit, respectively, for designing the unified service interface according to the functional requirements, the interface requirements, the data mapping table, and the preset business requirements to form a service interface design document; An interface encapsulation unit is connected to the interface design unit and is used to perform encapsulation according to the service interface design document to construct the unified service interface and form the data service interface.

6. The data bus based on multi-source remote sensing data processing according to claim 5, characterized in that: Also includes: A data service component, which is connected to the service construction component and is used to provide query, browsing, statistics, download, subscription and customization services of remote sensing data through the data service interface to form a unified data service; The data service components include: Demand identification unit, used to identify users' query, browsing, statistics, downloading, subscription and customization needs, and form a user demand set based on the data request input by the user; a demand processing unit connected to the demand identification unit, for processing and filtering the original data, standardized data and version control information to form a processed data set; A data interaction unit is connected to the demand processing unit and is used to provide the processed data set through the constructed service interface to form a unified data service.

7. The data bus based on multi-source remote sensing data processing according to claim 6, characterized in that: The unified data service includes general data service and dedicated data service: The general data services include metadata services, remote sensing data publishing services, remote sensing data subscription services and remote sensing data retrieval services; The dedicated data services include auxiliary data services, calibration data services, optical remote sensing image data services and multispectral remote sensing image services.

Images