An ar data cube format, system and method

Abstract

The application relates to the technical field of remote sensing data processing, and particularly discloses an ARD data cube format, a system and a method, which adopt a 'logical aggregation + TIFF / COG entity + central database description' architecture, store single-time-phase images as standard GeoTIFF / COG files, regard the standard GeoTIFF / COG files as independently stored cube grid slices, centrally manage the metadata of all the slices and data cubes (virtual folders) through a centralized metadata database, and maintain the logical aggregation relationship therebetween. The core of the application is that the strong consistency management and complex query of the metadata are realized through the database, and the logical aggregation of the data is realized through the virtual folder, so that the physical distributed storage of the data and the logical unified description and access are completely decoupled. The scheme has the standard format ecological compatibility, the efficient multi-dimensional retrieval capability, the fine data management support and the cloud native architecture adaptability, and is suitable for intelligent management of large-scale remote sensing analysis ready data.

Description

Technical Field

[0001] This invention relates to the field of remote sensing data processing technology, and more specifically, to a physically distributed, logically aggregated ARD data cube format, system, and method. Background Technology

[0002] With the rapid development of remote sensing technology, satellite Earth observation data exhibits significant characteristics of massive volume and multidimensionality (spatial, temporal, and spectral). To lower the barrier to entry for analyzing and using remote sensing data, raw remote sensing data needs to be processed into standardized "Analysis Ready Data" (ARD), making it directly usable for various remote sensing applications.

[0003] In existing technologies, ARD data organization formats are mainly divided into two categories: one is multidimensional data formats represented by NetCDF and HDF5. Although these formats natively support multidimensional data models, they have obvious defects in engineering applications. When processing massive amounts of ARD data in single-file storage mode, I / O performance bottlenecks easily occur, and their compatibility with the currently widely used GIS software ecosystem is limited, increasing the cost of data use. The other category is mature image formats represented by GeoTIFF and COG (Cloud Optimized GeoTiff, cloud-optimized geotagged image file format). These formats have strong compatibility and high user acceptance, and are the mainstream choice for remote sensing image storage. However, they lack the native ability to describe multidimensional spatiotemporal data structures, making it difficult to effectively organize and manage multi-temporal, multi-level, and multi-type ARD cube data, and failing to meet the needs of multidimensional retrieval and refined governance of large-scale remote sensing data.

[0004] In summary, the lack of an ARD data organization scheme that combines mature image format ecosystem compatibility, multi-dimensional spatiotemporal data structure description capabilities, and adaptability to distributed storage and parallel computing scenarios has become a key issue restricting the efficient management and intelligent application of large-scale ARD data. Summary of the Invention

[0005] In view of the above-mentioned technical problems in related technologies, the present invention provides an ARD data cube format, system and method that can solve the above problems.

[0006] To achieve the above-mentioned technical objectives, the technical solution of the present invention is implemented as follows: An ARD data cube format includes cube grid slices, ARD data cubes, and a centralized meta-database, wherein: The cubic grid slice is the smallest physical storage unit, which is a single-temporal, single-resolution image data that conforms to the preset geographic grid standard and is stored independently as a GeoTIFF or COG format file. The ARD data cube is the core logical aggregation unit. It is defined and described by metadata records in the centralized metadata database and is logically represented as a virtual folder. It is used to aggregate multiple cube grid slices that cover the same geographical grid range and are stored in a distributed environment. The centralized metadata database is used to centrally store and manage the descriptive metadata of the cube grid slices and ARD data cubes.

[0007] Furthermore, the filenames of the cubic grid slices follow a preset naming convention, which includes at least: grid type, grid code, imaging time, data center code, and product type identifier; the product type identifier includes the data source type used to distinguish the data source and the data processing type used to distinguish the data lifecycle.

[0008] Furthermore, the centralized metadata database maintains a metadata record for each cube grid slice. This metadata record includes at least: a unique slice identifier, the identifier of the associated ARD data cube, the physical storage address of the entity file, the imaging time, the product type identifier, and data processing lifecycle information.

[0009] Furthermore, the centralized metadata database is also used to define and store the metadata of the ARD dataset, which is aggregated from multiple ARD data cubes, and to maintain the logical inclusion relationship between the ARD dataset and the ARD data cube.

[0010] A data organization and management system in ARD data cube format includes a distributed object storage layer, a centralized metadata database, and a logical aggregation service layer, wherein: The distributed object storage layer is used to store cube grid slice entity files named according to a unified rule; The centralized metadata database is used to centrally store the complete metadata of all cube grid slices, ARD data cubes, and ARD datasets; In response to a logical access request to an ARD data cube or ARD dataset, the logical aggregation service layer retrieves the logical views and physical storage addresses of all associated cube grid slices by querying the centralized metadata database.

[0011] Furthermore, the logical aggregation service layer provides a RESTful API or GraphQL interface to receive user query requests and return the access addresses of the aggregated logical view and entity files.

[0012] A data management method for ARD data cube format includes the following steps: Import single-chip ARD grid slice data; The slice data is renamed according to the naming convention to obtain the cube grid slice entity file; The entity file is stored in a distributed object storage layer, and the physical storage address of the entity file is obtained; in the centralized metadata database, a metadata record corresponding to the entity file is created, and the metadata record includes at least the physical storage address, imaging time, and product type identifier; In the centralized metadata database, the metadata record logic is logically associated with its corresponding ARD data cube.

[0013] Furthermore, the step of creating a metadata record in the centralized metadata database also includes: recording the data processing lifecycle information of the entity file from the original data to the current state in the metadata record.

[0014] Furthermore, it also includes: generating corresponding display thumbnail files for the cube grid slice entity files, storing the thumbnail files in a distributed object storage layer, and supplementing the physical storage address into the metadata record corresponding to the slice file.

[0015] The beneficial effects of this invention are: Strong metadata consistency and efficient multidimensional retrieval: Centralized management of global metadata is achieved through a centralized metadata database, ensuring strong metadata consistency and solving the problems of inconsistent metadata synchronization and inefficient retrieval in multi-file mode; the database supports composite queries based on multiple dimensions such as spatiotemporal range, product type, processing lifecycle, and spatial reference, which greatly improves the retrieval efficiency of massive ARD data and meets the needs of rapid data location.

[0016] Complete decoupling of physical and logical storage: Cube grid slice entity files are distributed and stored in a decentralized manner according to storage strategies, while logical aggregation of data is achieved through virtual folders (ARD data cubes) defined by the database. This decoupling means that data migration and changes in storage strategies (such as hot and cold tiering and storage node expansion) do not require modification of the upper-level logical view and access interface, greatly improving the flexibility and maintainability of data storage.

[0017] High ecosystem compatibility and scalability: The underlying data entities still adopt the standard GeoTIFF / COG format, which is fully compatible with existing GIS and remote sensing data processing toolchains. They can be used directly without additional format conversion, reducing the cost of data application. The centralized metadata database model design supports flexible metadata field expansion, and management dimensions can be added according to business needs to adapt to the ARD data management needs of different scenarios.

[0018] Supports refined data governance: The database inherently supports multi-dimensional refined management of data, enabling precise control over dimensions such as data ownership (official / user), lifecycle status (native / derived / preprocessed), data usage (raw analysis / display preview), and access permissions, meeting the standardized governance needs of large-scale ARD data.

[0019] Perfectly adapted to cloud-native architecture: The distributed object storage layer supports elastic scaling and is adapted to massive data storage in cloud environments; the centralized meta-database can be deployed in a microservice manner, and the logical aggregation service layer provides standardized network interfaces. The three work together in line with the design concept of cloud-native architecture, supporting elastic scaling, high availability and distributed parallel computing of the system, and are suitable for building a modern cloud-native remote sensing data intelligent management platform. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of the logical architecture of the ARD data cube format data organization and management system according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the relationship between datasets, grid cubes, and slice metadata in the centralized metadata database described in this embodiment of the invention. Figure 3 This is a schematic diagram of the logical view and physical storage mapping of the "virtual folder" as described in an embodiment of the present invention; Figure 4 This is a flowchart illustrating the process of storing cube grid slices in a database according to an embodiment of the present invention. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention are within the scope of protection of the present invention.

[0023] This invention discloses an ARD data cube format, which adopts a logical aggregation and database description architecture: Data entity unit: A single-temporal, single-resolution image file conforming to geographic grid partitioning standards is used as the smallest storage unit, called a cube grid tile. This tile is stored independently in a distributed storage system using GeoTIFF or COG format.

[0024] Data Logical Unit: A centralized metadata database logically aggregates and unifies the description of all multi-temporal and multi-type (native and derived) cube grid slices covering the same standard geographic grid area, forming an ARD data cube. Logically, this cube is represented as a virtual folder defined by database entries, and all its associated slice entity files can be physically stored in different locations.

[0025] Data set unit: Multiple ARD data cubes are logically aggregated at a higher level from an application perspective through a centralized meta-database to form an ARD dataset.

[0026] The entity file naming of the cube grid slice contains structured information, and its naming can be as follows: Grid type_Grid encoding_Imaging time_Data center encoding_Product type identifier_Serial number_Brief description.File extension.

[0027] The product type identifier further includes information on data source type and data processing type.

[0028] The centralized metadata database is used to store and manage descriptive metadata for all ARD data cubes and cube grid slices. Each cube grid slice maintains an independent metadata record, which includes at least the slice's unique identifier, the associated ARD data cube identifier, the physical storage address of the entity file, the imaging time, the product type identifier, and the data processing lifecycle information. It can also be expanded to include spatial reference information, band information, global identifier information, storage time, etc. At the same time, the database also supports defining and storing metadata for ARD datasets aggregated from multiple ARD data cubes, maintaining the logical inclusion relationship between ARD datasets and ARD data cubes, and realizing multi-level logical aggregation of data.

[0029] This invention also discloses a data organization and management system for the ARD data cube format, including: Distributed object storage layer: Used for physically storing cube grid slice entity files (COG / TIFF format) named according to a unified naming rule. The physical storage path of the files can be optimized according to the storage strategy and is not constrained by logical relationships.

[0030] Centralized Metadatabase: Used for centralized storage and management of complete metadata for all cube grid slices, ARD data cubes (virtual folders), and ARD datasets. This database provides a highly consistent metadata operation interface and efficient multi-dimensional composite query capabilities.

[0031] Logical Aggregation Service Layer: Based on the centralized metadata database, this layer provides a logical view of the ARD data cube (virtual folder) and the ARD dataset. This layer responds to user queries for logical units and retrieves all associated entity slices and their storage locations from the database, enabling transparent access to logically aggregated entities.

[0032] This invention also discloses a data management method for ARD data cube format, comprising the following steps: receiving raw or processed single-piece ARD grid slice data; renaming the slice data according to a preset naming convention to form a cube grid slice entity file; storing the entity file in a determined location in a distributed object storage layer; generating a thumbnail file for displaying the slice as needed, storing it in the object storage layer for displaying and previewing the entity file; wherein, in the centralized metadata database, creating or updating a metadata record corresponding to the entity file, the record at least including the file physical address, imaging information, product type, and processing lifecycle; wherein, in the centralized metadata database, logically associating the metadata record of the entity file with its respective ARD data cube (virtual folder) and ARD dataset.

[0033] Example 1: like Figure 1 The diagram shown illustrates the logical architecture of an ARD data cube format data organization and management system, including: Distributed Object Storage Layer: Employing an S3-compatible object storage service, all TIFF / COG entity files and their thumbnails for cube grid tiles are stored here. File keys follow the rule / data type / user / grid / filename, for example, / OFF_RAW / T0 / J50E012016 / T0_J50E012016_20250701_HLJ_001_00_Second Quarter Raw Data.tif.

[0034] Centralized metadata database: This database uses a relational database (such as PostgreSQL / PostGIS) or a document-oriented database that supports JSON. It stores the core metadata of all logical entities.

[0035] Logical Aggregation Service Layer: Provides RESTful API or GraphQL interface to receive user queries for ARD data cubes (such as "get all data for the second quarter of 2025 for grid J50E012016") or ARD datasets, initiates complex queries to the metadata database, and returns the access addresses of the aggregated logical view and entity files.

[0036] Example 2: like Figure 2 As shown, the centralized metadata database mainly maintains the following logical tables or sets: The grid_cube table defines an ARD data cube (virtual folder) and contains fields such as cube_id, grid_id, description, and bbox. It also establishes relationships with multiple data_slice records through a join table.

[0037] The data_slice table defines each cube grid slice, containing slice_id, cube_id (foreign key, pointing to the virtual folder it belongs to), storage_uri (object storage address), imaging_time, product_type, authority_type, lineage (JSON field), bands (JSON field), etc.

[0038] The dataset table defines the ARD dataset and establishes many-to-many relationships with multiple grid_cube records through association tables.

[0039] The cube-slice-rel and dataset-cub-rel tables are association tables between data cubes and cube grid slices, and between datasets and cube grid slices.

[0040] Example 3: like Figure 3 As shown, when a user queries the virtual folder T0_J50E012016 through the logical aggregation service layer, the service layer queries the database to obtain a logical list of all slices under that folder (including metadata such as filenames and timestamps), as well as the physical storage_uri and browse_uri corresponding to each slice. Users or applications can directly access the entity files in the object storage based on these URIs, achieving transparent access that is logically unified and physically distributed.

[0041] Example 4: The data entry process for this application is as follows: Figure 4As shown, the process includes the following steps: a) Receive the single ARD grid slice data to be stored; b) Generate the final filename according to the naming convention; c) Upload the entity file to the object storage layer to obtain its unique storage_URI; d) Generate a display thumbnail for the entity file, upload it to object storage, and obtain its unique storage_URL; e) In the centralized metadata database, insert a new record into the data_slice table, fully populate the metadata of the slice, and set the storage_uri field to the browse_URL obtained in step c, and the browse_uri field to the browse_URL obtained in step d; f) In the database, logically associate the data_slice record with its corresponding ARD data cube (virtual folder) by setting the cube_id foreign key. Similarly, it can be associated with the corresponding ARD dataset.

[0042] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. An ARD data cube format, characterized in that, This includes cube grid slices, ARD data cubes, and a centralized meta-database, among which: The cube grid slice is the smallest physical storage unit. It is a single-temporal, single-resolution image data that conforms to the preset geographic grid standard and is stored independently as a GeoTIFF or COG format file. The ARD data cube is the core logical aggregation unit. It is defined and described by metadata records in the centralized metadata database. Logically, it is represented as a virtual folder and is used to aggregate multiple cube grid slices that cover the same geographical grid range and are stored in a distributed environment. The centralized metadata database is used to centrally store and manage the descriptive metadata of the cube grid slices and ARD data cubes.

2. The ARD data cube format according to claim 1, characterized in that, The filenames of the cubic grid slices follow a preset naming convention, which includes at least: grid type, grid code, imaging time, data center code, and product type identifier; the product type identifier includes the data source type used to distinguish the data source and the data processing type used to distinguish the data lifecycle.

3. The ARD data cube format according to claim 1, characterized in that, The centralized metadata database maintains a metadata record for each cube grid slice. This metadata record includes at least: a unique slice identifier, the identifier of the associated ARD data cube, the physical storage address of the entity file, the imaging time, the product type identifier, and data processing lifecycle information.

4. The ARD data cube format according to claim 1, characterized in that, The centralized metadata database is also used to define and store the metadata of the ARD dataset, which is formed by aggregating multiple ARD data cubes, and to maintain the logical inclusion relationship between the ARD dataset and the ARD data cube.

5. A data organization and management system in ARD data cube format as described in any one of claims 1 to 4, characterized in that, It includes a distributed object storage layer, a centralized metadata database, and a logical aggregation service layer, among which: The distributed object storage layer is used to store cube grid slice entity files named according to a unified rule; The centralized metadata database is used to centrally store the complete metadata of all cube grid slices, ARD data cubes, and ARD datasets; In response to a logical access request to an ARD data cube or ARD dataset, the logical aggregation service layer retrieves the logical views and physical storage addresses of all associated cube grid slices by querying the centralized metadata database.

6. The data organization and management system according to claim 5, characterized in that, The logical aggregation service layer provides a RESTful API or GraphQL interface to receive user query requests and return the access addresses of the aggregated logical view and entity files.

7. A data management method for ARD data cube format as described in any one of claims 1 to 4, characterized in that, Includes the following steps: Import single-chip ARD grid slice data; The slice data is renamed according to the naming convention to obtain the cube grid slice entity file; The entity file is stored in the distributed object storage layer, and the physical storage address of the entity file is obtained; In the centralized metadata database, a metadata record corresponding to the entity file is created. This metadata record includes at least the physical storage address, imaging time, and product type identifier. In the centralized metadata database, the metadata record logic is logically associated with its corresponding ARD data cube.

8. The data management method according to claim 7, characterized in that, The steps of creating a metadata record in a centralized metadata database also include: recording the data processing lifecycle information of the entity file from its original data to its current state in the metadata record.

9. The data management method according to claim 6 or 7, characterized in that, Also includes: Generate corresponding thumbnail files for display for the cube grid slice entity files, store the thumbnail files in the distributed object storage layer, and supplement the physical storage address into the metadata record corresponding to the slice file.

Images