A multi-source ocean environment data storage method and system based on HDF5

Abstract

The present invention belongs to the technical field of data storage, and discloses a multi-source marine environment data storage method and system based on HDF5. The method constructs a hierarchical storage framework, constructs a domain-type-source three-level directory architecture, implements a dynamic expansion mechanism of spatiotemporal dimensions, and spatiotemporal composite coding; designs an HDF5 nested group storage architecture, including grid topology definition, dynamic resolution adaptive storage, multi-source data processing, and metadata management; constructs distributed data association and indexing, including data standardization preprocessing and cross-source data association; optimizes and expands storage performance, including density-sensitive blocking and hierarchical compression. The present invention significantly reduces redundant storage in high-latitude areas and supports dynamic adaptation of locally encrypted grids. The present invention contributes to the open circulation and collaborative use of interdisciplinary data resources.

Description

Technical Field

[0001] The present invention belongs to the technical field of data storage, and in particular relates to a multi-source ocean environment data storage method and system based on HDF5. Background Art

[0002] With the rapid development of marine science and environmental monitoring technology, scientific research institutions around the world have built a massive data collection network covering multiple disciplines such as atmosphere, ocean, geology, and ecology. The data types span multimodal forms such as numerical simulation, field observation, satellite remote sensing, AI inversion, and video surveillance. However, the integration and management of multi-source data face three core contradictions. First, data standards are fragmented among research institutes. For example, the NetCDF-CF convention is widely used in the atmospheric field, while ocean buoy data mostly follows the WMO binary format, and environmental monitoring data relies on the ISO 19115 metamodel. These differences in standard systems result in a significant amount of time spent on format conversion and semantic alignment when sharing data across institutions. Second, data types are significantly heterogeneous. The storage requirements of scientific data (such as temperature and salinity profiles), images (such as satellite cloud images), videos (such as deep-sea exploration images), and AI-generated products (such as wave prediction tensors) vary. Traditional solutions rely on decentralized storage systems (such as databases for numerical values ​​and object storage for images), exacerbating the data silo effect. Third, the spatial specificity of Earth spherical data (such as meridian convergence and latitude dependence of resolution) conflicts with the rigid structure of traditional storage formats. For example, when storing 0.1° resolution data at the equator on a regular grid, a large amount of redundant storage is generated in high-latitude regions due to the sudden decrease in grid cell area, and it is incompatible with the locally refined grid required for eddy tracking.

[0003] The current mainstream technical solutions have significant limitations in addressing the above challenges: (1) Although relational databases support structured metadata management, they are difficult to support the storage needs of irregular grids and high-dimensional time series data, and their scalability is limited by fixed table structures; (2) Although scientific data formats such as NetCDF have the ability to store multidimensional arrays, their preset grid models (such as regular latitude and longitude grids) cannot dynamically adapt to irregular spherical topologies (such as adaptive encrypted grids or AI-generated unstructured grids), and lack native support for multimodal data such as images and videos; (3) Emerging cloud-native storage optimizes query performance through columnar storage, but lacks a dedicated indexing mechanism for earth spherical data, resulting in low spatiotemporal query efficiency and easy spatial distortion when retrieving cross-latitude data. The contradiction between data organization granularity, spatial expression capabilities and multi-source compatibility in the above methods has become a key obstacle to the sharing of atmospheric and marine environmental data in my country.

[0004] A prior art method for storing and managing multi-resolution, block-based, stacked grids based on HDF5 proposes a hierarchical storage architecture that combines regular grids with adaptively encrypted grids. This architecture implements multi-resolution data management by constructing a multi-level Kd-tree index. While this method optimizes spatial query efficiency, its storage model still relies on fixed topological rules, making it incompatible with the spatial irregularities of satellite remote sensing and buoy-based data. Furthermore, it fails to define the semantics of the spherical grid.

[0005] Through the above analysis, the problems and defects of the existing technology are as follows:

[0006] Problem 1: Fragmented data standards lead to inefficient data sharing. Data standards differ significantly between research institutions (e.g., the NetCDF-CF convention, the WMO binary format, and the ISO 19115 metamodel). Incompatibility between atmospheric and ocean data standards complicates data integration and hinders multidisciplinary collaborative research.

[0007] Problem 2: Multi-source, heterogeneous data storage creates a serious silo effect. Scientific data, images, videos, and AI-generated products rely on decentralized storage systems due to varying storage requirements. Traditional solutions lack a unified storage framework, exacerbating data silos. For example, numerical simulation data and remote sensing imagery must be managed separately, making cross-modal data correlation analysis difficult.

[0008] Problem 3: Data storage redundancy and topological conflicts on the Earth's spherical surface. Regular grid storage methods generate up to 45% redundant storage in high-latitude regions due to the sudden decrease in grid cell area, and are unable to adapt to locally refined grids. Fixed latitude and longitude grids cannot dynamically respond to unstructured grids or adaptive refinement requirements, resulting in an imbalance between spatial representation accuracy and storage efficiency.

[0009] Problem 4: Traditional storage technologies lack adaptability. Relational databases are limited by fixed table structures and struggle to support the storage of irregular grids and high-dimensional time series data. While scientific data formats such as NetCDF support multidimensional arrays, their pre-set grid models cannot dynamically adapt to unstructured grids generated by AI and lack compatibility with multimodal data. Summary of the Invention

[0010] In order to overcome the problems existing in the related art, the disclosed embodiments of the present invention provide a multi-source marine environment data storage method and system based on HDF5. The purpose of the present invention is to propose an adaptive storage method for multi-source marine environment data based on HDF5, by constructing a storage framework with dynamic decoupling of domain-type-source, and designing an ontology expression model of spherical irregular grids, to break through the technical barriers of multi-source data integration and accurate representation of earth space. The method innovatively introduces a spatiotemporal composite coding mechanism and a hybrid grid unified interface, supports the automated alignment and efficient retrieval of cross-scale, multimodal data, and provides a highly compatible, low-redundant integrated data base for ocean digital twins and data sharing.

[0011] The technical solution is as follows: a multi-source marine environment data storage method based on HDF5, comprising the following steps:

[0012] S1: Build a hierarchical logical catalog and coding system. Based on the domain-type-source principle, a three-level catalog structure is constructed. This system combines a dynamic expansion mechanism for spatiotemporal dimensions with a spatiotemporal composite coding method to form a unified data logical organization system. This provides a catalog index foundation and refined spatiotemporal expression for subsequent data storage and retrieval.

[0013] S2: Design storage structures and multi-source adaptation mechanisms. Based on the constructed directory and spatiotemporal coding system, design an HDF5 nested group storage structure to support topological definition storage of regular and irregular grids, dynamic resolution adaptive storage, and multimodal data storage of images and videos. Build a metadata management mechanism to ensure data description consistency and cross-source compatibility.

[0014] S3: Build a data indexing and association mechanism for multi-source collaboration. To meet the integration needs of multi-source heterogeneous data, design a standardized preprocessing process, unify the time base and physical unit system, and build a distributed spatiotemporal indexing mechanism to achieve accurate matching and efficient linkage query of cross-source data.

[0015] S4: Implement performance optimization and storage compression strategies; after completing logical organization, physical storage, and data indexing, introduce density-sensitive blocking and multi-level compression strategies, dynamically adjust the blocking method based on grid density and access patterns, and use ZFP and JPEG2000 compression algorithms to achieve high-performance, high-compression ratio multi-source data storage optimization.

[0016] In step S1, a three-level directory structure is constructed with field-type-source as the main line, including: creating multiple main groups under the root directory to represent the attributes of the subject field; establishing subgroups of different types of products under each field group; grouping by data source within the type subgroup to determine the type of data acquisition equipment.

[0017] In step S1, the spatiotemporal dimension dynamic expansion mechanism includes:

[0018] Timeline organization: Create a time series directory by nesting / Year / Month / Day within the data source group; store the timestamp array in / Index / TimeIndex, and establish a mapping relationship between the time directory and the dataset;

[0019] Spherical spatiotemporal hybrid indexing introduces a latitude-dependent resolution function to dynamically adjust the grid cell size and generates a unique grid code using the H3 geographic gridding algorithm. A spatiotemporal R-tree index structure is embedded in the HDF5 file, where the index node contains the time range, spatial grid code, and data block physical address.

[0020] H3 hexagonal grid coding uses the H3 geographic grid to divide the sphere into multi-resolution hexagonal cells;

[0021] Hierarchical spatiotemporal coding dynamically selects the coding granularity and dimension according to the spatiotemporal characteristics of the data. At the same time, for the high-dimensional data of ocean and atmospheric profile data, the vertical dimension identifier is embedded in the coding to deeply associate with the ocean-air profile data.

[0022] Furthermore, the latitude-dependent resolution function is:

[0023] ;

[0024] Where, For latitude resolution, is the equatorial resolution, is the polar resolution, Switch threshold for resolution;

[0025] Generate a unique grid code using the H3 geo-grid algorithm, including:

[0026] (1) Convert latitude and longitude to three-dimensional coordinates;

[0027] ;

[0028] ;

[0029] ;

[0030] Where, is the longitude;

[0031] (2) Determine the resolution level, which includes 0-15 levels, based on the latitude-dependent resolution function;

[0032] (3) Calculate the hexagon center and calculate the center position of the hexagon formed by multiple nodes;

[0033] Assume the latitude and longitude coordinates of the hexagon vertices are:

[0034] ;

[0035] Then the geometric center for:

[0036] ;

[0037] ;

[0038] Where, and Hexagonal The longitude and latitude of each vertex;

[0039] (4) Generate H3 code and generate a 64-bit H3 index based on the center position of the hexagon.

[0040] In step S2, the topology definition includes: storing coordinate axes in a regular grid as independent datasets, linked to the main dataset through attributes;

[0041] The irregular grid stores the three-dimensional coordinate matrix of each node and links it to the main dataset through attributes. The irregular grid creates the `Adjacency_List` dataset, which uses a sparse matrix format to record the adjacent cell index of each grid cell. The grid_type field is defined in the dataset properties and the grid type description is attached. An independent Group is created for each time step to store the grid topology and geometry data of that time step.

[0042] In step S2, dynamic resolution adaptive storage is performed, including: storing the basic coordinates of the grid nodes in the form of a three-dimensional floating-point array, declaring the coordinate system standard in the attributes; creating an auxiliary dataset under the same group, storing the plane projection coordinates, and annotating the projection method; recording the fixed resolution for regular grids, dividing the resolution identifier by region for irregular grids, and recording the difference between the equator and the poles in the attributes;

[0043] Multi-source data processing includes: regular grids are directly stored as multi-dimensional arrays; irregular grids are linked to attributes through independent coordinate data sets; image data is decoded into pixel arrays; video data is used as a multi-dimensional array of time dimension;

[0044] The metadata management includes designing basic metadata of data sets for different types of data. The basic metadata of data sets include scientific data metadata, image data metadata, and video data metadata.

[0045] In step S3, a standardized preprocessing process is designed to meet the needs of integrating multi-source heterogeneous data and unify the time base and physical unit system, including:

[0046] Design a standardized preprocessing process to uniformly convert raster data of satellite images in different formats, resample the time series data of buoys to a unified time base unit system and normalize them; define a standard unit conversion rule library and declare it through the attribute `unit` field;

[0047] Achieve accurate matching and efficient linkage query of cross-source data, including spatiotemporal alignment of cross-source data and spatiotemporal matching of multi-source data.

[0048] In step S4, the density-sensitive block partitioning includes: dynamically adjusting the HDF5 block size according to the grid node density, dynamically adjusting the block dimension according to the data access mode, and partitioning the time series data by time axis; and partitioning the spatial raster data by latitude and longitude.

[0049] Achieve high-performance, high-compression ratio multi-source data storage optimization, including: using the ZFP floating-point compression algorithm for floating-point scientific data and lossless compression for coordinate data; and applying JPEG2000 lossy compression for image / video data.

[0050] Furthermore, node density is defined as:

[0051] ;

[0052] Where, is the total number of nodes in the current region, is the area of ​​the region;

[0053] Chunk size is defined as:

[0054] ;

[0055] Another object of the present invention is to provide a multi-source ocean environment data storage system based on HDF5, which implements the multi-source ocean environment data storage method based on HDF5. The system includes:

[0056] The hierarchical storage framework construction module is used to build a hierarchical logical directory and coding system. Based on the domain-type-source principle, a three-level directory architecture is constructed. This combines a dynamic expansion mechanism for spatiotemporal dimensions with a spatiotemporal composite coding method to form a unified data logical organization system, providing a directory index foundation and refined time-space expression for subsequent data storage and retrieval.

[0057] The HDF5 nested group storage architecture design module is used to design storage structures and multi-source adaptation mechanisms. Based on the constructed directory and spatiotemporal coding system, the HDF5 nested group storage structure is designed to support topological definition storage of regular and irregular grids, dynamic resolution adaptive storage, and multimodal data storage of images and videos. The metadata management mechanism is built to ensure data description consistency and cross-source compatibility.

[0058] A distributed data association and indexing module is used to build data indexing and association mechanisms for multi-source collaboration. To meet the integration needs of multi-source heterogeneous data, a standardized pre-processing process is designed, time bases and physical unit systems are unified, and a distributed spatiotemporal indexing mechanism is constructed to achieve accurate matching and efficient linkage query across data sources.

[0059] The optimization and expansion module is used to implement performance optimization and storage compression strategies. After completing logical organization, physical storage, and data indexing, density-sensitive blocking and multi-level compression strategies are introduced. The blocking method is dynamically adjusted according to grid density and access patterns. The ZFP and JPEG2000 compression algorithms are used to achieve high-performance, high-compression ratio multi-source data storage optimization.

[0060] In combination with all the above technical solutions, the beneficial effects of the present invention are as follows:

[0061] First, this invention addresses the bottlenecks of existing marine environmental data storage technologies, including fragmented multi-source data standards, redundant spatial representation, insufficient storage architecture adaptability, and inefficient spatiotemporal retrieval. By proposing an adaptive storage method for multi-source marine environmental data based on HDF5, this method addresses the contradictions between data organization granularity, spatial topology adaptability, and multi-source compatibility in traditional storage solutions by constructing a storage framework that dynamically decouples domains, types, and sources, designing an ontology expression model for spherical irregular grids, and integrating a spatiotemporal composite coding mechanism with a unified hybrid grid interface.

[0062] Second, compared with the existing technology, the present invention improves the integration efficiency and spatial representation accuracy of multi-source marine environmental data by constructing a dynamically decoupled hierarchical storage framework and a spherical grid ontology expression model. Through the four-level directory architecture of domain-type-source-modality, unified management and rapid retrieval of interdisciplinary and multimodal data are achieved, effectively solving the data island problem caused by standard fragmentation and storage dispersion in traditional solutions. At the same time, through the spatiotemporal composite coding mechanism and latitude adaptive resolution design, the redundant storage in high-latitude areas is greatly reduced, and the dynamic adaptation of local encrypted grids is supported. The present invention provides an efficient data sharing base for global marine research institutions and industry applications, which is conducive to the open circulation and collaborative use of interdisciplinary data resources.

[0063] Third, the adaptive storage method for multi-source marine environmental data proposed in this invention boasts excellent engineering feasibility and interdisciplinary adaptability. This conversion significantly reduces redundant storage in high-latitude regions and significantly improves the integration efficiency and retrieval performance of multi-source data. It is suitable for use in national marine data centers, scientific research institutions, and ship meteorological systems. It holds broad commercial application prospects and promotional value in key areas such as smart oceans, maritime power strategies, climate change monitoring, and offshore resource development.

[0064] Fourth, current mainstream international marine environmental data management solutions still rely on fixed-structure NetCDF or traditional databases. These solutions lack unified support for irregular grids, heterogeneous modalities (images, videos, AI prediction tensors, etc.), and spherical coordinate systems, and lack an index structure deeply coupled with the Earth's spatial characteristics. This paper introduces a spherical hybrid grid representation mechanism based on H3 hexagonal coding for the first time, and implements a cross-modal, cross-source dynamically decoupled directory design and index compression mechanism. This represents a fundamental breakthrough in supporting the storage and query of complex marine data, filling a technological gap in data organization and coding in this field. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure;

[0066] Figure 1 This is a flow chart of a multi-source marine environment data storage method based on HDF5 provided by an embodiment of the present invention;

[0067] Figure 2 This is a diagram of the domain-type-source three-level directory architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] To make the above-mentioned objects, features, and advantages of the present invention more readily apparent, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. The following description sets forth numerous specific details to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art may make similar modifications without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0069] The innovation of the present invention lies in: by constructing a multi-level directory system with dynamic decoupling of domains, types and sources, and combining spherical adaptive resolution functions with H3 hexagonal grid coding, the present invention proposes a spatiotemporal composite index and multimodal unified storage mechanism; the system supports the coexistence of regular and irregular grids, adaptive access and compression optimization of multi-source data, and realizes high-latitude redundancy reduction, efficient cross-source retrieval and integrated management of complex environmental data, providing technical support for the sharing and intelligent application of marine environmental information.

[0070] Dynamically decoupled hierarchical storage framework: Constructing a four-level directory architecture of domain-type-source-modality, and achieving unified management and efficient retrieval of interdisciplinary and multimodal data through dynamic expansion of spatiotemporal dimensions and spatiotemporal composite coding;

[0071] Spherical Grid Ontology Expression Model: Design a unified hybrid grid interface and latitude-adaptive resolution mechanism, integrating regular and irregular grid topology definitions to resolve the conflict between the spatial specificity of Earth spherical data and the rigid structure of traditional storage formats;

[0072] Distributed data association and performance optimization: By combining density-sensitive block and layered compression technology with a cross-source spatiotemporal alignment algorithm, storage efficiency is improved and high-dimensional redundancy is reduced, supporting dynamic adaptation and rapid access to unstructured data and high-dimensional time series data.

[0073] Example 1, as Figure 1 As shown, the multi-source marine environment data storage method based on HDF5 provided by the embodiment of the present invention is:

[0074] S1: Build a hierarchical logical catalog and coding system. Based on the domain-type-source principle, a three-level catalog structure is constructed. This system combines a dynamic expansion mechanism for spatiotemporal dimensions with a spatiotemporal composite coding method to form a unified data logical organization system. This provides a catalog index foundation and refined spatiotemporal expression for subsequent data storage and retrieval.

[0075] For example, a three-level directory structure of domain-type-source is constructed. Multiple main groups are created under the root directory to represent the attributes of the subject domain; subgroups such as different types of products are established under each domain group; and data sources are grouped within the type subgroup to clarify the type of data collection equipment. Figure 2 The domain-type-source three-level directory architecture diagram shown;

[0076] Constructing the domain-type-source three-level directory structure specifically includes:

[0077] Domain-level division: Create the / Meteorology, / Oceanography, / Environmental, and / Geography main groups in the root directory to represent the attributes of the subject areas. / Meteorology is the meteorological domain, covering observation, simulation, and forecast data. / Oceanography is the ocean domain, including measured profiles, numerical simulations, and forecast products. / Environmental is the environmental domain, managing data such as air quality, water quality, and soil quality. / Geography is the geographic information domain, integrating remote sensing imagery, GIS analysis, and urban modeling data.

[0078] Type layer division: Under each domain group, establish subgroups called ` / Observation`, ` / Simulation`, ` / Forecast`, ` / Image`, and ` / Video`. Observation refers to observation data (including buoy, ship, and satellite observations). Simulation refers to numerical simulation data (including WRF, ROMS, and WW3). Forecast refers to forecast products (including fractional numerical models and AI models). Image refers to image data, and Video refers to video data.

[0079] Source layer division: Within the type subgroup, data are grouped by ` / Satellite`, ` / Buoy`, ` / Radar`, ` / Ship`, ` / AI`, and ` / NWP` sources to clarify the type of data collection equipment. Satellite refers to satellite remote sensing data, Buoy refers to buoy observation data, Radar refers to radar observation data, Ship refers to ship measurement data, AI refers to artificial intelligence forecast data, and NWP refers to numerical model simulation data.

[0080] An exemplary, dynamic expansion mechanism of space-time dimensions.

[0081] Create a time series directory within the data source group; for datasets covering global oceans, divide spatial subgroups by latitude and longitude ranges, with each subgroup storing corresponding regional data; design a spatiotemporal joint coding directory for high-frequency updated data, integrating spatial location and collection time.

[0082] The dynamic expansion mechanism of the spatiotemporal dimension specifically includes the following contents:

[0083] Timeline organization: Create a time series directory within the data source group by nesting / Year / Month / Day (four-level expansion of year / month / day / hour). Store a timestamp array (datetime64 format) in / Index / TimeIndex, and establish a mapping relationship between the time directory and the dataset.

[0084] Spherical Spatiotemporal Hybrid Index: To address the convergence of longitude and latitude grids in high-latitude regions when the Earth's spherical surface is partitioned, a latitude-dependent resolution function is introduced to dynamically adjust the grid cell size. Unique grid codes are generated using the H3 geographic gridding algorithm. A spatiotemporal R-tree index structure is embedded in the HDF5 file. Index nodes contain the time range, spatial grid code, and data block physical address.

[0085] The latitude-dependent resolution function is:

[0086] ;

[0087] Where, For latitude resolution, is the equatorial resolution, is the polar resolution, Switch threshold for resolution;

[0088] H3 Hexagonal Grid Encoding: The H3 geographic grid is used to divide the sphere into multi-resolution hexagonal cells. The H3 geographic grid reduces high-latitude data redundancy through multi-resolution division of hexagonal cells and adapts to the depth of the Earth's spherical surface characteristics. The unique code of each grid is generated by the following steps:

[0089] (1) Convert latitude and longitude to three-dimensional coordinates:

[0090] ;

[0091] ;

[0092] ;

[0093] Where, is the longitude;

[0094] (2) Determine the resolution level. The resolution level (0-15) is determined based on the latitude-dependent resolution function. The higher the level, the finer the grid.

[0095] (3) Calculate the center of the hexagon. Calculate the center position of the hexagon formed by multiple nodes.

[0096] (4) Generate H3 code. Generate a 64-bit H3 index based on the center position of the hexagon.

[0097] Hierarchical spatiotemporal coding: The coding granularity and dimension are dynamically selected based on the spatiotemporal characteristics of the data. High-frequency data, such as satellite second-level observations, uses the "Geohash_xxxx_YYYYMMDDHHMMSS" format, with timestamp accuracy down to the second level and geohash accuracy increased to 6 digits (approximately 0.6 km resolution). Low-frequency data, such as monthly mean simulations, has its timestamp simplified to "YYYYMM," with geohash accuracy reduced to 4 digits (approximately 20 km resolution). Furthermore, for high-dimensional data, such as ocean and atmospheric profiles, vertical dimension identifiers are embedded in the coding to enable deep correlation of ocean-air profile data.

[0098] S2: Design storage structures and multi-source adaptation mechanisms. Based on the constructed directory and spatiotemporal coding system, design an HDF5 nested group storage structure to support topological definition storage of regular and irregular grids, dynamic resolution adaptive storage, and multimodal data storage of images and videos. Build a metadata management mechanism to ensure data description consistency and cross-source compatibility.

[0099] For example, a grid topology definition includes: Regular grids store coordinate axes as independent datasets, linked to the main dataset via attributes. Irregular grids store the three-dimensional coordinate matrix of each node, linked to the main dataset via attributes. Irregular grids create an `Adjacency_List` dataset, which uses a sparse matrix format to record the neighboring cell indices of each grid cell. A grid_type field is defined in the dataset properties, along with a grid type description. A separate Group is created for each time step to store the grid topology and geometry data for that time step. Only the differences from the previous time step are stored to reduce redundancy.

[0100] Exemplarily, the grid topology definition specifically includes:

[0101] Basic Coordinate Storage: A regular grid stores the three-dimensional coordinate axes (longitude, latitude, depth / pressure) as independent one-dimensional arrays: / Grid / Coordinates / Latitude (latitude array, float32, shape N, where N is the latitude array length); / Grid / Coordinates / Longitude (longitude array, float32, shape M, where M is the longitude array length); and / Grid / Coordinates / Depth (depth array, float32, shape K, where K is the depth array length). This structured storage improves data access efficiency and reduces the complexity of multidimensional data parsing. An irregular grid stores the three-dimensional coordinates of each node as a matrix (float32, shape Px3, where P is the total number of nodes). Basic coordinates are linked to the main dataset via attributes.

[0102] Adjacency Relationship Description: For irregular grids, attribute fields in the main dataset point to the adjacency list dataset. A sparse matrix format (CSR) is used to record the global index of adjacent cells (int32, shape Q, where Q is the total number of adjacent cell indices in the adjacency list) and the starting position of each cell's adjacency index (int32, shape P+1, where P is the total number of cells). This reduces redundant space usage, allows for faster location of adjacent cells, and optimizes topology query performance.

[0103] Metadata mapping: Define the grid_type field (RegularGrid or IrregularGrid) in the dataset properties and attach a grid type description to simplify data classification and processing logic.

[0104] Dynamic Mesh Deformation: Creates a separate group for each time step to store the mesh topology and geometry data for that time step. Only the geometric differences from the previous time step are stored, reducing duplicate storage and lowering dynamic data storage overhead.

[0105] Exemplarily, dynamic resolution adaptive storage involves storing the base coordinates of grid nodes as a 3D floating-point array, with the coordinate system standard declared in attributes. An auxiliary dataset is created under the same group, storing the planar projection coordinates and annotating the projection method. Regular grids record fixed resolution, while irregular grids use regional resolution identifiers, with the difference between the equator and the poles recorded in attributes.

[0106] Specifically, dynamic resolution adaptive storage includes spherical coordinate body storage, projected coordinate additional storage, and hierarchical resolution annotation;

[0107] The spherical coordinate ontology storage includes:

[0108] 1) Spherical 3D floating-point array structure definition. The longitude, latitude, and altitude / pressure of each spherical grid node are stored sequentially as a 3D floating-point array with a shape of (N,3), where N is the total number of nodes. If the node coordinates are high-precision data (such as satellite positioning data), float64 (double precision) is used; if they are conventional observation data (such as buoys), float32 (single precision) is used. The precision type is specified using the precision attribute.

[0109] 2) Spherical coordinate system standard labeling. Declare the coordinate system standard in the dataset properties in the format of "standard name (EPSG code)". For example:

[0110] Attributes: {"crs": "WGS84 (EPSG:4326)"}

[0111] For special coordinate systems such as CGCS2000, the ellipsoid information is expanded in the properties:

[0112] Attributes: {

[0113] "crs": "CGCS2000 (EPSG:4490)",

[0114] "ellipsoid": "GRS80",

[0115] "semi_major_axis": 6378137.0,

[0116] "inverse_flattening": 298.257222101

[0117] }

[0118] The additional storage of projection coordinates includes: creating an auxiliary dataset in the same group as the spherical coordinates, storing planar projection coordinates such as UTM and Web Mercator, marking the projection method through the attribute projection_method, and storing the projected planar coordinates (x, y) in a one-to-one correspondence with the spherical coordinates.

[0119] The hierarchical resolution annotation includes: regular grids record fixed resolution in the main dataset attributes. Irregular grids divide resolution into regions (such as / High_Res_0.1deg, / Low_Res_1.0deg) and declare resolution gradients in the attributes (such as 0.1° at the equator, 1.0° at the poles).

[0120] Exemplarily, multi-source data processing includes: regular grids are directly stored as multi-dimensional arrays; irregular grids are linked to attributes through independent coordinate data sets; image data is decoded into pixel arrays; and video data is stored as a multi-dimensional array of time dimension.

[0121] The multi-source data processing specifically includes:

[0122] Scientific data storage: Regular grids are directly stored as multidimensional arrays (float32 or float64) with a data shape of [depth dimension][latitude dimension][longitude dimension]. Attribute fields point to the coordinate axis dataset of the regular grid. Irregular grids store node coordinates as independent matrices (float32, shape Px3, where P is the total number of nodes). Attribute fields point to the node coordinate dataset.

[0123] Image data storage: Decode the original image into a pixel array and store it in uint8 format, and declare the image resolution, color type, and compression algorithm in the properties.

[0124] Video data storage: Store video data as a multidimensional array of the time dimension ([time step][number of channels][height][width]), and declare the frame rate, encoding format, and resolution in the properties.

[0125] Exemplarily, the metadata management includes: designing basic metadata of data sets for different types of data.

[0126] Specifically, the metadata management includes:

[0127] For different types of data, the dataset contains basic metadata: scientific data metadata, image data metadata, and video data metadata. See Table 1, Table 2, and Table 3;

[0128] Table 1 Scientific data metadata

[0129]

[0130] Table 2 Image data metadata

[0131]

[0132] Table 3 Video data metadata

[0133]

[0134] The final file structure is as follows:

[0135] # Geographic coordinate ontology storage

[0136] / Geospatial / Coordinates (Dataset: float32, shape=1000x3)

[0137] #Store the three-dimensional geographic coordinates (longitude, latitude, elevation) of 1000 nodes

[0138] # Attribute declaration coordinate system standard (WGS84)

[0139] Attributes: {"crs": "WGS84 (EPSG:4326)"}

[0140] # Additional storage of plane projection coordinates

[0141] / Geospatial / Projection (Dataset: float32, shape=1000x2)

[0142] #Store the plane projection coordinates of 1000 nodes

[0143] # Attribute annotation projection algorithm

[0144] Attributes: {"projection_method": "Web Mercator (EPSG:3857)"}

[0145] # Meteorological observation data (regular / irregular grid)

[0146] / Meteorology / Observation / Satellite / 2025-05-21

[0147] ├── Temperature_Regular (Dataset: float32, shape=10x180x360)

[0148] │#Store regular grid temperature data (10 depth layers x 180 latitude x 360 longitude)

[0149] │# Metadata declaration source, units, grid type, resolution

[0150] │Attributes: {"source": "GOES-16", "unit": "°C", "grid_type": "regular", "resolution": "0.5°"}

[0151] │# Coordinate links to independent coordinate axis datasets

[0152] │Coordinates: {"longitude": / Geospatial / Coordinates_Longitude, "latitude": / Geospatial / Coordinates_Lat+itude}

[0153] └── Temperature_Unstructured (Dataset: float32, shape=1000)

[0154] #Store irregular grid temperature data (1000 nodes)

[0155] # Metadata declaration source, unit, grid type

[0156] Attributes: {"source": "Buoy Network", "unit": "°C", "grid_type": "unstructured"}

[0157] # Coordinates are linked to a unified geographic coordinate dataset

[0158] Coordinates: {"coordinates": / Geospatial / Coordinates}

[0159] # Ocean simulation data

[0160] / Oceanography / Simulation / ROMS / 2025-05-21

[0161] ├── Salinity_Regular (Dataset: float32, shape=50x50x10)

[0162] │#Store regular grid salinity data (50x50 spatial grid x 10 time steps)

[0163] │# Metadata declaration model version, unit, mesh type

[0164] │Attributes: {"model_version": "ROMS v4.0", "unit": "psu", "grid_type": "regular"}

[0165] └── GridTopology_Unstructured (Dataset: int32, shape=50x50x10)

[0166] #Store irregular grid topology (adjacency)

[0167] # Metadata declaration grid type

[0168] Attributes: {"grid_type": "unstructured"}

[0169] # Image data

[0170] / Imaging / Satellite / 2025-05-21

[0171] ├── Image_Visible (Dataset: uint8, shape=1000x2000x3)

[0172] │# Store visible light image (1000x2000 pixels x 3 channels RGB)

[0173] │# Metadata declaration image format, resolution, color space, geographic coordinates link

[0174] │Attributes: {"image_format": "PNG", "resolution": "1000x2000", "color_space": "RGB", "geospatial_coords": / Geospatial / Coordinates}

[0175] └── Image_Infrared (Dataset: uint16, shape=1000x2000)

[0176] #Store infrared band image (single channel)

[0177] # Metadata declaration image format, bit depth, geographic coordinates link

[0178] Attributes: {"image_format": "TIFF", "bit_depth": 16, "geospatial_coords": / Geospatial / Coordinates}

[0179] # Video data

[0180] / Video / Drone / 2025-05-21

[0181] ├── Video_Drone_Flight (Dataset: uint8, shape=100x1080x1920x3)

[0182] │# Store aerial video (100 frames x 1080x1920 pixels x 3 channels RGB)

[0183] │# Metadata declaration video format, frame rate, resolution, geographic coordinates link

[0184] │Attributes: {"video_format": "H.264", "frame_rate": 30.0, "resolution": "1920x1080", "geospatial_coords": / Geospatial / Coordinates}

[0185] └── Video_Metadata (Group)

[0186] #Store video metadata groups (optional extension)

[0187] └── geospatial_coords (SoftLink to / Geospatial / Coordinates)

[0188] # Indexing data

[0189] / Index / TimeIndex (Dataset: datetime64, shape=1000)

[0190] #Store time series index (such as 1000 time points)

[0191] # Used to quickly locate time dimension data

[0192] / Index / GeoIndex (Dataset: float32, shape=2x2)

[0193] # Store geographic range index (e.g. longitude [-180,180] x latitude [-90,90])

[0194] # Used to quickly filter geographic area data

[0195] S3 builds distributed data association and indexing, including data standardization preprocessing and cross-source data association.

[0196] Design a standardized preprocessing process to uniformly convert raster data such as satellite imagery, and resample time series data such as buoys to a unified time base unit system. Define a standard unit conversion rule library, declared through the `unit` attribute field.

[0197] Exemplarily, data standardization preprocessing specifically includes:

[0198] Unified format conversion: Decode raster data (such as satellite imagery) into a three-dimensional array of `Height×Width×Channels`, and expand multi-band imagery by the channel dimension; convert the original timestamps of time series data such as buoys (such as local time, GPS time) to UTC time, and interpolate data with non-uniform time steps to generate equally spaced series.

[0199] Unit system normalization: define a standard unit conversion rule base (such as temperature is unified as °C, salinity is unified as PSU), declared through the attribute `unit` field.

[0200] Exemplarily, cross-source data association includes performing spatiotemporal alignment of cross-source data and performing spatiotemporal matching of multi-source data. Specifically, the spatiotemporal alignment of cross-source data involves appending the `time_reference` attribute (UTC timestamp) and the `spatial_extent` attribute (latitude and longitude range) to all datasets to perform spatiotemporal matching of multi-source data.

[0201] S4 optimizes and expands storage performance, including density-sensitive segmentation and layered compression.

[0202] Exemplarily, the density-sensitive chunking includes dynamically adjusting the HDF5 chunk size based on the grid node density and dynamically adjusting the chunk dimension based on the data access pattern, chunking time series data based on the time axis, and chunking spatial raster data based on latitude and longitude.

[0203] Exemplarily, the density-sensitive block specifically includes:

[0204] Density-sensitive chunking: Dynamically adjusts HDF5 chunk size based on grid node density, using 32×32 chunks for high-density areas and 128×128 chunks for sparse areas. Chunk dimensions are also dynamically adjusted based on data access patterns. Time series data is chunked by time (e.g., chunk_shape=(100,1,1)). Spatial raster data is chunked by latitude and longitude (e.g., chunk_shape=(1,512,512)).

[0205] Node density is defined as:

[0206] ;

[0207] Where, is the total number of nodes in the current region, is the area of ​​the region;

[0208] Chunk size is defined as:

[0209] ;

[0210] Exemplarily, the layered compression includes using a floating-point compression algorithm to maintain numerical precision for floating-point scientific data and using lossless compression for coordinate data; and applying JPEG2000 lossy compression to image / video data, with a configurable compression ratio.

[0211] In another exemplary embodiment, the layered compression specifically includes: using the ZFP floating-point compression algorithm for floating-point scientific data to maintain numerical accuracy; using lossless compression for coordinate data; and applying JPEG2000 lossy compression for image / video data with a configurable compression ratio.

[0212] The ZFP algorithm achieves data compression through the following steps:

[0213] Blocking. Divide the original data into multidimensional blocks of fixed size (4×4×4 for 3D blocks and 4×4 for 2D blocks).

[0214] Quantization. Quantize the data in each block, that is, map the data to a smaller range to reduce the loss of accuracy.

[0215] Entropy coding. Use run-length coding or Huffman coding to losslessly compress the quantized coefficients to generate the final compressed data stream.

[0216] Example 2: This embodiment of the present invention provides a multi-source marine environment data storage system based on HDF5, the system comprising:

[0217] A hierarchical storage framework construction module is used to build a three-level directory structure of domain-type-source, implement dynamic expansion mechanism of spatiotemporal dimensions, and implement spatiotemporal composite encoding;

[0218] HDF5 nested group storage architecture design module for grid topology definition, dynamic resolution adaptive storage, multi-source data processing, and metadata management;

[0219] Distributed data association and index construction module, used for data standardization preprocessing and cross-source data association;

[0220] The optimization and expansion module is used to optimize and expand storage performance, including density-sensitive blocking and layered compression.

[0221] To further illustrate the effects of the embodiments of the present invention, the following experiments were conducted.

[0222] Test Example 1: Polar Region Data Storage. Extract and store one month of sea surface temperature data for the Arctic region (60°N-90°N) from the global sea surface temperature data. See Table 4.

[0223] Table 4 Data storage in polar regions

[0224]

[0225] To address the storage redundancy caused by the convergence of longitude and latitude in high-latitude areas, the grid resolution is dynamically adjusted through hierarchical resolution annotation, significantly reducing the storage size.

[0226] Test Example 2: Cross-modal spatiotemporal query efficiency test. Search for "numerical simulation and corresponding satellite remote sensing temperature values ​​of the equatorial Pacific in the summer of 2023", see Table 5;

[0227] Table 5 Cross-modal spatiotemporal query efficiency test

[0228]

[0229] The solution of the present invention uses a four-level directory architecture to directly connect data according to the / Oceanography / Observation / Satellite path, and uses the spatiotemporal R-tree index to filter the target spatiotemporal range at one time, thereby significantly reducing query time and IO times.

[0230] The above description is only a preferred specific implementation method of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A multi-source marine environment data storage method based on HDF5, characterized in that: The method comprises the following steps: S1: Build a hierarchical logical catalog and coding system. Based on the domain-type-source principle, a three-level catalog structure is constructed. This system combines a dynamic expansion mechanism for spatiotemporal dimensions with a spatiotemporal composite coding method to form a unified data logical organization system. This provides a catalog index foundation and refined spatiotemporal expression for subsequent data storage and retrieval. S2: Design storage structures and multi-source adaptation mechanisms. Based on the constructed directory and spatiotemporal coding system, design an HDF5 nested group storage structure to support topological definition storage of regular and irregular grids, dynamic resolution adaptive storage, and multimodal data storage of images and videos. Build a metadata management mechanism to ensure data description consistency and cross-source compatibility. S3: Build a data indexing and association mechanism for multi-source collaboration. To meet the integration needs of multi-source heterogeneous data, design a standardized preprocessing process, unify the time base and physical unit system, and build a distributed spatiotemporal indexing mechanism to achieve accurate matching and efficient linkage query of cross-source data. S4: Develop performance optimization and storage compression strategies. After completing logical organization, physical storage, and data indexing, introduce density-sensitive partitioning and multi-level compression strategies. Dynamically adjust the partitioning method based on grid density and access patterns. Use ZFP and JPEG2000 compression algorithms to achieve high-performance, high-compression multi-source data storage optimization. In step S1, the spatiotemporal dimension dynamic expansion mechanism includes: Timeline organization: Create a time series directory by nesting / Year / Month / Day within the data source group; store the timestamp array in / Index / TimeIndex, and establish a mapping relationship between the time directory and the dataset; Spherical spatiotemporal hybrid indexing introduces a latitude-dependent resolution function to dynamically adjust the grid cell size and generates a unique grid code using the H3 geographic gridding algorithm. A spatiotemporal R-tree index structure is embedded in the HDF5 file, where the index node contains the time range, spatial grid code, and data block physical address. H3 hexagonal grid coding uses the H3 geographic grid to divide the sphere into multi-resolution hexagonal cells; Hierarchical spatiotemporal coding dynamically selects coding granularity and dimension based on the spatiotemporal characteristics of the data. For high-dimensional data of ocean and atmospheric profiles, vertical dimension identifiers are embedded in the coding to deeply correlate with the sea-air profile data. The latitude-dependent resolution function is: ； Where, For latitude resolution, is the equatorial resolution, is the polar resolution, Switch threshold for resolution; Generate a unique grid code using the H3 geo-grid algorithm, including: (1) Convert latitude and longitude to three-dimensional coordinates; ； ； ； Where, is the longitude; (2) Determine the resolution level based on the latitude-dependent resolution function; (3) Calculate the hexagon center and calculate the center position of the hexagon formed by multiple nodes; Assume the latitude and longitude coordinates of the hexagon vertices are: ； Then the geometric center for: ； ； Where, and Hexagonal The longitude and latitude of each vertex; Generate H3 code and generate 64-bit H3 index based on the center position of the hexagon; In step S2, dynamic resolution adaptive storage is performed, including: storing the basic coordinates of the grid nodes in the form of a three-dimensional floating-point array, declaring the coordinate system standard in the attributes; creating an auxiliary dataset under the same group, storing the plane projection coordinates, and annotating the projection method; recording the fixed resolution for regular grids, dividing the resolution identifier by region for irregular grids, and recording the difference between the equator and the poles in the attributes; The metadata management includes designing basic metadata of data sets for different types of data, and the basic metadata of data sets include scientific data metadata, image data metadata, and video data metadata; In step S4, the density-sensitive block partitioning includes: dynamically adjusting the HDF5 block size according to the grid node density, dynamically adjusting the block dimension according to the data access mode, and partitioning the time series data by time axis; and partitioning the spatial raster data by latitude and longitude. Achieve high-performance, high-compression ratio multi-source data storage optimization, including: using the ZFP floating-point compression algorithm for floating-point scientific data and lossless compression for coordinate data; and applying JPEG2000 lossy compression for image / video data.

2. The multi-source marine environment data storage method based on HDF5 according to claim 1 is characterized in that: In step S1, a three-level directory structure is constructed with field-type-source as the main line, including: creating multiple main groups under the root directory to represent the attributes of the subject field; establishing subgroups of different types of products under each field group; grouping by data source within the type subgroup to determine the type of data acquisition equipment.

3. The multi-source marine environment data storage method based on HDF5 according to claim 1 is characterized in that: In step S2, the topology definition includes: storing coordinate axes in a regular grid as independent datasets, linked to the main dataset through attributes; The irregular grid stores the three-dimensional coordinate matrix of each node and links it to the main dataset through attributes. The irregular grid creates the `Adjacency_List` dataset, which uses a sparse matrix format to record the adjacent cell index of each grid cell. The grid_type field is defined in the dataset properties and the grid type description is attached. An independent Group is created for each time step to store the grid topology and geometry data of that time step.

4. The multi-source marine environment data storage method based on HDF5 according to claim 1, characterized in that: In step S3, a standardized preprocessing process is designed to meet the needs of integrating multi-source heterogeneous data and unify the time base and physical unit system, including: Design a standardized preprocessing process to uniformly convert raster data of satellite images in different formats, resample the time series data of buoys to a unified time base unit system and normalize them; define a standard unit conversion rule library and declare it through the attribute `unit` field; Achieve accurate matching and efficient linkage query of cross-source data, including spatiotemporal alignment of cross-source data and spatiotemporal matching of multi-source data.

5. The multi-source marine environment data storage method based on HDF5 according to claim 1 is characterized in that: Node density is defined as: ； Where, is the total number of nodes in the current region, is the area of ​​the region; Chunk size is defined as: 。 6. A multi-source marine environment data storage system based on HDF5, characterized in that: The system implements the multi-source marine environment data storage method based on HDF5 as described in any one of claims 1 to 5, and the system includes: The hierarchical storage framework construction module is used to build a hierarchical logical directory and coding system. Based on the domain-type-source principle, a three-level directory architecture is constructed. This combines a dynamic expansion mechanism for spatiotemporal dimensions with a spatiotemporal composite coding method to form a unified data logical organization system, providing a directory index foundation and refined time-space expression for subsequent data storage and retrieval. The HDF5 nested group storage architecture design module is used to design storage structures and multi-source adaptation mechanisms. Based on the constructed directory and spatiotemporal coding system, the HDF5 nested group storage structure is designed to support topological definition storage of regular and irregular grids, dynamic resolution adaptive storage, and multimodal data storage of images and videos. The metadata management mechanism is built to ensure data description consistency and cross-source compatibility. A distributed data association and indexing module is used to build data indexing and association mechanisms for multi-source collaboration. To meet the integration needs of multi-source heterogeneous data, a standardized pre-processing process is designed, time bases and physical unit systems are unified, and a distributed spatiotemporal indexing mechanism is constructed to achieve accurate matching and efficient linkage query across data sources. The optimization and expansion module is used to implement performance optimization and storage compression strategies. After completing logical organization, physical storage, and data indexing, density-sensitive blocking and multi-level compression strategies are introduced. The blocking method is dynamically adjusted according to grid density and access patterns. The ZFP and JPEG2000 compression algorithms are used to achieve high-performance, high-compression ratio multi-source data storage optimization.

Images