A storage and management method for multi-resolution block stacked grids based on HDF5

Abstract

The present invention discloses a storage and management method for multi-resolution block stacked grids based on HDF5. The method divides the model into multiple three-dimensional models according to different resolution levels according to actual needs and model complexity, constructs a hybrid data storage structure of a hierarchical K-ary tree, integrates and manages models of different scales, and uniformly schedules them across scales. Then, after the three-dimensional model is converted into a sparse stacked grid model, the stacked grid model is stored and managed by utilizing the characteristics of HDF5 files that support unlimited data types and are designed for flexible and efficient I / O as well as large-capacity and complex data.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional geological model data management, and more specifically, to a storage and management method of multi-resolution block stacked grids based on HDF5. Background Art

[0002] Visualization helps to extract and identify necessary information from large amounts of spatiotemporal data and has become a hot topic in the field of geoscience. The access, processing and application of real-time data visualization of three-dimensional geological models in web browsers is the main development trend. Online visualization is limited by software and hardware resources. In the past few decades, in order to make the data volume of 3D geological models smaller and easier to analyze and process on the Internet, researchers have proposed various data structures to express 3D geological models. Among them, the stacked grid model is widely used in the field of online visualization due to its advantages in data volume and spatial query (Natali et al. 2014; Tegtmeier et al. 2014; Graciano et al. 2018).

[0003] The grid model needs to be able to capture the minute details of the solid geology, but the uniform and fine volume representation requires huge memory usage. Processing large data sets when performing volume rendering is not a simple task, especially when interactivity is required. The computational intensity required is very high. Although many studies have constructed stacked grid models for different purposes, Graciano et al. (Graciano et al. 2018) also used stacked grids for real-time visualization, which saves most of the memory space compared to voxel models. However, current research focuses on single-block, single-scale stratigraphic and lithological structure modeling (Prinds et al. 2020). The construction of high-precision stacked grid model spatial index to support rapid data update remains an unresolved problem. Summary of the invention

[0004] The present invention provides a storage and management method for multi-resolution block stacked grids based on HDF5, which solves the technical problem of constructing a high-precision stacked grid model spatial index to support rapid data update in the prior art.

[0005] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0006] The present invention provides a storage and management method for multi-resolution block stacked grids based on HDF5, comprising the following steps:

[0007] Generating a plurality of three-dimensional models from the preset three-dimensional uniform grid model according to the resolution level, wherein the detail expression degree of the three-dimensional model increases step by step with the resolution level;

[0008] Based on the multiple three-dimensional models, a multi-level index structure is constructed, wherein the upper layer of the multi-level index structure adopts a multi-level resolution index to integrate and manage the three-dimensional models of different resolution levels, and the lower layer of the multi-level index structure defines a plurality of K-ary trees, each of which records a three-dimensional model of a resolution level;

[0009] Based on the multi-level index structure, storing the multiple three-dimensional models;

[0010] Convert each of the three-dimensional models into a stacked grid model to obtain a plurality of stacked grid models;

[0011] The plurality of stacked mesh models are converted into HDF5 files.

[0012] In the above technical means, first of all, according to actual needs and model complexity, the model is divided into multiple three-dimensional models according to different resolution levels, and a hierarchical K-ary tree hybrid data storage structure is constructed to integrate and manage models of different scales for unified scheduling across scales. Then, after the three-dimensional model is converted into a sparse stacked grid model, the HDF5 file is used to support an unlimited number of data types and is designed for flexible and efficient I / O as well as large-capacity and complex data to store the stacked grid model.

[0013] Furthermore, each K-ary tree records a three-dimensional model of a resolution level, including:

[0014] The K-ary tree divides the 3D uniform grid model into grids, each grid has the same time complexity and space complexity, the grid is expressed as an instance object of the StackedGridBlock class, and each node of the K-ary tree records the model information of the corresponding grid in the three-dimensional model;

[0015] When the resolution level is n, the number of nodes in the K-ary tree is the number of grids into which the three-dimensional uniform grid model is divided. for:

[0016] .

[0017] Furthermore, the K-ary tree uses a linear list to store leaf nodes and a compact array to implement continuous array storage. The compact array member type is a BlockInformation structure. The index position dataIndex(i) of the grid with a given index i in the linear list is calculated, where i is the index value of the full tree:

[0018] dataIndex(i)= BlockInformation[i]

[0019] In the formula, BlockInformation[ ] represents the BlockInformation structure array;

[0020] When the return value is -2, the subspace is empty; otherwise, the return value is the index position of the grid with the given index i in the linear table.

[0021] Furthermore, based on the multi-level index structure, before storing the multiple three-dimensional models, the method further includes extracting data from physical storage into the multi-level index structure, including:

[0022] The original data of the three-dimensional model of different resolution levels are stored in different computer disk spaces, and getLevelbyFile is used to extract the detail information corresponding to the different resolution levels from the computer disk space, wherein the detail information includes the resolution level, the grid scale, the grid block spacing in three directions, and the grid block width in three directions;

[0023] Initialize a StackedGrid object according to the detail information to store three-dimensional models of different resolution levels, calculate and set the spatial parameters of the object, the spatial parameters include grid block position, grid block spacing in three directions, and grid block width in three directions; for each file in the computer disk space associated with the resolution level, create a StackedGridBlock object to store the grid model, read block data from the computer disk space, the block data includes PillarIndex and PillarData;

[0024] Updates the state of the grid based on the presence of data and sets pointers and flags accordingly, and finally appends the populated StackedGrid object to the LODStackedGrid array of the LODSStackedGrid object.

[0025] Further, each of the three-dimensional models is converted into a stacked grid model, comprising:

[0026] The three-dimensional model is evenly divided into a plurality of uniform grids according to the resolution level;

[0027] Perform binary tree search on all the uniform grids attribute by attribute to obtain the top and bottom of each stratum, and arrange them in order from small to large according to depth, and do not record the top and bottom of adjacent strata in contact repeatedly, so as to form a linear array;

[0028] Expressing the linear array as two one-dimensional arrays yields a stacked grid model:

[0029] Arrays , Indicates the stratigraphic unit index, and sequentially stores the stratigraphic objects that need to be stored, wherein the stratigraphic objects include stratigraphic attribute and depth data pairs;

[0030] Arrays , Represents the grid unit index, records the location and data offset of the first stratigraphic object in the unit to clarify the relationship between the stratigraphic unit and the object.

[0031] Further, converting the plurality of stacked grid models into HDF5 files comprises:

[0032] The StackGrid array, StackGrid, and StackGridBlock are used to represent the three-dimensional uniform grid model, the single-resolution three-dimensional model, and the stacked grid model respectively. The arrays D and I in the stacked grid model correspond to PillarGridData and PillarIndex. In HDF5, the H5File object is used to store the complete three-dimensional uniform grid model. The subordinate levels are organized by Group, in which the CommonFG class is used for Group and DataSet operations, DataSpace is used to apply for the space required for data storage, Attribute is used to store the description information of the three-dimensional model at each level of resolution, and CompType is used to specify the stored data type. The System class manages the model data HDF5 file by calling instance objects of the four classes Operate\Write\Reader\Query.

[0033] Furthermore, it also includes using the UpdateTileData function to update the data stored in the HDF5 file.

[0034] Furthermore, it also includes reading and writing large-scale data to HDF5 files, including the following steps:

[0035] Initialization phase:

[0036] Define the structure HDF5Tile, which contains three integer fields: PositionPointer, LODChildBloc and Editable, which represent the position pointer, sub-block pointer and editable flag respectively, and are used to describe the basic information and status of each grid;

[0037] File creation and property setting:

[0038] Create a new HDF5 file and write the attributes 'MaxLevel' and 'MinLevel' at the file level, which describe the highest and lowest level of detail of the data based on the resolution level of the grid;

[0039] Data organization and writing:

[0040] For each grid in the LODStackedGrid array, the program iterates and performs the following steps:

[0041] a. Create a group for each resolution level of the grid to organize the spatial data at that level, including the grid block spacing in three directions and the grid width in three directions;

[0042] b. Write group attributes, which are the spatial information of the stacked grid model. The spatial information describes the spatial data layout at each resolution level. Create data space H5::DataSpace and attribute variable H5::Attribute to store the spatial information of the stacked grid model and store it in the HDF5 file:

[0043] c. Define a single-resolution model information dataset to store the attribute information of all stacked grid models at this resolution level, initialize an HDF5Tile structure array to record the attribute data of the stacked grid models, and then write it to the group just created;

[0044] d. Create and write the TileInfomationArray dataset to store the compact array;

[0045] e. Create a subgroup within each resolution level group specifically for storing stacked grid model data. For each stacked grid model, define and create datasets of grid index PillarIndex and grid data PillarData and corresponding data spaces to store arrays I and D. Finally, write the data into an HDF5 file.

[0046] Furthermore, it also includes cleaning up all dynamically allocated memory resources.

[0047] Furthermore, it also includes adopting a capture and processing mechanism to handle possible exceptions that may occur in operations on files, groups, data sets, data spaces and data types.

[0048] Furthermore, it also includes closing and deleting HDF5 files and dynamically allocated resources.

[0049] Compared with the prior art, the technical solution of the present invention has the following beneficial effects:

[0050] The present invention generates several three-dimensional models according to the resolution level from the three-dimensional uniform grid model, displays the three-dimensional models of the corresponding resolution level according to actual needs, and converts the three-dimensional models into sparse stacked grid models to save storage space. Finally, the data management technology based on HDF5 is used to realize the data management of the stacked grid models. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic flow chart of a method for storing and managing multi-resolution block stacked grids based on HDF5 provided in an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a hierarchical K-ary tree index structure provided by an embodiment of the present invention;

[0053] Figure 3 A UML class diagram of a hierarchical K-ary tree index provided in an embodiment of the present invention;

[0054] Figure 4 A schematic diagram of a compact array provided by an embodiment of the present invention;

[0055] Figure 5 A schematic diagram of a stacked grid model and a data storage method provided by an embodiment of the present invention, wherein: Figure 5 (a) is a stacked grid model. Figure 5 (b) is the data storage method;

[0056] Figure 6 A multi-resolution block stacked grid UML class diagram based on HDF5 is provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The accompanying drawings are only used for illustrative purposes and are not to be construed as limiting the present application;

[0058] In order to better illustrate the present embodiment, some parts in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product;

[0059] It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0060] The technical solution of the present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0061] Example

[0062] This embodiment provides a storage and management method for multi-resolution block stacked grids based on HDF5, such as Figure 1 As shown, the following steps are included:

[0063] Generating a plurality of three-dimensional models from the preset three-dimensional uniform grid model according to the resolution level, wherein the detail expression degree of the three-dimensional model increases step by step with the resolution level;

[0064] Based on the multiple three-dimensional models, a multi-level index structure is constructed, wherein the upper layer of the multi-level index structure adopts a multi-level resolution index to integrate and manage the three-dimensional models of different resolution levels, and the lower layer of the multi-level index structure defines a plurality of K-ary trees, each of which records a three-dimensional model of a resolution level;

[0065] Based on the multi-level index structure, storing the multiple three-dimensional models;

[0066] Convert each of the three-dimensional models into a stacked grid model to obtain a plurality of stacked grid models;

[0067] The plurality of stacked mesh models are converted into HDF5 files.

[0068] In a further embodiment, each K-ary tree records a three-dimensional model of a resolution level, including:

[0069] The K-ary tree divides the 3D uniform grid model into grids, each grid has the same time complexity and space complexity, the grid is expressed as an instance object of the StackedGridBlock class, and each node of the K-ary tree records the model information of the corresponding grid in the three-dimensional model;

[0070] When the resolution level is n, the number of nodes in the K-ary tree is the number of grids into which the three-dimensional uniform grid model is divided. for:

[0071] .

[0072] In a specific embodiment, a K-ary tree records a single-scale 3D model, and each 3D model information is recorded in a leaf node associated with the K-ary tree. The hierarchical structure is as follows: Figure 2 As shown, the corresponding class diagram is shown in Figure 3 , showing the data structure composition of three main levels: LODStackedGrid, StackedGrid, and StackedGridBlock.

[0073] exist Figure 2 In the , the resolution level ranges from 0 to N, and N+1 3D models are generated according to the resolution level, called models 0-N, and the degree of model detail expression increases step by step. The multi-resolution model corresponds to the class LODStackedGrid, which records the maximum and minimum resolutions of the model, and the single-resolution model (StackedGrid) is stored in the member variable LodStackedGrid defined in it.

[0074] In a further embodiment, the K-ary tree uses a linear table to store leaf nodes. Since the subspace may be empty during the level-by-level division process, this embodiment uses a non-full tree to avoid the generation of blank space. The linear structure used at the beginning of creation needs to be combined with encoding to complete positioning and query. In order to save memory, a compact array is used to implement continuous array storage, such as Figure 4 As shown, the compact array member type is a BlockInformation structure, which calculates the index position dataIndex(i) of the grid with a given index i in the linear list, where i is the index value of the full tree:

[0075] dataIndex(i)= BlockInformation[i]

[0076] In the formula, BlockInformation[ ] represents the BlockInformation structure array;

[0077] When the return value is -2, the subspace is empty; otherwise, the return value is the index position of the grid with the given index i in the linear table.

[0078] In a further embodiment, based on the multi-level index structure, before storing the multiple three-dimensional models, the data is extracted from the physical storage into the multi-level index structure, allowing the user to specify different levels and block indexes to support multi-level and multi-granular data management. This is particularly important for processing large-scale geospatial data or three-dimensional modeling data, where different levels of detail or precision are usually organized in different levels. The specific extraction process includes:

[0079] The original data of 3D models of different resolution levels are stored in different computer disk spaces. GetLevelbyFile is used to extract the detail information corresponding to different resolution levels from the computer disk space. The detail information includes resolution level, grid scale, grid block spacing in three directions, and grid block width in three directions. First, memory allocation and StackedGrid data structure initialization are managed to store grid data. The hierarchical structure of the data is processed to allow different levels of detail to be loaded and accessed as needed. The LOD level is set according to the file content.

[0080] Initialize a StackedGrid object according to the detail information to store three-dimensional models of different resolution levels, calculate and set the spatial parameters of the object, the spatial parameters include grid block position, grid block spacing in three directions, and grid block width in three directions; for each file in the computer disk space associated with the resolution level, create a StackedGridBlock object to store the grid model, read block data from the computer disk space, the block data includes PillarIndex and PillarData;

[0081] Updates the state of the grid based on the presence of data and sets pointers and flags accordingly, and finally appends the populated StackedGrid object to the LODStackedGrid array of the LODSStackedGrid object.

[0082] The pseudo code of the extraction process is as follows:

[0083] InitLODLayeredPillarGrid(filePath){

[0084] levels[]←GetLevelbyFile(filePath) / / Get level information

[0085] Foreach level in levels{

[0086] stackedGrid←new StackedGrid() / / Apply for memory space to store single-resolution models

[0087] stackedGrid.SetStackedGrid(level) / / Set basic information of single-resolution model

[0088] files←ReadBinaryFilename(filePath+"\\"+level) / / Read the binary file name of the mesh model

[0089] For each file in files{

[0090] stackedGridBlock←new StackedGridBlock() / / Apply for memory space to store grid blocks

[0091] pillarIndex←ReadPillarIndex(file)

[0092] pillarData←ReadPillarData(file)

[0093] If pillarData is not null

[0094] stackedGridBlock.SetStackedGridBlock(pillarIndex,pillarData) / / If there is data in pillarData, store the data in the grid block, otherwise, setting the information of the grid block is invalid

[0095] Else

[0096] stackedGridBlock.SetInvalid()

[0097] }

[0098] }

[0099] }

[0100] So far, most three-dimensional grid models use uniform grids and explicitly store the attributes of each grid. However, for geological models, there are usually large uniform grid blocks, which means that using a sparse grid structure saves more space. Since geological models generally have layered characteristics, there are often a large number of grids with the same lithology or other attribute characteristics in the vertical direction. Therefore, this embodiment uses the idea of ​​a stacked grid model to construct a sparse grid.

[0101] The original mesh is a 3D model at a resolution of Uniform grid model generated by uniform division, number of grids Inversely proportional to the resolution, the global space is :

[0102]

[0103] in is the number of grids in dimension i, is the scale of the bounding box of the global space grid in dimension i , is the scale of a single grid cell in dimension i .

[0104] Data is represented and managed using a stacked grid, which consists of two static data structures.

[0105] Converting each of the three-dimensional models into a stacked mesh model, comprising:

[0106] The three-dimensional model is evenly divided into a plurality of uniform grids according to the resolution level;

[0107] On all the uniform grids, binary tree search is performed attribute by attribute to obtain the top and bottom of each stratum, and the top and bottom of each stratum are arranged in order from small to large according to the depth. The top and bottom of adjacent strata that touch each other are not recorded repeatedly to form a linear array. The intermediate data structure at this time is as follows: Figure 5 As shown, the intermediate data structure stores pointers on each cell, which wastes a lot of storage space on blank cells and is not suitable for physical storage. In order to achieve a compact grid representation in the global space and easy storage, the data structure is further improved and expressed as two one-dimensional arrays:

[0108] Arrays , Indicates the stratigraphic unit index, and sequentially stores the stratigraphic objects that need to be stored, wherein the stratigraphic objects include stratigraphic attribute and depth data pairs;

[0109] Arrays , Represents the grid unit index, records the location and data offset of the first stratigraphic object in the unit to clarify the relationship between the stratigraphic unit and the object.

[0110] Recording only the data offset is sufficient to clearly identify the location of the data object. However, the subsequent spatial analysis and dynamic update of the grid model will require repeated calculation of the sum of the offsets. Here, recording the object position can avoid the time complexity caused by repeated calculation.

[0111] In a further embodiment, converting the plurality of stacked grid models into HDF5 files comprises:

[0112] HDF5 files are containers for storing a variety of scientific data using two basic data objects, Group and Dataset. HDF5 technology consists of data models, libraries, and file formats for storing and managing data. It supports an unlimited variety of data types and is designed for flexible and efficient I / O as well as large and complex data. In the HDF5 data model, the container aspect of the HDF5 information set is represented by an HDF5 file with a specified root. HDF5 files contain array variables, groups, and types, which are called HDF5 datasets, HDF5 groups, and HDF5 datatype objects in the HDF5 data model, respectively. The HDF5 data model defines simple and extended linking mechanisms for creating associations between HDF5 information items. Finally, the HDF5 data model defines tools for annotating HDF5 information items using HDF5 Attributes. The above features of HDF5 provide a storage solution worth exploring for the hierarchical K-ary tree-stacked grid hybrid index management of the three-dimensional geological model in this study. Based on the characteristics of HDF5 file storage and the hybrid spatial index of this study, we designed a storage solution for three-dimensional geological structure models, corresponding to the class diagram ( Figure 6 ) as shown.

[0113] In this embodiment, the StackGrid array, StackGrid, and StackGridBlock are used to represent the three-dimensional uniform grid model, the single-resolution three-dimensional model, and the stacked grid model, respectively. The arrays D and I in the stacked grid model correspond to PillarGridData and PillarIndex. In HDF5, the H5File object is used to store the complete three-dimensional uniform grid model. The subordinate levels are organized by Group, in which the CommonFG class is used for Group and DataSet operations, DataSpace is used to apply for the space required for data storage, Attribute is used to store the description information of the three-dimensional model at each level of resolution, and CompType is used to specify the stored data type. The System class manages the model data HDF5 file by calling instance objects of the four classes Operate\Write\Reader\Query.

[0114] In a further embodiment, the method further includes using the UpdateTileData function to update the data stored in the HDF5 file.

[0115] In a specific embodiment, the UpdateTileData function is used to update the data of a specific tile in a layered tile grid model stored in the HDF5 file format. The UpdateTileData function determines the location of the tile and its internal grid that needs to be updated by receiving user input, and updates it according to the data provided by the user. The function allows dynamic modification and updating of specific parts of the data set by receiving coordinates and new data from user input. This flexibility is essential for real-time data analysis and editing tasks. The data processing logic within the function (such as layer counting and data index management) allows precise control of which data is read and modified, ensuring accurate data updates and reducing the risk of data redundancy and errors. Through the exception handling mechanism, the function can handle errors that may occur during the reading or writing of data, and ensure that files and memory resources are properly closed and released after the operation is completed, thereby increasing the robustness and stability of the program. The UpdateTileData function is not only a technical data update tool, it also reflects the comprehensive consideration of efficiency, flexibility and reliability when processing complex data structures, and is an important component of the three-dimensional data processing and analysis of this embodiment.

[0116] Furthermore, it also includes reading and writing large-scale data to HDF5 files, storing spatial data in the form of multi-resolution three-dimensional geological models in HDF5 file format, and supporting efficient reading and writing of large-scale spatial data, including the following steps:

[0117] Initialization phase:

[0118] First, the program tries to shield the default exception printing mechanism to prevent the exception information appearing in the file operation from interfering with the normal program output.

[0119] Define the structure HDF5Tile, which contains three integer fields: PositionPointer, LODChildBloc and Editable, which represent the position pointer, sub-block pointer and editable flag respectively, and are used to describe the basic information and status of each grid;

[0120] H5::CompType tileInformation(sizeof(HDF5TileInfo));

[0121] tileInformation.insertMember(MEMBER6, HOFFSET(HDF5TileInfo,PositionPointer), H5::PredType::NATIVE_INT);

[0122] tileInformation.insertMember(MEMBER7, HOFFSET(HDF5TileInfo,LODChildBlock), H5::PredType::NATIVE_INT);

[0123] tileInformation.insertMember(MEMBER8, HOFFSET(HDF5TileInfo,Editable), H5::PredType::NATIVE_INT);

[0124] File creation and property setting:

[0125] Create a new HDF5 file and write the attributes 'MaxLevel' and 'MinLevel' at the file level, which describe the highest and lowest level of detail of the data based on the resolution level of the grid;

[0126] hsize_t dimlod[1] = { 2};

[0127] H5::DataSpace file_attrspace(1, dimlod); / / Apply for file attribute space to store resolution information

[0128] H5::Attribute file_attr = file->createAttribute(FILE_ATTR_NAME.c_str(), H5::PredType::STD_I16BE, file_attrspace);

[0129] short attr_level[2];

[0130] attr_level[0] = stackedGrid.GetMinlevel();

[0131] attr_level[1] = stackedGrid .GetMaxlevel();

[0132] file_attr.write(H5::PredType::NATIVE_INT16, attr_level); / / Write the resolution information to the file

[0133] Data organization and writing:

[0134] For each grid in the LODStackedGrid array, the program iterates and performs the following steps:

[0135] a. Create a group for each resolution level of the grid to organize the spatial data at that level, including the grid block spacing in three directions and the grid width in three directions;

[0136] H5::Group gridBlock_group = file->createGroup((" / level_" +level);

[0137] b. Write group attributes, which are the spatial information of the stacked grid model. The spatial information describes the spatial data layout at each resolution level. Create data space H5::DataSpace and attribute variable H5::Attribute to store the spatial information of the stacked grid model and store it in the HDF5 file. Perform the following operations on the five file attributes in turn:

[0138] hsize_t dimsize[1] = { 3};

[0139] H5::DataSpace group_attrspace(1, dimsize); / / Apply for the data space required for the attribute

[0140] H5::Attribute gridBlock_group_attr; / / Declare HDF5 file attribute variables

[0141] gridBlock_group_attr=gridBlock_group.createAttribute("TileGridSize",H5::PredType::STD_I16BE, group_attrspace); / / Define HDF5 file attribute variables

[0142] gridBlock_group_attr.write(H5::PredType::NATIVE_INT16,BlockGridSize); / / Write attributes to HDF5 file

[0143] c. Define a single-resolution model information dataset to store the attribute information of all stacked grid models at this resolution level, initialize an HDF5Tile structure array to record the attribute data of the stacked grid models, and then write it to the group just created;

[0144] d. Create and write the TileInfoOffset dataset to store the compact array (BlockInformation structure);

[0145] e. Create a subgroup within each resolution level group specifically for storing stacked grid model data. For each stacked grid model, define and create datasets of grid index PillarIndex and grid data PillarData and corresponding data spaces to store arrays I and D. Finally, write the data into an HDF5 file.

[0146] H5::Group tile_group = gridBlock_group.createGroup("TileData");

[0147] / / Create datasets PillarIndex and PillarData

[0148] for (int j = 0; j <num_datasets; j += 2) {

[0149] pillarIndex_dataset[j] = tile_group.createDataSet(("Pillarindex_" +std::to_string(j / 2)).c_str(), H5::PredType::NATIVE_INT, dataspace);

[0150] pillarData_dataset[j + 1] = tile_group.createDataSet(("Pillardata_" +std::to_string(j / 2)).c_str(), H5::PredType::NATIVE_FLOAT, dataspace_);

[0151] }

[0152] In a further embodiment, after the program completes data writing, it is responsible for cleaning up all dynamically allocated memory resources, including grid block information arrays and data set resources. For exceptions that may occur in file, group, data set, data space and data type operations, the program adopts a capture and processing mechanism to ensure the correct release of resources and the stable execution of the program.

[0153] In a further embodiment, the method further includes closing and deleting the HDF5 file and dynamically allocated resources.

[0154] In a specific embodiment, the memory usage of the HDF5 file obtained by using this embodiment is shown in Table 1. The data source is a binary file. This embodiment compares the storage capacity difference between the HDF5 file and the source file. The results show that the memory compression ratio of the HDF5 file is about 31%, which proves the advantage of the HDF5 file management method in storing large-scale hierarchical and block data.

[0155] Table 1

[0156]

[0157] The generated HDF5 file is queried. According to the data hierarchical storage strategy, global data, data sub-blocks, and grid unit data are randomly obtained. Multiple queries are performed to obtain the average value. The results show that the three query methods are all at the millisecond level and have high query efficiency, as shown in Table 2.

[0158] Table 2

[0159]

[0160] The same or similar reference numerals correspond to the same or similar components;

[0161] The terms used to describe the positional relationship in the drawings are only used for illustrative purposes and should not be construed as limiting the present application;

[0162] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the embodiments here. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for storing and managing multi-resolution block stacked grids based on HDF5, characterized in that: The following steps are involved: Generating a plurality of three-dimensional models from the preset three-dimensional uniform grid model according to the resolution level, wherein the detail expression degree of the three-dimensional model increases step by step with the resolution level; Based on the plurality of three-dimensional models, a multi-level index structure is constructed, wherein the upper layer of the multi-level index structure adopts a multi-level resolution index to integrate and manage three-dimensional models of different resolution levels, and the lower layer of the multi-level index structure defines a plurality of K-ary trees, each of which records a three-dimensional model of a resolution level; Based on the multi-level index structure, storing the multiple three-dimensional models; Convert each of the three-dimensional models into a stacked grid model to obtain a plurality of stacked grid models; Converting the plurality of stacked grid models into HDF5 files; Converting each of the three-dimensional models into a stacked mesh model, comprising: The three-dimensional model is evenly divided into a plurality of uniform grids according to the resolution level; Perform binary tree search on all the uniform grids attribute by attribute to obtain the top and bottom of each stratum, and arrange them in order from small to large according to depth, and do not record the top and bottom of adjacent strata in contact repeatedly, so as to form a linear array; Expressing the linear array as two one-dimensional arrays yields a stacked grid model: Arrays , Indicates the stratigraphic unit index, and sequentially stores the stratigraphic objects that need to be stored, wherein the stratigraphic objects include stratigraphic attribute and depth data pairs; Arrays , Represents the grid unit index, records the location and data offset of the first stratigraphic object in the unit to clarify the relationship between the stratigraphic unit and the object.

2. The storage and management method of multi-resolution block stacked grids based on HDF5 according to claim 1, characterized in that: Each K-ary tree records a three-dimensional model of a resolution level, including: The K-ary tree divides the 3D uniform grid model into grids, each grid has the same time complexity and space complexity, the grid is expressed as an instance object of the StackedGridBlock class, and each node of the K-ary tree records the model information of the corresponding grid in the three-dimensional model; When the resolution level is n, the number of nodes in the K-ary tree is the number of grids into which the three-dimensional uniform grid model is divided. for: The K-ary tree uses a linear list to store leaf nodes and a compact array to implement continuous array storage. The compact array member type is a BlockInformation structure. The index position dataIndex(i) of the grid with a given index i in the linear list is calculated, where i is the index value of the full tree: dataIndex(i)= BlockInformation[i] In the formula, BlockInformation[ ] represents the BlockInformation structure array; When the return value is -2, the subspace is empty; otherwise, the return value is the index position of the grid with the given index i in the linear table.

3. The storage and management method of multi-resolution block stacked grids based on HDF5 according to claim 2, characterized in that: Based on the multi-level index structure, before storing the multiple three-dimensional models, the method further includes extracting data from physical storage into the multi-level index structure, including: The original data of the three-dimensional model of different resolution levels are stored in different computer disk spaces, and getLevelbyFile is used to extract the detail information corresponding to the different resolution levels from the computer disk space, wherein the detail information includes the resolution level, the grid scale, the grid block spacing in three directions, and the grid block width in three directions; Initialize a StackedGrid object according to the detail information to store three-dimensional models of different resolution levels, calculate and set the spatial parameters of the object, the spatial parameters include grid block position, grid block spacing in three directions, and grid block width in three directions; for each file in the computer disk space associated with the resolution level, create a StackedGridBlock object to store the grid model, read block data from the computer disk space, the block data includes PillarIndex and PillarData; Updates the state of the grid based on the presence of data and sets pointers and flags accordingly, and finally appends the populated StackedGrid object to the LODStackedGrid array of the LODSStackedGrid object.

4. The storage and management method of multi-resolution block stacked grids based on HDF5 according to claim 3 is characterized in that: Convert the multiple stacked grid models into HDF5 files, including: The StackGrid array, StackGrid, and StackGridBlock are used to represent the three-dimensional uniform grid model, the single-resolution three-dimensional model, and the stacked grid model respectively. The arrays D and I in the stacked grid model correspond to PillarGridData and PillarIndex. In HDF5, the H5File object is used to store the complete three-dimensional uniform grid model. The subordinate levels are organized by Group, in which the CommonFG class is used for Group and DataSet operations, DataSpace is used to apply for the space required for data storage, Attribute is used to store the description information of the three-dimensional model at each level of resolution, and CompType is used to specify the stored data type. The System class manages the model data HDF5 file by calling instance objects of the four classes Operate\Write\Reader\Query.

5. The storage and management method of multi-resolution block stacked grids based on HDF5 according to claim 4, characterized in that: It also covers updating data stored in HDF5 files using the UpdateTileData function.

6. The storage and management method of multi-resolution block stacked grids based on HDF5 according to claim 5, characterized in that: It also includes reading and writing large-scale data to HDF5 files, including the following steps: Initialization phase: Define the structure HDF5Tile, which contains three integer fields: PositionPointer, LODChildBloc and Editable, which represent the position pointer, sub-block pointer and editable flag respectively, and are used to describe the basic information and status of each grid; File creation and property setting: Create a new HDF5 file and write the attributes 'MaxLevel' and 'MinLevel' at the file level, which describe the highest and lowest level of detail of the data based on the resolution level of the grid; Data organization and writing: For each grid in the LODStackedGrid array, the program iterates and performs the following steps: a. Create a group for each resolution level of the grid to organize the spatial data at that level, including the grid block spacing in three directions and the grid block width in three directions; b. Write group attributes, which are the spatial information of the stacked grid model. The spatial information describes the spatial data layout at each resolution level, and sequentially creates a data space H5::DataSpace and an attribute variable H5::Attribute to store the spatial information of the stacked grid model and store it in the HDF5 file; c. Define a single-resolution model information dataset to store the attribute information of all stacked grid models at this resolution level, initialize an HDF5Tile structure array to record the attribute data of the stacked grid models, and then write it to the group just created; d. Create and write the TileInfomationArray dataset to store the compact array; e. Create a subgroup within each resolution level group specifically for storing stacked grid model data. For each stacked grid model, define and create datasets of grid index PillarIndex and grid data PillarData and corresponding data spaces to store arrays I and D. Finally, write the data into an HDF5 file.

7. The storage and management method of multi-resolution block stacked grids based on HDF5 according to claim 6, characterized in that: This also includes cleaning up any dynamically allocated memory resources.

8. The method for storing and managing multi-resolution block stacked grids based on HDF5 according to claim 7, characterized in that: It also includes the use of capture and processing mechanisms to handle exceptions that may occur in operations on files, groups, data sets, data spaces, and data types.

9. The method for storing and managing multi-resolution block stacked grids based on HDF5 according to claim 7, characterized in that: It also includes closing and deleting HDF5 files and dynamically allocated resources.

Images