A spatial-temporal data storage method based on geospatial grid

Abstract

The present invention discloses a data space-time storage method based on the earth space grid, the method obtains all earth space-time big data to be organized and managed, obtains the Beidou grid position code of the local area according to the data space information, defines the clustered spatial data storage architecture, establishes the mapping relationship between the storage space and the geographic space, and defines the storage architecture protocol for the application layer, management layer, cluster layer and data layer of the data storage, uses the Beidou grid position code and grid index code generation method to provide a unified location identification index for the earth space-time big data, provides location identification codes for various types of geographic space information, and a unified grid index for various types of geographic space information, and realizes information association retrieval and data resource statistics based on the Beidou grid. The present invention realizes the storage of spatial data according to the actual geographical location, and instantly aggregates the full amount of multi-source data in the local area, and provides an efficient data calculation interface.

Description

Technical Field

[0001] The present invention belongs to the field of distributed storage and calculation of earth spatiotemporal big data, and specifically relates to a data spatiotemporal storage method based on earth space grid. Background Art

[0002] With the development of cutting-edge technologies such as earth satellite observation, in-situ sensors, the Internet of Things, and social perception, people have obtained more and more geospatial data on the earth's surface, atmosphere, ocean, solid earth, etc. Among them, the storage of geospatial data is increasingly attracting widespread attention from academia and industry. At present, the mainstream storage mode of geospatial data is time-first spatiotemporal storage. Taking high-resolution remote sensing data as an example, with the rapid development of earth observation technology and the widespread deployment of various sensors around the world, the volume of high-resolution remote sensing data has exploded. In order to disperse the huge storage pressure, the mainstream storage strategy adopted by major remote sensing data centers is to judge the newness of the data according to the storage time and store it on different storage devices. Therefore, there are three different storage methods: online, near-line, and offline. The realization of the Beidou space-time storage system can provide strong support for subsequent data query, processing, analysis, and calculation, which has important research significance and application prospects.

[0003] The storage, migration and calculation of spatiotemporal big data mainly exist in the following difficulties: data of spatial guiding principles, resource allocation, and support for high-performance computing.

[0004] First, the existing technical solutions virtualize distributed storage resources into a whole, and the spatial data of the same area is dispersed and stored in the entire storage space, and the heterogeneity of spatial data distribution cannot be intuitively reflected. In other words, the existing technical solutions use mature technologies in the computer field to temporarily solve the problem of massive storage of spatial data, but lack the basic storage structure that uses spatial principles to guide the allocation and scheduling of storage resources. It is difficult for data managers to intuitively grasp the distribution of spatial data in different regions in storage resources. At the same time, due to the lack of storage structure guided by spatial principles, it is also difficult for the existing technical solutions to perform dynamic and adjustable storage resource configuration according to the actual storage needs of different regions.

[0005] Secondly, for edge computing and fog computing scenarios represented by the Internet of Things, their edge nodes often have obvious regional characteristics, and the storage architecture should reflect the spatial distribution characteristics of "cloud + fog + terminal". However, most of the existing technical implementation solutions are based on centralized cloud storage and cloud computing, and rarely consider distributed storage scenarios such as edge computing and fog computing. It is also difficult to achieve distributed deployment of storage systems guided by spatial principles. Existing ultra-large-scale data storage and computing methods do not take into account geographic spatial characteristics. Computing resources and data work independently and dispersedly. Data scheduling and parallel IO often make it difficult to avoid unnecessary data reading and preprocessing.

[0006] Therefore, it is an important issue that needs to be solved urgently to develop a new storage system with ultra-large-scale geographic spatial data storage capabilities and spatial data storage according to actual geographic locations. This system can instantly aggregate the full amount of multi-source data in a local area and provide an efficient computing interface. Summary of the invention

[0007] In view of the above problems, the present invention provides a method for spatiotemporal data storage based on the earth space grid, which realizes the effective management of spatiotemporal big data, provides distributed storage based on the Beidou space grid as the guiding principle, and lays the foundation for rapid regional data aggregation, migration and grid high-performance computing.

[0008] To achieve the above object, the present invention provides a data space-time storage method based on geospatial grid, comprising the following steps:

[0009] S1. Obtain all the earth's spatiotemporal big data to be organized and managed, obtain the Beidou grid position code of the area according to the data space information, and follow the definition and division method of the height domain specified in 20141448-T-466. Ten basic grids of 4°, 30′, 15′, 1′, 4″, 2″, 1 / 4″, 1 / 32″, 1 / 256″, and 1 / 2048″ on the earth's surface are selected as the θ in the Beidou grid position code to define the earth's height;

[0010] S2. Based on Beidou grid location coding, define a clustered spatial data storage architecture and establish a mapping relationship between storage space and geographic space;

[0011] S3. Define storage architecture protocols for the application layer, management layer, cluster layer, and data layer of data storage for all geospatial data and their corresponding Beidou grid location codes;

[0012] S4. Utilize Beidou grid location coding and grid index code generation methods to provide a unified location identification index for the Earth's spatiotemporal big data, provide location identification coding for various types of geographic spatial information, and a unified grid index for various types of geographic spatial information, to achieve information association retrieval and data resource statistics based on the Beidou grid.

[0013] Furthermore, the Beidou grid height dimension direction coding structure and code values ​​are as follows:

[0014] In the first part, 0 and 1 are used to mark the ground and underground, among which: below the surface is marked with 1 and above the surface is marked with 0; in the second part, the division consistent with the length of 4º of the equator is adopted, and the coding marks are 0-63, and each division is about 445.28km; in the third part, the division consistent with the length of 30' of the equator is adopted, and the coding marks are 0-7, and each division is about 55.66km; in the fourth part, the division consistent with the length of 15' of the equator is adopted, and the coding marks are 0-1, and each division is about 27.83km; in the fifth part, the division consistent with the length of 1' of the equator is adopted, and the coding marks are 0-9, AE, and each division is about 1.85km; in the sixth part, the division consistent with the length of 4" of the equator is adopted, and the coding marks are Use 0-9, AE, each division is about 123.69m; the seventh part, adopts the division consistent with the length of 2" of the equator, the coding mark uses 0-1, each division is about 61.84m; the eighth part, adopts the division consistent with the length of 1 / 4" of the equator, the coding mark uses 0-7, each division is about 7.73m; the ninth part, adopts the division consistent with the length of 1 / 32" of the equator, the coding mark uses 0-7, each division is about 0.97m; the tenth part, adopts the division consistent with the length of 1 / 256" of the equator, the coding mark uses 0-7, each division is about 12.1cm; the eleventh part, adopts the division consistent with the length of 1 / 2048" of the equator, the coding mark uses 0-7, each division is about 1.5cm.

[0015] Furthermore, a mapping relationship between geographic space and storage space is established, where the geographic space is expressed by the Beidou grid framework and the storage space is expressed by the storage nodes that can be actually allocated. Storage nodes correspond to grid units one by one, and the number of storage units at each level is consistent with the number of Beidou grids. In other words, storage clusters allocate storage resources according to the spatial principle. Each grid corresponds to a storage device, and the geospatial data is allocated to the corresponding grid according to the actual geographical location, and then stored in the corresponding storage device, realizing unified logical division and physical storage on a global scale.

[0016] Furthermore, for all geospatial data and their corresponding Beidou grid location codes, storage architecture protocols are defined for the application layer, management layer, cluster layer and data layer of data storage:

[0017] The data layer protocol includes a data segmentation protocol: used to organize the earth's spatiotemporal data into multi-level grids; a data encoding protocol: used to encode the grids according to the hierarchical segmentation protocol to reflect the spatial division corresponding to the grids; and a data expression protocol: based on the segmentation and encoding protocols, a spatial data gridding expression method is defined.

[0018] The cluster layer protocol includes the data storage protocol: the cluster topology structure and expansion method are designed according to the Beidou grid division principle; the data addressing protocol: the IP address of the cluster node in the cluster corresponds to the Beidou grid code, and the data is addressed and stored according to the spatial position relationship.

[0019] The management layer protocols include data organization protocol: each grid storage cluster device uses a distributed database and Beidou grid index table to manage and organize spatiotemporal big data; data fault-tolerant protocol: the online backup mechanism of spatial data within the spatiotemporal storage system ensures the correctness of data retrieval and access when one, some or some nodes fail; data retrieval protocol: Beidou grid coding provides rapid indexing of massive data.

[0020] The application layer protocols include data aggregation protocol: based on the retrieval protocol, single grid aggregation, user-defined space-time range aggregation and other application methods are developed; data migration protocol: based on the aggregation protocol, the plug-in backup storage device is studied to replace the traditional data aggregation, so as to realize the function of real-time migration of data in a certain area; data computing protocol: based on the aggregation protocol, the actual data storage capacity in each grid storage is considered, and the corresponding computing power is equipped to further develop grid accelerators to support ultra-large-scale grid parallel computing.

[0021] The present invention has the following beneficial effects:

[0022] The present invention provides an effective and unified storage method for multi-source spatiotemporal big data, which utilizes the spatial location information of the spatiotemporal big data and Beidou grid coding for association, identification, organization and storage; the present invention stores the spatiotemporal big data in blocks through Beidou grid coding, so that the spatial range corresponding to the spatiotemporal data of each grid is gridded and grid-coded.

[0023] The present invention realizes physical aggregation and storage of local data under the Beidou grid data organization system, and can be quickly aggregated into various types of data products according to the application needs of various industries.

[0024] According to the grid level definition, the present invention provides different numbers of memories corresponding to different numbers of grids at different levels, and provides scalability within the memory based on factors such as data volume and access hotspots. It also supports methods such as ultra-large-scale earth science data distributed computing and grid-related computing, and provides fast data management interfaces and business support for ultra-large-scale earth science computing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flow chart of the data space-time storage method based on the geospatial grid of the present invention;

[0026] Figure 2 Beidou space-time storage logical model and physical architecture;

[0027] Figure 3 Overall design of Beidou space-time storage system. DETAILED DESCRIPTION

[0028] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0029] The present invention provides a method and system for storing data in space and time based on the geospatial grid. In order to make the purpose, technical solution and advantages of the present invention more clearly described, the following is a detailed description of the implementation method of the present invention in conjunction with the accompanying drawings and examples. Figure 1 As shown, the following steps are included:

[0030] S1. Obtain all the earth's spatiotemporal data to be organized and managed, and obtain the Beidou grid code of the area based on the data space information; the spatiotemporal data includes typhoon intensity data and ocean data (ocean current, seawater salinity, seawater temperature, sea surface height); the database stored in the cluster device of the corresponding grid area is divided according to the grid position data, some of which are stored in the form of data points, and some are stored in the form of data ranges.

[0031] S2. Based on the Beidou grid model, define a clustered spatial data storage architecture and establish a mapping relationship between storage space and geographic space.

[0032] Based on the Beidou grid model, a clustered spatial data storage architecture is defined. A mapping relationship between storage space and geographic space is established, where the geographic space is expressed by the Beidou grid framework, the storage space is composed of N storage units, and the space-time storage cluster topology structure. Each storage unit can be expanded according to the quadtree splitting method of the Beidou grid based on the growth scale of data storage or access popularity. The number of storage units at each level is consistent with the number of Beidou grids. For example, when the earth is divided once, 4 grid units will be generated, then 4 storage nodes will be allocated to the corresponding grid units and store the earth data within the grid range. Compared with traditional distributed systems (such as distributed file systems, non-relational databases, and parallel databases), the biggest advantage of space-time storage clusters is that each storage node has a clear geographical meaning. In other words, space-time storage clusters allocate storage resources according to spatial principles. Each grid corresponds to a storage device. The earth's spatial data is allocated to the corresponding grid according to the actual geographical location, and then stored in the corresponding storage device, realizing unified logical division and physical storage on a global scale. Each storage node in the storage cluster has a unique device identification code (i.e., Beidou grid code), which can be used to quickly address and locate the storage node, thus achieving efficient storage and access of spatial data, as well as data aggregation and protection in multi-scale regions. In addition, spatial data will also carry corresponding grid code identification after gridding, which is called data coding. Therefore, in the storage cluster, any data file to be stored, as long as it carries a data code, will automatically match the device code and be stored in the corresponding storage device, thus physically realizing a space-time integrated storage and management system in which data in the same region is aggregated in the same storage device.

[0033] Each storage unit corresponds to a small grid data cluster, which is materialized as a grid storage and deployed in a distributed manner according to the space-time storage logic model, such as Figure 2 As shown in the figure. During deployment, different storage resources can be configured according to the volume of regional historical data storage and the growth rate of current data scale, or the importance of the region. From a global perspective, data in different grid spaces are physically isolated when stored. From a local perspective, multi-source spatial data in a grid unit are physically aggregated in the same grid storage.

[0034] S3. For all geospatial data and their corresponding Beidou grid location codes, define storage architecture protocols for the application layer, management layer, cluster layer, and data layer of data storage, such as Figure 3 shown.

[0035] Data layer protocol: Based on the Beidou grid space partition theory, it provides the underlying mechanism for encoding and storing earth space data; the partition protocol defines the method and process of hierarchical partition of earth space. The specific implementation is based on the Beidou grid earth partition framework: a geospatial reference grid framework that extends from 50,000 kilometers of space outside the earth to the center of the earth, and partitions the entire earth space into trillions of global spaces and centimeter-level precision, consisting of 32 levels of partitioned voxels; the specific implementation of the encoding protocol is based on the Beidou grid partition coding system, and through a partition code, it can quickly determine the spatial range of the corresponding grid, the partition level it is in, and the code of the adjacent grid; the expression protocol is responsible for implementing the expression method of earth space data (vector, raster) based on the partition code, and currently implements specific expression methods for point, line, and surface objects. For example, the grid expression method of a point object is to express a point entity as a grid at a specific level in the partition space. The meshing expression method of line objects is to regard the linear entity as a one-dimensional entity obtained by connecting a group of point entities in a certain order in the meshing space, and express it as a single-scale mesh set at a specific level. The meshing expression method of surface objects is to regard the surface entity as a closed and connected two-dimensional entity obtained by connecting a group of point entities in a certain order in the meshing space, and express it as a single-scale boundary and internal mesh set at a specific level.

[0036] Cluster layer protocol: Through distributed storage and computer cluster technology, the storage, processing and computing services of spatial data are realized on the basis of the earth's grid division; the storage protocol guides the topological structure design and expansion method of the space-time storage cluster, and defines the clustered spatial data storage method. This sub-protocol is responsible for establishing the mapping relationship between the storage space and the geographic space, where the geographic space is expressed by the Beidou grid framework, and the storage space is composed of N storage units, which correspond one to one with the grid units. Each storage unit can be expanded according to the four-branch split method of the Beidou grid based on the growth scale of data storage or the access popularity; the addressing protocol is responsible for quickly locating and accessing the space-time storage device in the space-time cluster, and its addressing basis is that each storage device has a unique device identification code, which is the grid code. At the specific implementation level, a mapping relationship between the device identification code and the device physical address will be established, that is, the mapping between the logical address and the physical address, and an addressing server will be set up to maintain the mapping between the storage device identification code and its physical address (IP) to support the efficient storage and access of spatial data by the space-time storage system; in addition, the sub-protocol also specifies the specific storage rules for spatial data: under the above-mentioned distributed space-time storage architecture and addressing mechanism, the spatial data is placed in which or which storage devices, and its storage rules are determined according to the Beidou grid units and their number covered by the geographic spatial data. First, the spatial data is physically divided according to the grid to which it belongs, and then the divided data files are deployed to the corresponding storage devices according to the addressing strategy. Therefore, in the space-time storage system, the essence of storage is the complete match between data encoding and device encoding.

[0037] Management layer protocol: Based on the cluster layer, it realizes the effective management and access of spatial data; the organization protocol acts on the spatial data organization and management method within a single storage device. The data organization scheme currently used in a single storage device is the distributed non-relational database Hbase plus the Beidou grid index table. The primary key of the data table is the grid code, and the time of the data is stored in the table as an attribute column; the fault-tolerant protocol is responsible for the space-time storage system to provide reliable services, and stipulates that the space-time storage system can be configured with a master-slave storage cluster, the master cluster provides all functions, and the slave cluster only provides reads. The master cluster will synchronize all operations to the slave cluster. This not only provides load balancing, but also does not interrupt service or lose data in the event of an abnormality in the main cluster. Retrieval protocol: Beidou grid coding provides rapid indexing of massive data.

[0038] Application layer protocol: It provides the application interface of the space-time storage system. It is at the top level of the system and realizes the interactive operation of earth space data. Specifically, it includes: 1) Aggregation protocol: Based on the retrieval protocol, it develops application methods such as single grid aggregation and user-defined space-time range aggregation; 2) Migration protocol: Based on the aggregation protocol, it studies the plug-in backup storage device to replace the traditional data aggregation to achieve immediate emergency protection; 3) Computing protocol: Based on the aggregation protocol, it considers the actual data storage capacity in each grid storage device, equips with corresponding computing power, and further develops grid accelerators to support ultra-large-scale grid parallel computing.

[0039] S4. Under the above data protocol, the Beidou grid location coding and grid index code generation method are used to provide a unified location identification index for multi-source spatiotemporal big data, provide location identification coding for various types of geographic spatial information, a unified grid index for various types of geographic spatial information, grid-based information association retrieval and data resource statistics.

[0040] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for storing data in space and time based on geospatial grid, comprising the following steps: 1) Obtain all the earth's spatiotemporal big data to be organized and managed, and obtain the Beidou grid position code of the area according to the data space information. The principle of Beidou grid height domain definition and division follows the height domain definition and division method specified in 20141448-T-466, and select ten basic grids of 4°, 30′, 15′, 1′, 4″, 2″, 1 / 4″, 1 / 32″, 1 / 256″, and 1 / 2048″ on the earth's surface as the θ in the Beidou grid position code to define the geodetic height; 2) Based on Beidou grid location coding, define a clustered spatial data storage architecture and establish a mapping relationship between storage space and geographic space; 3) Define storage architecture protocols for the application layer, management layer, cluster layer, and data layer of data storage for all geospatial data and their corresponding Beidou grid location codes; 4) Relying on the Beidou grid location coding and grid index code generation method, it provides a unified location identification index for the Earth's spatiotemporal big data, provides location identification coding for various types of geographic spatial information, and a unified grid index for various types of geographic spatial information, realizing information association retrieval and data resource statistics based on the Beidou grid; In step 2), the geographic space is expressed by the Beidou grid framework. The storage space consists of N storage units. Each storage unit corresponds to a small Beidou grid data cluster, which is materialized as a grid storage and distributedly deployed according to the space-time storage logic model. During deployment, different storage resources are configured according to the regional historical data storage volume and the current data scale growth rate, or the importance of the region; in, The management layer protocol includes the organization protocol: each grid storage cluster device uses a distributed database and Beidou grid index table to manage and organize spatiotemporal big data; Fault-tolerant protocol: The online backup mechanism of spatial data within the space-time storage system ensures the correctness of data retrieval and access when one, some or a certain node fails; Retrieval protocol: Beidou grid coding provides rapid indexing of massive data; The data layer protocol includes a segmentation protocol: used to organize the Earth's spatiotemporal data into multi-level grids; coding Protocol: used to encode the grid according to the hierarchical subdivision protocol, reflecting the spatial division corresponding to the grid; Expression protocol: Define the gridding expression method of spatial data according to the segmentation and encoding protocol; Cluster layer protocols include storage protocols: cluster topology and expansion methods are designed according to Beidou grid division principles; Addressing protocol: The IP addresses of cluster nodes in the cluster correspond to the Beidou grid codes, and data is addressed and stored according to the spatial location relationship; The application layer protocols include aggregation protocols: single grid aggregation is developed based on the retrieval protocol, and user-defined spatial and temporal range aggregation application methods are used; Migration protocol: Based on the aggregation protocol, research on plugging and unplugging backup storage devices to replace traditional data aggregation to achieve the function of instant migration of data in a certain area; Computing protocol: Based on the aggregation protocol, consider the actual data storage capacity in each grid storage, equip it with corresponding computing power, and further develop grid accelerators to support ultra-large-scale grid parallel computing.

2. The method for storing data in space and time based on geospatial grid according to claim 1, characterized in that: In step 1), the Beidou grid height dimension direction coding structure and code values ​​are as follows: In the first part, 0 and 1 are used to mark the ground and underground, among which: below the surface is marked with 1 and above the surface is marked with 0; in the second part, the division consistent with the length of 4º of the equator is adopted, and the coding marks are 0-63, each division is 445.28km; in the third part, the division consistent with the length of 30′ of the equator is adopted, and the coding marks are 0-7, each division is 55.66km; in the fourth part, the division consistent with the length of 15′ of the equator is adopted, and the coding marks are 0-1, each division is 27.83km; in the fifth part, the division consistent with the length of 1′ of the equator is adopted, and the coding marks are 0-9, AE, each division is 1.85km; in the sixth part, the division consistent with the length of 4″ of the equator is adopted, and the coding marks are The lengths of 0-9 and AE are adopted, and each division is 123.69m. The seventh part adopts the division consistent with the length of 2″ of the equator, and the coding marks are 0-1, and each division is 61.84m. The eighth part adopts the division consistent with the length of 1 / 4″ of the equator, and the coding marks are 0-7, and each division is 7.73m. The ninth part adopts the division consistent with the length of 1 / 32″ of the equator, and the coding marks are 0-7, and each division is 0.97m. The tenth part adopts the division consistent with the length of 1 / 256″ of the equator, and the coding marks are 0-7, and each division is 12.1cm. The eleventh part adopts the division consistent with the length of 1 / 2048″ of the equator, and the coding marks are 0-7, and each division is 1.5cm.

Images