A GeoWebCache distributed storage method and device based on Hbase

Abstract

The present application provides a GeoWebCache distributed storage method and device based on Hbase. The storage method includes the following steps: storing tile data cached in GeoWebcache in Hbase; grouping and merging tile data stored in Hbase; storing the processed tile data in the Moosefs file system. The present invention integrates small tile data into an Hfile large file data structure for storage, without generating unnecessary storage overhead; through rowkey, required tile data can be quickly obtained from massive data to meet the demand for reading speed; by storing tile data in a distributed file system through Hbase, the capacity can be increased by adding storage nodes, providing scalability for the storage of tile data; at the same time, the distributed file system shares the read and write load, improving the read and write performance.

Description

Technical Field

[0001] The present invention belongs to the field of geographic information technology, and in particular relates to a GeoWebCache distributed storage method and device based on Hbase. Background Art

[0002] Currently, GeoServer supports WCS 1.0, WMS 1.1.1, and WFS 1.0 standards, and integrates the GeoWebCache component by default.

[0003] GeoWebCache provides tile caching function for WMS services. When the client requests map service, GeoWebCache intercepts the WMS call request and returns the generated Tiles cache. If the request does not have a corresponding server cache, GeoServer will generate the corresponding Tiles for the requested layer. For maps with relatively small data, the tile caching function provided by GeoWebCache is completely sufficient, but if it is a map with a large amount of data, the tile caching function provided by GeoWebCache often cannot meet the storage requirements.

[0004] Setting the cache directory of GeoWebcache to the HDFS file system can achieve large-capacity storage. However, in the HDFS file system, any file, directory or block is represented by a metadata in the memory of the NameNode node, which is limited by the physical memory capacity of the NameNode. Therefore, the effect of processing small files (files whose size is significantly smaller than the block size on HDFS (default 128MB)) is very inefficient. Massive small files will cause a large amount of metadata to be generated, resulting in large redundancy in storage. Reading the required tiles from massive small files requires high performance of the database, which seriously affects data reading and writing performance. Summary of the invention

[0005] The purpose of the present invention is to provide a GeoWebCache distributed storage method and device based on Hbase, which realizes online capacity expansion and improves data reading and writing performance by processing tile data.

[0006] To this end, according to a first aspect of the present invention, a GeoWebCache distributed storage method based on Hbase is provided, comprising the following steps:

[0007] Store tile data cached in GeoWebcache in Hbase;

[0008] Group and merge tile data stored in Hbase;

[0009] The processed tile data is stored in the Moosefs file system.

[0010] Furthermore, storing the tile data cached in GeoWebcache in Hbase includes:

[0011] The tile data cached in GeoWebcache is first encoded according to the layer and longitude and latitude, and then the data obtained by hashing the encoding is stored in Hbase.

[0012] Furthermore, the grouping and merging of tile data stored in Hbase includes:

[0013] The tile data is grouped according to the rowkey in Hbase, and the same column family is merged into the same Hfile according to the storage order of the rowkey.

[0014] Further, storing the processed tile data into the Moosefs file system includes:

[0015] When the data volume of the tile data reaches a threshold, the processed tile data is stored in the Moosefs file system.

[0016] Furthermore, it is characterized in that it also includes:

[0017] The Moosefs file system evenly distributes the processed tile data to multiple nodes according to the size of the total space of the nodes.

[0018] According to a second aspect of the present invention, there is provided a GeoWebCache distributed storage device based on Hbase, comprising:

[0019] A GeoWebcache unit, used for caching tile data and storing the cached tile data in Hbase;

[0020] Hbase database, used for grouping and merging the tile data, and storing the processed tile data in a Moosefs file system;

[0021] The Moosefs file system is used to store the processed tile data.

[0022] Furthermore, it is characterized in that the GeoWebcache unit includes:

[0023] The tile data cached in GeoWebcache is first encoded according to the layer and longitude and latitude, and then the data obtained by hashing the encoding is stored in Hbase.

[0024] Furthermore, the Hbase database includes:

[0025] The tile data is grouped according to the rowkey in Hbase, and the same column family is merged into the same Hfile according to the storage order of the rowkey.

[0026] Furthermore, the Moosefs file system includes:

[0027] When the data volume of the tile data reaches a threshold, the processed tile data is stored in the Moosefs file system.

[0028] Furthermore, it is characterized in that it also includes:

[0029] The Moosefs file system evenly distributes the processed tile data to multiple nodes according to the size of the total space of the nodes.

[0030] The beneficial effects of the present invention are as follows: the present invention integrates small tile data into an Hfile large file data structure for storage, so that the file system will not generate metadata for each small file and will not generate unnecessary storage overhead; the required tile data can be quickly obtained from massive data through rowkey to meet the demand for reading speed; the tile data is stored in a distributed file system through Hbase, and the capacity can be increased by adding storage nodes, providing scalability for the storage of tile data; at the same time, the distributed file system shares the read and write loads and improves the read and write performance.

[0031] Additional aspects and advantages of the present invention will be given in part in the following description, and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0033] Figure 1 The figure is a flow chart of a GeoWebCache distributed storage method based on Hbase according to an embodiment of the present invention.

[0034] Figure 2 4 is a framework diagram of a GeoWebCache distributed storage device based on Hbase according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0036] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0037] like Figure 1 As shown, according to an embodiment of the present invention, a GeoWebCache distributed storage method based on Hbase is provided, comprising the following steps:

[0038] S11, storing the tile data cached in GeoWebcache in Hbase;

[0039] GeoWebcache places the cached tile data in the specified directory and generates the cache directory through the Java program;

[0040] For the cached tile data, it is encoded according to the tile data layer and longitude and latitude, and the encoded tile data is hashed and transferred to Hbase according to the generated cache directory, and the temporary cache directory is deleted at the same time.

[0041] For example, if the tile data layer in the cache is 12, and the longitude and latitude are projected by Mercator and Hilbert encoded to obtain row and column values ​​of 450 and 350 respectively, then the unhashed rowkey primary key of this tile in Hbase is 12_450_350. In order to reduce the hot spot problem of data reading, the rowkey is hashed and the obtained rowkey value is 1610693504. The adjacent tile data 12_450_300 is hashed to obtain a rowkey of 1610693349. After hashing, the two will not be adjacent.

[0042] S12, grouping and merging the tile data stored in Hbase;

[0043] For tile data stored in Hbase, the tile data is first grouped according to the rowkey in Hbase, and then tiles of the same column family are merged into the same Hfile according to the storage order of the rowkey.

[0044] A tile data is a row of data in Hbase, including two column families, tile name and tile data. The size of each tile data is about 100k. HBase will merge multiple rows of data into a large Hfile file (128M or 256M in size) as a storage unit and store it in the Hdfs file system, which is equivalent to merging about 1280 tiles into one file for storage. When a file is stored in Hdfs, a metadata will be generated. Compared with single tile storage, this storage method will greatly reduce metadata generation and management, and improve the reading and writing speed of data in the Hdfs file system. In addition, the hash method will reduce data hot spots and optimize read and write performance.

[0045] S13, storing the processed tile data in the Moosefs file system

[0046] When the amount of tile data reaches the threshold, the Hfile file is flushed into the Moosefs file system. The Moosefs file system will evenly distribute the Hfile file to multiple nodes according to the size of the total node space.

[0047] The embodiment of the present invention also provides a GeoWebCache distributed storage system based on Hbase, such as Figure 2 As shown, including:

[0048] A GeoWebcache unit 21 is used to cache tile data and store the cached tile data in Hbase;

[0049] In the GeoWebcache unit 21, the cached tile data is placed in a specified directory, encoded according to the tile data layer and longitude and latitude, and a cache directory is generated through a Java program. After the encoded tile data is hashed, it is transferred to the Hbase database 22 according to the generated cache directory, and the temporary cache directory is deleted and cleaned at the same time.

[0050] For example, if the tile data layer in the cache is 12, and the longitude and latitude are projected by Mercator and Hilbert encoded to obtain row and column values ​​of 450 and 350 respectively, then the unhashed rowkey primary key of this tile in Hbase is 12_450_350. In order to reduce the hot spot problem of data reading, the rowkey is hashed and the obtained rowkey value is 1610693504. The adjacent tile data 12_450_300 is hashed to obtain a rowkey of 1610693349. After hashing, the two will not be adjacent.

[0051] Hbase database 22, used for grouping and merging the tile data, and storing the processed tile data in Moosefs file system 23;

[0052] The tile data stored in the Hbase database 22 is first grouped according to the rowkey in the Hbase database 22, and then the tiles of the same column family are merged into the same Hfile according to the storage order of the rowkey, and the Hfile is flushed into the Moosefs file system 23.

[0053] A tile data is a row of data in Hbase, including two column families, tile name and tile data. The size of each tile data is about 100k. HBase will merge multiple rows of data into a large Hfile file (128M or 256M in size) as a storage unit and store it in the Hdfs file system, which is equivalent to merging about 1280 tiles into one file for storage. When a file is stored in Hdfs, a metadata will be generated. Compared with single tile storage, this storage method will greatly reduce metadata generation and management, and improve the reading and writing speed of data in the Hdfs file system. In addition, the hash method will reduce data hot spots and optimize read and write performance.

[0054] The Moosefs file system 23 is used to store the processed tile data.

[0055] The Moosefs file system 23 evenly distributes the Hfile data stored in the Hbase database 22 to multiple nodes according to the size of the total space of the nodes.

[0056] The above method and system integrate small tile data into an Hfile large file data structure for storage, so that the file system will not generate metadata for each small file and will not generate unnecessary storage overhead; the required tile data can be quickly obtained from massive data through rowkey to meet the reading speed requirements; the tile data is stored in a distributed file system through Hbase, and the capacity can be increased by adding storage nodes, providing scalability for the storage of tile data; at the same time, the distributed file system shares the read and write load and improves the read and write performance.

[0057] After testing, the QPS of the dual-node Hbase and Moosefs cluster storage tiles can reach 12250 records / second, with a throughput of 122.5M / s, and the QPS of the scan mode reading tiles is 14229 records / second, with a throughput of 1423M / s. Hbase itself has a data compression mechanism, which reduces the space occupied by data on the disk to one-third of the size through compression, and metadata accounts for less than 1%, further saving storage space.

[0058] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0059] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A GeoWebCache distributed storage method based on Hbase, characterized in that: include: The tile data cached in GeoWebcache is first encoded according to the layer and longitude and latitude, and then the data obtained by hashing the encoding is stored in Hbase; The tile data stored in Hbase is grouped and merged, specifically including: grouping the tile data according to the rowkey in Hbase, and merging the same column family into the same Hfile according to the storage order of the rowkey; The processed tile data is stored in the Moosefs file system.

2. The method according to claim 1, characterized in that The storing the processed tile data into the Moosefs file system comprises: When the data volume of the tile data reaches a threshold, the processed tile data is stored in the Moosefs file system.

3. The method according to claim 1, characterized in that Also includes: The Moosefs file system evenly distributes the processed tile data to multiple nodes according to the size of the total space of the nodes.

4. A GeoWebCache distributed storage device based on Hbase, characterized in that: include: The GeoWebcache unit is used to encode the tile data cached in GeoWebcache according to the layer and longitude and latitude, and then store the data obtained by hashing the encoding in Hbase; The Hbase database is used to group and merge the tile data and store the processed tile data in the Moosefs file system, specifically including: grouping the tile data according to the rowkey in the Hbase, and merging the same column family into the same Hfile according to the storage order of the rowkey; The Moosefs file system is used to store the processed tile data.

5. The device according to claim 4, characterized in that The Moosefs file system includes: When the data volume of the tile data reaches a threshold, the processed tile data is stored in the Moosefs file system.

6. The device according to claim 4, characterized in that Also includes: The Moosefs file system evenly distributes the processed tile data to multiple nodes according to the size of the total space of the nodes.

Images