A method and system for migrating and converting massive tile data

Abstract

The present application provides a method and system for migrating and converting massive tile data. The conversion method comprises the following steps: segmenting the tile data imported into the HBase database according to the latitude and longitude range and the format setting of the MBTiles file; establishing multiple tasks according to the segmented tile data, and placing the multiple tasks in the thread pool queue; running the tasks in the thread pool queue in parallel, migrating and converting the tile data in the HBase database into MBTiles files. The present invention reduces the design complexity and improves the running efficiency of the program.

Description

Technical Field

[0001] The present invention belongs to the technical field of geographic information systems, and in particular relates to a method and system for migrating and formatting massive tile data. Background Art

[0002] Currently, more and more map services are using tile technology, which requires storing terabytes of massive tile data. The HBase database can well meet this storage requirement. Generally speaking, storing tile data in the HBase database is a good choice.

[0003] Most map services require access to MBTiles files to obtain the tile data they need. This requires migrating the massive amount of tile data from the HBase database into MBTiles files. Furthermore, during the migration process, the program must handle the different indexing methods of the data in the two databases and convert the data indexes. GeoServer has a built-in program for generating MBTiles files, but this involves many intermediate steps, resulting in inefficient migration.

[0004] In practice, the estimated migration time for 38TB of tile data is five years, which is far from meeting actual production needs. Summary of the Invention

[0005] The purpose of the present invention is to provide a method and system for migrating and converting massive tile data into different formats, which reduces the complexity of the design and improves the running efficiency of the program by processing the tile data.

[0006] To this end, according to a first aspect of the present invention, a method for migrating and converting massive tile data is provided, comprising the following steps:

[0007] Split the tile data imported into the HBase database according to the latitude and longitude range and the format settings of the MBTiles file;

[0008] Creating multiple tasks based on the segmented tile data, and placing the multiple tasks in a thread pool queue;

[0009] The tasks in the thread pool queue are run in parallel to migrate and convert the tile data in the HBase database into MBTiles files.

[0010] Furthermore, the segmentation of tile data imported into the HBase database according to the latitude and longitude range and the format setting of the MBTiles file includes:

[0011] Determine the rowkey range of the tile data in the HBase database according to the latitude and longitude range of the tile data and the coordinate system conversion rules;

[0012] According to the format setting of the MBTiles file, the tile data in the rowkey range is divided into multiple independent area ranges.

[0013] Furthermore, the step of creating multiple tasks based on the segmented tile data includes:

[0014] Each task corresponds to tile data within an independent area.

[0015] Furthermore, the parallel execution of tasks in the thread pool queue to migrate and convert tile data in the HBase database into MBTiles files includes:

[0016] In each thread, the index of the tile data in the respective task is calculated and converted into the rowkey in the HBase database. Then, a read request is sent to each HRegion according to the storage order of the tiles in HBase.

[0017] Parse the rowkey of the acquired data into the MBTiles index format, then migrate it into the MBTiles SQLite file in batches and store the migrated data.

[0018] Furthermore, the process of running the tasks in the thread pool queue in parallel also includes:

[0019] It is used to store completed tasks in a log file. If the thread pool queue is interrupted, the unfinished tasks will be continued to run next time according to the information in the log file.

[0020] According to a second aspect of the present invention, a system for migrating and converting massive tile data is provided, comprising:

[0021] The data segmentation module is used to segment tile data imported into the HBase database according to the latitude and longitude ranges and the format settings of the MBTiles file;

[0022] A task creation module is used to create multiple tasks according to the segmented tile data and place the multiple tasks in the thread pool queue;

[0023] The migration and conversion module is used to run the tasks in the thread pool queue in parallel, migrate the tile data in the HBase database and convert it into MBTiles files.

[0024] Furthermore, the data segmentation module further includes a determination unit and a segmentation unit:

[0025] A determination unit, configured to determine a rowkey range of the tile data in the HBase database according to the latitude and longitude range of the tile data and a coordinate system conversion rule;

[0026] The segmentation unit is used to segment the tile data in the rowkey range into multiple independent area ranges according to the format setting of the MBTiles file.

[0027] Furthermore, the task creation module creates multiple tasks based on the segmented tile data, including:

[0028] Each task corresponds to tile data within an independent area.

[0029] Furthermore, the migration conversion module includes:

[0030] The data acquisition unit is used to calculate the index of the tile data in each task in each thread, convert the index into the rowkey in the HBase database, and then send a read request to each HRegion according to the storage order of the tiles in HBase;

[0031] The migration conversion unit is used to parse the rowkey of the acquired data into the MBTiles index format, then migrate it into the MBTiles SQLite file in batches and store the migrated data.

[0032] Furthermore, the migration conversion module is further configured to store completed tasks in a log file. If the thread pool queue is interrupted, the unfinished tasks will be continued to be executed next time according to the information in the log file.

[0033] The beneficial effects of the present invention are as follows: compared with the existing technology, the present invention adopts an efficient and multi-concurrent data conversion mechanism, which reduces the complexity of the design and improves the operating efficiency of the program. Under the same equipment conditions, it takes only 18 hours to migrate 38T tile data, which is an exponential performance improvement compared to the 5 years of the existing technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The above and / or additional aspects and advantages of the present invention will become apparent and readily understood from the following description of the embodiments with reference to the accompanying drawings, in which:

[0035] Figure 1 4 is a flowchart of a method for migrating and converting massive tile data according to an embodiment of the present invention.

[0036] Figure 2is a framework diagram of a system for migrating and converting massive tile data according to an embodiment of the present invention;

[0037] Figure 3 2 is a structural diagram of a data segmentation module in a system for migrating and converting massive tile data according to an embodiment of the present invention;

[0038] Figure 4 2 is a schematic structural diagram of a migration and conversion module in a massive tile data migration and format conversion system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present application and the features therein can be combined with each other.

[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0041] like Figure 1 As shown, according to an embodiment of the present invention, a method for migrating and converting massive tile data is provided, comprising the following steps:

[0042] S11. Segment the tile data imported into the HBase database according to the latitude and longitude range and the format setting of the MBTiles file;

[0043] Receive HBase tile data directly imported from the HBase database. First, according to the latitude and longitude ranges passed in by the HBase tile data and the coordinate system conversion rules, determine its rowkey range in the HBase database. Then, according to the format settings of the MBTiles file, split the tile data with the determined rowkey range in the HBase database into multiple independent area ranges according to the format of the MBTiles file.

[0044] S12, creating multiple tasks according to the segmented tile data, and placing the multiple tasks in a thread pool queue;

[0045] The tile data that has been divided into multiple independent area ranges is used to create multiple tasks, each task corresponding to the tile data within an independent area range, and the multiple tasks are placed in a thread pool queue.

[0046] S13. Running the tasks in the thread pool queue in parallel, migrating the tile data in the HBase database and converting it into MBTiles files;

[0047] Each thread placed in the thread pool queue calculates the index of the tile data in its own task and converts the index into the rowkey in the HBase database. It then sends a read request to each HRegion according to the storage order of the tiles in HBase. After obtaining the requested data, it parses the rowkey of the obtained data into the MBTiles index format. The data that has been parsed into the MBTiles index format is then migrated in batches to the MBTiles SQLite file and stored.

[0048] The method for migrating and converting massive tile data will store completed slicing tasks in a log file during the data migration and conversion process. If the thread pool queue is interrupted, the unfinished tasks will continue to run next time according to the information in the log file.

[0049] By migrating and converting massive tile data, the migrated data can communicate directly with the HBase database, without the need to obtain tile data through the wms service. The wms service is also obtained by accessing the HBase database. Direct access to the HBase database reduces many unnecessary HTTP requests, which greatly reduces the complexity of the design. At the same time, the completed tasks will be recorded in the log file during the migration and conversion process to ensure that no repeated data extraction will be performed during the subsequent migration process, thereby improving the operation efficiency of the program.

[0050] The embodiment of the present invention also provides a system for migrating and converting massive tile data. Figure 2 Shown, including:

[0051] The data segmentation module 21 is used to segment the tile data imported into the HBase database according to the latitude and longitude range and the format setting of the MBTiles file;

[0052] like Figure 3 As shown, the data segmentation module 21 specifically includes:

[0053] A determining unit 211 is configured to determine a rowkey range of the tile data in the HBase database according to the latitude and longitude range of the tile data and a coordinate system conversion rule;

[0054] The segmentation unit 222 is configured to segment the tile data in the rowkey range into a plurality of mutually independent area ranges according to the format setting of the MBTiles file.

[0055] The HBase tile data directly imported from the HBase database is received by the determination unit 211 in the data segmentation module 21. The rowkey range in the HBase database is first determined according to the latitude and longitude range of the HBase tile data input according to the coordinate system conversion rules. Then, according to the format setting of the MBTiles file, the tile data with the determined rowkey range in the HBase database is divided into multiple independent area ranges according to the format of the MBTiles file through the segmentation unit 212 in the data segmentation module 21.

[0056] The task creation module 22 is configured to receive tile data that has been segmented into multiple independent regions by the segmentation unit 212 , create multiple tasks, and then place the multiple tasks in a thread pool queue.

[0057] Each task corresponds to tile data within an independent area.

[0058] The migration and conversion module 23 is used to run the task placed in the thread pool queue by the task establishment module 22, migrate the tile data in the HBase database and convert it into MBTiles files;

[0059] like Figure 4 As shown, the migration conversion module 23 specifically includes:

[0060] The acquisition unit 231 is used to calculate the index of the tile data in each task in each thread and convert the index into a rowkey in the HBase database, and then send a read request to each HRegion according to the storage order of the tiles in HBase;

[0061] The migration conversion unit 232 is used to parse the rowkey of the acquired data into the MBTiles index format, and then migrate the data into the MBTiles SQLite file in batches and store the migrated data;

[0062] In the task establishment module 22, each thread placed in the thread pool queue calculates the index of the tile data in its respective task, converts the index into the rowkey in the HBase database through the acquisition unit 231 in the migration conversion module 23, and then sends a read request to each HRegion according to the storage order of the tiles in HBase. After obtaining the requested data, the rowkey of the obtained data is parsed into the index format of MBTiles through the migration conversion unit 232 in the migration conversion module 23, and the data converted into the MBTiles index format is migrated to the Sqlite file of MBTiles in batches, and the migrated data is stored.

[0063] The system for massive tile data migration and format conversion will also store the completed slicing tasks in a log file through the migration conversion module 23 during the data migration and conversion process. If the thread pool queue is interrupted, the next time the program is started, the next slicing migration task will be continued according to the information in the log file.

[0064] Explanation of terms:

[0065] Tile: The tile map pyramid model is a multi-resolution hierarchical model. From the bottom to the top of the tile pyramid, the resolution decreases, but the represented geographic range remains unchanged. First, determine the number of zoom levels N that the map service platform will provide. The map image with the highest zoom level and the largest map scale is used as the bottom layer of the pyramid, that is, the 0th layer. It is divided into blocks. Starting from the upper left corner of the map image, it is cut from left to right and from top to bottom into square map tiles of the same size (for example, 256x256 pixels), forming the 0th layer tile matrix. Based on the 0th layer map image, the 1st layer map image is generated by synthesizing 2x2 pixels into one pixel. It is also divided into blocks, divided into square map tiles of the same size as the next layer, forming the 1st layer tile matrix. The same method is used to generate the 2nd layer tile matrix. This process continues until the N-1st layer, forming the entire tile pyramid.

[0066] HBase: A distributed, column-oriented open source database suitable for storing massive amounts of data.

[0067] MBTiles: MBTiles is an open source standard hosted by MAPBOX that seeks to standardize and improve the efficiency of tiled maps. Essentially, it consists of a SQLite package containing 256x256 tiled map images. This database indexing improves tile indexing efficiency, significantly improving the efficiency of tile distribution in folders.

[0068] GeoServer: GeoServer is a J2EE implementation of the OpenGIS Web server specification. GeoServer can be used to easily publish map data, allowing users to update, delete, and insert feature data. GeoServer can easily and quickly share spatial geographic information among users.

[0069] Rowkey: In HBase, rowkey can uniquely identify a row of records and can quickly locate data in HBase.

[0070] HRegionServer: HBase service deployed on a physical server.

[0071] HRegion: The representation of a Region in an HRegionServer.

[0072] Region: is the basic unit of HBase data storage and management.

[0073] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0074] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A method for migrating and converting massive tile data, characterized in that: include: According to the latitude and longitude range and the format setting of the MBTiles file, the tile data imported into the HBase database is segmented. Receive HBase tile data directly imported from the HBase database, first determine its rowkey range in the HBase database according to the latitude and longitude ranges passed in by the HBase tile data and the coordinate system conversion rules, and then divide the tile data with the determined rowkey range in the HBase database into multiple independent area ranges according to the format setting of the MBTiles file; Creating multiple tasks according to the segmented tile data, and placing the multiple tasks in a thread pool queue; Running the tasks in the thread pool queue in parallel, migrating the tile data in the HBase database and converting it into MBTiles files, wherein: Each thread placed in the thread pool queue calculates the index of the tile data in its own task and converts the index into the rowkey in the HBase database. It then sends a read request to each HRegion according to the storage order of the tiles in HBase. After obtaining the requested data, it parses the rowkey of the obtained data into the MBTiles index format. The data that has been parsed into the MBTiles index format is then migrated in batches to the MBTiles Sqlite file and the migrated data is stored.

2. The method according to claim 1, characterized in that The step of establishing multiple tasks according to the segmented tile data includes: Each task corresponds to tile data within an independent area.

3. The method according to claim 1, characterized in that The parallel execution of tasks in the thread pool queue also includes: It is used to store completed tasks in a log file. If the thread pool queue is interrupted, the unfinished tasks will continue to run next time according to the information in the log file.

4. A system for mass tile data migration and format conversion, characterized in that: include: The data segmentation module is used to segment the tile data imported into the HBase database according to the latitude and longitude range and the format setting of the MBTiles file. Receive HBase tile data directly imported from the HBase database, first determine its rowkey range in the HBase database according to the latitude and longitude ranges passed in by the HBase tile data and the coordinate system conversion rules, and then divide the tile data with the determined rowkey range in the HBase database into multiple independent area ranges according to the format setting of the MBTiles file; A task creation module, used to create multiple tasks according to the segmented tile data, and place the multiple tasks in a thread pool queue; The migration and conversion module is used to run the tasks in the thread pool queue in parallel, migrate the tile data in the HBase database and convert it into MBTiles files, wherein: Each thread placed in the thread pool queue calculates the index of the tile data in its own task and converts the index into the rowkey in the HBase database. It then sends a read request to each HRegion according to the storage order of the tiles in HBase. After obtaining the requested data, it parses the rowkey of the obtained data into the MBTiles index format. The data that has been parsed into the MBTiles index format is then migrated in batches to the MBTiles Sqlite file and the migrated data is stored.

5. The system according to claim 4, characterized in that The task establishment module is used to establish multiple tasks according to the segmented tile data, including: Each task corresponds to tile data within an independent area.

6. The system according to claim 4, characterized in that The migration conversion module is also used to store the completed tasks in a log file. If the thread pool queue is interrupted, the unfinished tasks will continue to be executed next time according to the information in the log file.

Images