A method and device for quickly exporting regional sub-datasets from Hbase tile storage

Abstract

The present application provides a method and device for quickly exporting a regional sub-dataset from HBase tile storage. The method comprises: obtaining a rectangular area of ​​tile data to be exported according to the longitude and latitude coordinates of two coordinate points; determining a scan boundary according to the rectangular area; reading tile data in each Region of HBase in parallel according to the scan boundary, and storing it in a global variable HashMap; storing the tile data in the global variable HashMap in a SQLite database in a read-write separation manner. The present invention realizes the function of user-defined export of tile data area and greatly improves the efficiency of exporting tile data.

Description

Technical Field

[0001] The present invention relates to computer technology, and in particular to a method and device for quickly exporting a regional sub-dataset from HBase tile storage. Background Art

[0002] The stand-alone digital earth product requires map data that can be accessed by a single machine, and the current slice data in HBase cannot meet the requirements. SQLite provides a specification that allows millions of tile data to be stored in a file, so one of the current mature stand-alone solutions is to store tile data in the SQLite database as mbtiles. To access it through GeoWebCache, it is necessary to migrate part of the tile data stored in HBase to SQLite.

[0003] However, how to write the massive tile data in HBase into SQLite is a big problem. The usual method is to generate MBTiles files through GeoWebCache's built-in method. However, it takes about 5 years to export a global map in this way. The efficiency is obviously too low, and it is not possible to customize the area export, and the flexibility is also poor. Summary of the invention

[0004] The object of the present invention is to provide a method and device for quickly exporting a regional sub-dataset from an HBase tile storage, so that a user can customize the export area of ​​tile data and improve the export efficiency.

[0005] To this end, according to a first aspect of the present invention, a method for quickly exporting a regional sub-dataset from HBase tile storage is provided, comprising:

[0006] Obtain the rectangular area of ​​tile data to be exported according to the latitude and longitude coordinates of the two coordinate points;

[0007] Determine the scan boundary according to the rectangular area;

[0008] Read tile data in each Region of HBase in parallel according to the scan boundary, and store it in the global variable HashMap;

[0009] The tile data in the global variable HashMap is stored in the SQLite database in a read-write separation manner.

[0010] Furthermore, the two coordinate points are not at the same longitude or latitude.

[0011] Further, determining the scan boundary according to the rectangular area includes:

[0012] The startKey and endKey of each layer and each column of tiles in the rectangular area are calculated, and the startKey and endKey are scan boundaries.

[0013] Furthermore, the reading of tile data in each Region of HBase in parallel according to the scan boundary and storing the tile data in the global variable HashMap includes:

[0014] For each layer of the tile pyramid, calculate the layer, row, and column (lrc) information of the starting and ending tiles in the rectangular area;

[0015] Distribute tasks to the thread pool according to the lrc information to generate multiple thread tasks;

[0016] Executing the multiple thread tasks in parallel, each thread task reads corresponding tile data from a Region according to the scan boundary;

[0017] Store the read tile data in the global variable HashMap;

[0018] In the global variable HashMap, key is used to store tile data read from Region, value is listClass, two Arraylist linked lists and an index variable are defined in the listClass class, the index variable is used to distinguish the functions of the Arraylist linked lists, index0 indicates that the first Arraylist is responsible for reading data and the second Arraylist is responsible for writing data, index1 indicates that the first Arraylist is responsible for writing data and the second Arraylist is responsible for reading data.

[0019] Furthermore, storing the tile data in the global variable HashMap in a read-write separation manner into a SQLite database includes:

[0020] Determine the Arraylist linked list function according to the index variable in the global variable HashMap;

[0021] According to the function of the Arraylist linked list, the tile data in the global variable HashMap is stored in the SQLite database in a read-write separation manner.

[0022] According to a second aspect of the present invention, there is provided a device for quickly exporting a regional sub-dataset from an HBase tile storage, comprising:

[0023] An acquisition module is used to acquire a rectangular area of ​​tile data to be exported according to the longitude and latitude coordinates of two coordinate points;

[0024] A determination module, used to determine a scan boundary according to the rectangular area;

[0025] A reading module, used to read tile data in each Region of HBase in parallel according to the scan boundary, and store it in a global variable HashMap;

[0026] The storage module is used to store the tile data in the global variable HashMap into the SQLite database in a read-write separation manner.

[0027] Furthermore, the determination module is specifically used for:

[0028] The startKey and endKey of each layer and each column of tiles in the rectangular area are calculated, and the startKey and endKey are scan boundaries.

[0029] Furthermore, the reading module is specifically used for:

[0030] For each layer of the tile pyramid, calculate the layer, row, and column (lrc) information of the starting and ending tiles in the rectangular area;

[0031] Distribute tasks to the thread pool according to the lrc information to generate multiple thread tasks;

[0032] Executing the multiple thread tasks in parallel, each thread task reads corresponding tile data from a Region according to the scan boundary;

[0033] Store the read tile data in the global variable HashMap.

[0034] Furthermore, in the global variable HashMap, the key is used to store tile data read from the Region, the value is listClass, two Arraylist linked lists and an index variable are defined in the listClass class, and the index variable is used to distinguish the functions of the Arraylist linked lists. Index0 indicates that the first Arraylist is responsible for reading data and the second Arraylist is responsible for writing data. Index1 indicates that the first Arraylist is responsible for writing data and the second Arraylist is responsible for reading data.

[0035] Furthermore, the storage module is specifically used for:

[0036] Determine the Arraylist linked list function according to the index variable in the global variable HashMap;

[0037] According to the function of the Arraylist linked list, the tile data in the global variable HashMap is stored in the SQLite database in a read-write separation manner.

[0038] Additional aspects and advantages of the present invention will be given in part in the following description, and in part will be obvious from the following description, or will be learned through practice of the present invention.

[0039] The beneficial effects of the present invention are as follows: by first converting the region boundary into the scan boundary of HBase, and then reading data from HBase in parallel with multi-threaded scheduling, the efficiency of exporting tile data is greatly improved; by storing the exported data in a SQLite database in a read-write separation manner, the speed of data storage is improved, so that there is no bottleneck between reading and writing. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0041] Figure 1 The present invention is a flowchart of a method for quickly exporting a regional sub-dataset from HBase tile storage according to an embodiment of the present invention.

[0042] Figure 2 The present invention is a framework diagram of a device for quickly exporting a regional sub-dataset from HBase tile storage according to an embodiment of the present invention.

[0043] Figure 3 1 is an architectural diagram for quickly exporting a regional sub-dataset from HBase tile storage according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0045] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited to the specific embodiments disclosed below.

[0046] The terms used in the scheme of the present invention are explained as follows:

[0047] Tile: refers to a map within a certain range that is cut into a number of rows and columns of square grid images according to a certain size and format, according to the zoom level or scale. The sliced ​​square grid images are figuratively called tiles.

[0048] GWC: GeoWebCache is a Java web application for caching map tiles from various sources, such as OGC Web Map Service (WMS). It implements various service interfaces (such as WMS-C, WMTS, TMS, Google MapsKML, Virtual Earth) to accelerate and optimize map image transmission.

[0049] MBTiles: MBTiles is a specification for storing arbitrary tiled map data in a SQLite database, ready for use and efficient transfer.

[0050] Image pyramid: refers to the storage and display of different resolutions according to user needs under the same spatial reference, forming a pyramid structure with resolutions from coarse to fine and data volume from small to large. Image pyramid structure is used for image coding and progressive image transmission. It is a typical hierarchical data structure form, suitable for multi-resolution organization of raster data and image data, and is also a lossy compression method for raster data or image data.

[0051] like Figure 1 As shown, according to an embodiment of the present invention, a method for quickly exporting a regional sub-dataset from HBase tile storage is provided, comprising the following steps:

[0052] S11, obtaining a rectangular area of ​​tile data to be exported according to the latitude and longitude coordinates of the two coordinate points;

[0053] Wherein, the two coordinate points are not at the same longitude or latitude, that is, a rectangular area can be confirmed with the two coordinate points as diagonal points of a rectangle. The longitude and latitude coordinates of the two coordinate points can be obtained according to the user's input, or generated according to the user's click on the map interface.

[0054] S12, determining a scan boundary according to the rectangular area;

[0055] The startKey and endKey of each layer and each column of tiles in the rectangular area are calculated, and the startKey and endKey are scan boundaries.

[0056] S13, reading tile data in each Region of Hbase in parallel according to the scan boundary, and storing it in a global variable HashMap;

[0057] For each layer of the tile pyramid, the layer, row, and column (lrc) information of the tiles that start and end in the rectangular area is calculated, and tasks are distributed to the thread pool according to the layer, row, and column information to generate multiple thread tasks. The multiple thread tasks read the corresponding tile data from each Region in parallel according to the scan boundary.

[0058] In the global variable HashMap, the key value is used to store tile data read from Region, and the value is listClass. Two Arraylist linked lists and an index variable are defined in the listClass class. The index variable is used to distinguish the functions of the Arraylist linked lists. Index0 indicates that the first Arraylist is responsible for reading data and the second Arraylist is responsible for writing data. Index1 indicates that the first Arraylist is responsible for writing data and the second Arraylist is responsible for reading data. The "read" refers to reading data from the HashMap, and the "write" refers to writing into the SQLite database.

[0059] S14, storing the tile data in the global variable HashMap into the SQLite database in a read-write separation manner;

[0060] When reading tile data, first determine whether the tile data is stored in the key value of the HashMap. If not, initialize it. If it is, obtain the Arraylist linked list in the HashMap responsible for reading data according to the index variable, lock the Arraylist linked list through the synchronized method, and write data to the Arraylist. When the data in the key value reaches the specified threshold (for example, 10000), obtain the Arraylist linked list responsible for reading data according to the index value, lock the Arraylist linked list through the synchronized method, and change the index variable at the same time, so that the Arraylist linked list for reading data becomes the Arraylist linked list for writing data, and write to the database, and repeat this process until all tile data are stored in the SQLite database.

[0061] In this embodiment, by first converting the region boundary into the scan boundary of HBase, and then reading data from HBase in parallel with multi-threaded scheduling, the efficiency of exporting tile data is greatly improved; the exported data is stored in the SQLite database by separating read and write, which improves the speed of data storage and eliminates bottlenecks between reading and writing.

[0062] The embodiment of the present invention also provides a device for quickly exporting a regional sub-dataset from HBase tile storage, such as Figure 2 As shown, including:

[0063] The acquisition module 21 is used to acquire the rectangular area of ​​tile data to be exported according to the two-point longitude and latitude coordinates input;

[0064] Wherein, the two coordinate points are not at the same longitude or latitude, that is, a rectangular area can be confirmed with the two coordinate points as diagonal points of a rectangle. The longitude and latitude coordinates of the two coordinate points can be obtained according to the user's input, or generated according to the user's click on the map interface.

[0065] A determination module 22, configured to determine a scan boundary according to the rectangular area;

[0066] The startKey and endKey of each layer and each column of tiles in the rectangular area are calculated, and the startKey and endKey are scan boundaries.

[0067] A reading module 23 is used to read tile data in each Region of Hbase in parallel according to the scan boundary and store it in a global variable HashMap;

[0068] For each layer of the tile pyramid, the layer, row, and column (lrc) information of the tiles that start and end in the rectangular area is calculated, and tasks are distributed to the thread pool according to the layer, row, and column information to generate multiple thread tasks. The multiple thread tasks read the corresponding tile data from each Region in parallel according to the scan boundary.

[0069] In the global variable HashMap, the key value is used to store tile data read from Region, and the value is listClass. Two Arraylist linked lists and an index variable are defined in the listClass class. The index variable is used to distinguish the functions of the Arraylist linked lists. Index0 indicates that the first Arraylist is responsible for reading data and the second Arraylist is responsible for writing data. Index1 indicates that the first Arraylist is responsible for writing data and the second Arraylist is responsible for reading data. The "read" refers to reading data from Hash, and the "write" refers to writing into the SQLite database.

[0070] The storage module 24 is used to store the tile data in the global variable HashMap into the SQLite database in a read-write separation manner.

[0071] When reading tile data, first determine whether the tile data is stored in the key value of the HashMap. If not, initialize it. If it is, obtain the Arraylist linked list in the HashMap responsible for reading data according to the index variable, lock the Arraylist linked list through the synchronized method, and write data to the Arraylist. When the data in the key value reaches the specified threshold (for example, 10000), obtain the Arraylist linked list responsible for reading data according to the index value, lock the Arraylist linked list through the synchronized method, and change the index variable at the same time, so that the Arraylist linked list for reading data becomes the Arraylist linked list for writing data, and write to the database, and repeat this process until all tile data are stored in the SQLite database.

[0072] In this embodiment, the tile data to be exported is obtained through the acquisition module, and then the scan boundary of the tile data is determined through the determination module. The reading module reads the tile data in each Region of Hbase in parallel according to the scan boundary and stores it in the global variable HashMap. Finally, the tile data in the global variable HashMap is stored in the SQLite database in a read-write separation manner through the storage module, which improves the speed of data storage and eliminates bottlenecks between reading and writing.

[0073] Specific implementation method Figure 3 As shown:

[0074] First, a rectangular area is determined according to two coordinates that are not at the same longitude and latitude, and then the tile scan boundary is obtained by calculation, that is, the startKey and endKey values ​​of each layer and column of tiles. For each layer of the tile pyramid, the layer, row, and column (lrc) information of the tiles that start and end in the rectangular area is calculated, and tasks are distributed to the thread pool according to these layer, row, and column information to generate multiple scanner thread tasks. The multiple scanner thread tasks read corresponding tile data from the Region in parallel according to the scan boundary.

[0075] A global variable HashMap is set to store the corresponding tile data read from each Region1 and Region2 in parallel according to the scan boundary. In the HashMap, the key value is used to store the tile data read from Region1 and Region2, and the value is the listClass class. Two Arraylist linked lists and an index variable are defined in the listClass class. Index0 indicates that the first Arraylist linked list is responsible for reading data from the HashMap, and the second Arraylis linked list is responsible for writing the read data into the SQLite database. Index1 indicates that the second Arraylist linked list is responsible for reading data from the HashMap, and the first Arraylist linked list is responsible for writing the read data into the SQLite database.

[0076] As an example, when the tile data stored in the key value in the HashMap reaches 10000, the index value is index1, the thread queries the HashMap, obtains the index feedback value index1, reads the first Arraylist linked list, and writes the data in the first Arraylis linked list into the SQLite database. At this time, the second Arraylist linked list is responsible for reading data from the HashMap, and uses the synchronized method to lock the first Arraylist linked list until all the data in the first Arraylist linked list is stored in the SQLite database and then unlocked. At the same time, the index value becomes index0, that is, the second Arraylist linked list is used to store the read tile data into the SQLite database, and the first Arraylist linked list is responsible for reading data from the HashMap, and so on and so forth until all tile data are stored in the SQLite database.

[0077] According to tests, it takes more than 5 years to fully export the global map using the usual method, but it only takes 1 day using this method.

[0078] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0079] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for quickly exporting a regional sub-dataset from HBase tile storage, characterized in that: include: Obtain the rectangular area of ​​tile data to be exported according to the latitude and longitude coordinates of the two coordinate points; Determine the scan boundary according to the rectangular area, including Calculate the startKey and endKey of each layer and each column of tiles in the rectangular area, where the startKey and endKey are scan boundaries; Read the tile data in each Region of HBase in parallel according to the scan boundary and store it in the global variable HashMap, including For each layer of the tile pyramid, calculate the layer, row, and column information of the tiles that start and end in the rectangular area, distribute tasks to the thread pool according to the layer, row, and column information, and generate multiple thread tasks. The multiple thread tasks read corresponding tile data from each Region in parallel according to the scan boundary; In the global variable HashMap, the key value is used to store tile data read from Region, and the value is listClass. Two Arraylist linked lists and an index variable are defined in the listClass class. The index variable is used to distinguish the functions of the Arraylist linked lists. Index0 indicates that the first Arraylist is responsible for reading data and the second Arraylist is responsible for writing data. Index1 indicates that the first Arraylist is responsible for writing data and the second Arraylist is responsible for reading data. The reading refers to reading data from the HashMap, and the writing refers to writing into the SQLite database. The tile data in the global variable HashMap is stored in the SQLite database in a read-write separation manner.

2. The method according to claim 1, characterized in that The two coordinate points are not at the same longitude or latitude.

3. The method according to claim 1, characterized in that The step of storing the tile data in the global variable HashMap in the SQLite database in a read-write separation manner includes: Determine the Arraylist linked list function according to the index variable in the global variable HashMap; According to the function of the Arraylist linked list, the tile data in the global variable HashMap is stored in the SQLite database in a read-write separation manner.

4. A device for quickly exporting a regional sub-dataset from HBase tile storage, characterized in that: include: An acquisition module is used to acquire a rectangular area of ​​tile data to be exported according to the longitude and latitude coordinates of two coordinate points; A determination module, used to determine the scan boundary according to the rectangular area, including Calculate the startKey and endKey of each layer and each column of tiles in the rectangular area, where the startKey and endKey are scan boundaries; The reading module is used to read the tile data in each Region of HBase in parallel according to the scan boundary and store it in the global variable HashMap, including For each layer of the tile pyramid, calculate the layer, row, and column information of the tiles that start and end in the rectangular area, distribute tasks to the thread pool according to the layer, row, and column information, and generate multiple thread tasks. The multiple thread tasks read corresponding tile data from each Region in parallel according to the scan boundary; In the global variable HashMap, the key value is used to store tile data read from Region, and the value is listClass. Two Arraylist linked lists and an index variable are defined in the listClass class. The index variable is used to distinguish the functions of the Arraylist linked lists. Index0 indicates that the first Arraylist is responsible for reading data and the second Arraylist is responsible for writing data. Index1 indicates that the first Arraylist is responsible for writing data and the second Arraylist is responsible for reading data. The reading refers to reading data from the HashMap, and the writing refers to writing into the SQLite database. The storage module is used to store the tile data in the global variable HashMap into the SQLite database in a read-write separation manner.

5. The device according to claim 4, characterized in that The determination module is specifically used for: The startKey and endKey of each layer and each column of tiles in the rectangular area are calculated, and the startKey and endKey are scan boundaries.

6. The device according to claim 4, characterized in that The reading module is specifically used for: For each layer of the tile pyramid, calculate the layer, row, and column information of the starting and ending tiles in the rectangular area; Distribute tasks to the thread pool according to the layer, row, and column information to generate multiple thread tasks; Executing the multiple thread tasks in parallel, each thread task reads corresponding tile data from a Region according to the scan boundary; Store the read tile data in the global variable HashMap.

7. The device according to claim 4, characterized in that In the global variable HashMap, key is used to store tile data read from Region, value is listClass, two Arraylist linked lists and an index variable are defined in the listClass class, the index variable is used to distinguish the functions of the Arraylist linked lists, index0 indicates that the first Arraylist is responsible for reading data and the second Arraylist is responsible for writing data, index1 indicates that the first Arraylist is responsible for writing data and the second Arraylist is responsible for reading data.

8. The device according to claim 4, characterized in that The storage module is specifically used for: Determine the Arraylist linked list function according to the index variable in the global variable HashMap; According to the function of the Arraylist linked list, the tile data in the global variable HashMap is stored in the SQLite database in a read-write separation manner.

Images