GIS big data processing storage and real-time rendering method based on HBase

Abstract

The invention discloses a GIS (Geographic Information System) big data processing storage and real-time rendering method based on HBase, which belongs to the technical field of general geographic information systems, and comprises the following steps: S1, accessing original geographic space data, partitioning the original geographic space data based on GeoHash, and carrying out vectorization loading to obtain service vector data; s2, carrying out hierarchical splitting on the service vector data by utilizing a pyramid model, and carrying out attribute or space aggregation on the split vector data; s3, storing the aggregated vector data into an HBase (Hadoop Base); and S4, according to an input loading range, obtaining the aggregated vector data from the HBase, and performing real-time rendering. Vector data in geographic data are loaded to the HBase, retrieval testing is carried out, and the retrieval response time can reach the millisecond level.

Description

Technical Field

[0001] This invention relates to the field of general geographic information system technology, and in particular to a method for GIS big data processing, storage, and real-time rendering based on HBase. Background Technology

[0002] Currently, traditional geographic information systems (GIS) primarily use relational databases such as PostGIS, an extension of PostgreSQL, for storage and real-time rendering. However, relational databases have begun to constrain the development of GIS in areas such as the management of massive geographic business vector data, high-concurrency read and write operations, and real-time rendering.

[0003] HBase's column-based storage technology and its high availability, high reliability, high performance, and scalability give it a natural advantage in processing geographic vector data with discrete and sparse characteristics. Therefore, there is an urgent need for a method to load vector data from geographic data into HBase for GIS big data processing, storage, and real-time rendering. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing relational databases in managing massive geographic business vector data, handling high-concurrency read and write operations, and real-time rendering, which have begun to restrict the development of geographic information systems. This invention provides a GIS big data processing, storage, and real-time rendering method based on HBase.

[0005] To address the aforementioned technical problems, the present invention provides the following technical solution: A method for GIS big data processing, storage, and real-time rendering based on HBase includes the following steps: S1: Access the raw geospatial data, partition the raw geospatial data based on GeoHash, and load it into a vector to obtain business vector data; S2: Use the pyramid model to split the business vector data into layers, and perform attribute or spatial aggregation on the split vector data; S3: Store the aggregated vector data in HBase; S4: Based on the input loading range, retrieve the aggregated vector data from the HBase and perform real-time rendering.

[0006] As a preferred embodiment of the present invention, step S1 includes: S11: Access raw geospatial data, which uses the wkb business vector data format and the file uses text data with a common delimiter. S12: The original geographic business vector data received is vectorized based on the Geotools open-source library.

[0007] As a preferred embodiment of the present invention, the vectorization construction based on the Geotools open source library in step S12 includes: converting structured coordinate data into different types of Geometry data, including conversion of points, lines, and surfaces based on latitude and longitude, as well as wkb binary data format; Based on the elements converted into Geometry data, and combined with data attribute data, the Geometry data is transformed into business vector data with business attributes.

[0008] As a preferred embodiment of the present invention, step S2 includes: S21: The business vector data is rasterized at different levels according to the pyramid model, and a certain level is set as the original high-precision data; S22: Based on the input aggregation level and vectorization level parameters, divide the gridded data into vector data level and aggregation data level; S23: The business vector data is hierarchically divided by a multi-level pyramid model to establish a LOD hierarchical structure with spatial topological relationships. The original geographic business vector data is aggregated by a pixel aggregation algorithm. Aggregation units are generated based on the aggregation threshold and spatial distribution characteristics. The attribute data of each unit is converted into a set of pixels using a coordinate system transformation algorithm. The X-axis of the pixel corresponds to the Mercator projection longitude coordinate, and the Y-axis corresponds to the latitude coordinate. S24: Perform tile conversion processing. According to the Web Mercator projection specification, the vector data is spatially divided into triplet blocks. Each tile unit has an embedded quadtree index structure. The converted pixel set is generated into a standardized binary format by the spatial encoding engine and persistently stored in the HBase distributed database according to the xyz structure.

[0009] As a preferred embodiment of the present invention, the coordinate system transformation algorithm calculation formula in step S23 is as follows: Pixelx = |(X−Xmin) / resolution| Pixely = |(Ymax−Y) / resolution| Wherein, Pixelx is the horizontal axis position of a single spatial feature in the business vector data within the tile; Pixely is the vertical axis position of a single spatial feature in the business vector data within the tile; X is the data longitude; Y is the data latitude; Xmin is the longitude of the top left corner of the tile; and Ymax is the latitude of the top left corner of the tile.

[0010] As a preferred embodiment of the present invention, step S3 includes: S31: After the business vector data sharding and density clustering thinning process is completed, the HBase Connector interface is called through the distributed storage engine to import the optimized data entities into the HBase columnar database according to the preset row key rules. The row key design integrates spatial partitioning encoding and timestamp information, and multiple version column families are configured to support spatiotemporal data backtracking. S32: When the business vector data is of type point data, GeoHash is used to dynamically shard and aggregate the data volume using GeoHash. The GeoHash encoding bit length is automatically selected according to the data density of the data area where the business vector data is located. The original point cloud data is divided into N spatial subsets, each subset corresponding to an independent geographic grid unit. Then, according to the XYZ rules of the map tiles, HBase RowKey storage is performed.

[0011] As a preferred embodiment of the present invention, step S4 includes: S41: Receive and parse the geographic bounding box parameters in the map loading request, the geographic bounding box parameters including the latitude and longitude coordinates of the southwest corner point and the coordinates of the northeast corner point; S42: Based on the Web Mercator projection specification, a spatial coordinate transformation engine is built. Through the tile pyramid hierarchical calculation model, the geographic range of the geographic bounding box parameter is dynamically mapped to the set of tile coordinates under the target level, generating a tile index matrix containing the column number interval {x_start, x_end} and the row number interval {y_start, y_end}. S43: The tile coordinates are recombined into binary according to a preset bit width using a triplet encoding rule to generate a composite row key structure. The query engine converts the tile index matrix into a batch parallel Scan operation based on the row key prefix matching mechanism. S44: Use HBase API to retrieve data based on RowKey, build a business vector data processing pipeline based on GeoTools, parse the binary stream returned by HBase ResultScanner through FeatureIterator, and use SimpleFeatureTypeBuilder to build a vector feature model containing WKT geometry fields; S45: Design a data converter that incorporates the OGC WMS specification protocol, and achieve dynamic data conversion through a query engine. For vector data scenarios, configure a Protobuf binary serialization module to realize WMS image conversion and vector tile PBF format conversion.

[0012] As a preferred embodiment of the present invention, the expression for the composite row key structure described in step S43 is: RowKey = z<<40 | x<<20 | y, where z is the tile level, x is the column number, and y is the row number. This achieves linear ordered storage of business vector data in HBase through bitwise operations.

[0013] Compared with the prior art, the advantages of the present invention are as follows: Vector data from geographic data was loaded into HBase and retrieval tests were conducted. The retrieval response time can reach the millisecond level, which is suitable for storing billions of geographic business vector data and rendering vector data in real time. It can also output data in different geospatial vector formats according to different scenarios, which greatly expands the processing, storage and real-time rendering capabilities of GIS big data. Attached Figure Description

[0014] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts. In the drawings: Figure 1 The flowchart is a GIS big data processing, storage, and real-time rendering method based on HBase as described in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of big data splitting in an HBase-based GIS big data processing, storage, and real-time rendering method as described in Embodiment 1 of the present invention. Figure 3 This is a schematic diagram of attribute aggregation and pyramid tile cutting for a GIS big data processing, storage, and real-time rendering method based on HBase as described in Embodiment 1 of the present invention. Figure 4 This is a schematic diagram of loading the OWS / WMS standard interface for a GIS big data processing, storage, and real-time rendering method based on HBase, as described in Embodiment 1 of the present invention. Detailed Implementation

[0015] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0016] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this invention, the terms "first," "second," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance, or suggesting any such actual relationship or order between these entities or operations. Additionally, the terms "connected," "linked," etc., can refer to a direct connection between components or an indirect connection via other components.

[0017] Example 1 A method for GIS big data processing, storage, and real-time rendering based on HBase, such as... Figure 1 As shown, it includes the following steps: S1: Access the original geospatial data, partition the original geospatial data based on GeoHash, and load it into vector data to obtain business vector data; Specifically, it accesses raw geospatial vector data; the raw geospatial data uses the wkb spatial data service vector data format, and the file uses text data with a common delimiter; The raw geospatial data is vectorized based on the Geotools open-source library; the structured coordinate data is converted into different Geometry types, including point, line, and polygon data based on latitude and longitude, as well as WKB binary data format conversion. Based on the conversion to Geometry features, and combined with data attribute data, the Geometry data is converted into geospatial features with business attributes. For example, the access data uses 1 billion random point data points generated within a specified range, containing an auto-incrementing ID, latitude and longitude coordinates, random index value 1 (range 0-50), random index value 2 (range 50-100), and the name of the province, city, and county. Specifically, the fields are: id,lon,lat,value_0_50,value_50_100,province,city,county.

[0018] S2: Use the pyramid model to split the business vector data into layers, and perform attribute or spatial aggregation on the split vector data; Specifically, following the pyramid model, the business vector data is rasterized at different levels using multi-resolution rasterization. A certain level is designated as the original high-precision data, such as level 11. As the level decreases, rasterization is performed level by level, recursively dividing the region into sub-regions (e.g., dividing the map into four quadrants), forming a tree-like hierarchical structure. The resolution of level L is: ResolutionL = Resolution0 / 2LR. This forms a pyramid-like structure. Based on the input aggregation level and vectorization level parameters, the gridded data is divided into vector data level and aggregation data level. Aggregation adopts a pixelation algorithm and coordinate system transformation: Pixelx=|(X−Xmin) / resolution|, Pixely=|(Ymax−Y) / resolution|; Wherein, Pixelx is the horizontal axis position of a single spatial feature in the business vector data within the tile; Pixely is the vertical axis position of a single spatial feature in the business vector data within the tile; X is the data longitude; Y is the data latitude; Xmin is the longitude of the top left corner of the tile; and Ymax is the latitude of the top left corner of the tile.

[0019] The original geospatial data is hierarchically divided by a multi-level pyramid model to establish a LOD hierarchical structure with spatial topological relationships. Then, a pixel aggregation algorithm is used to aggregate the original data. Aggregation units are generated based on the aggregation threshold and spatial distribution characteristics, and the attribute data of each unit is converted into a set of pixels, where the X-axis corresponds to the Mercator projection longitude coordinates and the Y-axis corresponds to the latitude coordinates. Furthermore, such as Figure 2 and Figure 3 As shown, a tiled processing module is configured. This module spatially divides vector data into (zoom level, tile column, tile row) triples according to the Web Mercator projection specification. Each tile unit embeds a quadtree index structure to achieve multi-resolution data matching. The preprocessed aggregated data packet generates a standardized binary format through a spatial encoding engine and is persistently stored in the HBase distributed database according to the xyz structure. This storage architecture effectively improves the logicality of the data organization structure and retrieval efficiency, while supporting dynamic loading on multiple terminals and LOD progressive rendering, significantly optimizing network transmission efficiency and rendering performance.

[0020] S3: Store the aggregated vector data in HBase; Specifically, after the business vector data sharding and density clustering thinning process is completed, the HBase Connector interface is called through the distributed storage engine to import the optimized data entities into the HBase columnar database according to the preset row key rules. The row key design integrates spatial partitioning encoding and timestamp information, and multiple version column families are configured to support spatiotemporal data backtracking. For point data aggregation, GeoHash is used for dynamic GeoHash sharding and parallel aggregation of data volume. The GeoHash encoding bit length (5-8 bits) is automatically selected according to the data density of the data area where the business vector data is located. The original point cloud data is divided into N spatial subsets, each subset corresponding to an independent geographic grid unit. Then, according to the XYZ rules of the map tiles, HBase RowKey storage is performed.

[0021] S4: Based on the input loading range, retrieve the aggregated vector data from the HBase and perform real-time rendering; Specifically, the request resolution and spatial coordinate transformation mechanism: Configure the front-end request adapter to receive and parse the geographic bounding box (BBOX) parameters in the map loading request. The BBOX contains the latitude and longitude coordinates of the southwest corner (minLon, minLat) and the coordinates of the northeast corner (maxLon, maxLat).

[0022] Dynamic tile coordinate mapping algorithm: Based on the Web Mercator projection specification, a spatial coordinate transformation engine is built. Through the tile pyramid hierarchical calculation model, the geographical range of the BBOX is dynamically mapped to a set of tile coordinates under the target level (zoom level), generating a tile index matrix containing the column number interval {x_start, x_end} and the row number interval {y_start, y_end}.

[0023] Distributed row key construction and query optimization: A row key construction engine is designed, which uses triplet encoding rules to reassemble the tile coordinates (x, y, z) into binary data according to a preset bit width, generating a composite row key structure, the expression of which is: RowKey = z<<40 | x<<20 | y, where z is the tile level, x is the column number, and y is the row number. This uses bitwise operations to achieve linearly ordered storage of business vector data in HBase. The query engine, based on the row key prefix matching mechanism, converts the tile index matrix into batch parallel Scan operations, significantly reducing network latency for cross-node queries. Actual test data shows that regional query efficiency is improved by more than 2.3 times.

[0024] We use the HBase API to retrieve data based on RowKey, build a business vector data processing pipeline based on GeoTools, parse the binary stream returned by HBase ResultScanner using FeatureIterator, and build a vector feature model containing WKT geometric fields using SimpleFeatureTypeBuilder.

[0025] Design a data converter that incorporates the OGC WMS specification protocol. A query engine enables dynamic data conversion. For vector data scenarios, a Protobuf binary serialization module is configured to achieve WMS image conversion and vector tile PBF format conversion.

[0026] For example, loading based on the OWS / WMS standard interface, such as... Figure 4 As shown, BBOX is used to convert the data into a list of tiles (zoom level, tile column, tile row). The data of the corresponding tile is rendered according to (zoom level, tile column, tile row) and converted into the requested data format.

[0027] It should also be noted that the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0028] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A method for GIS big data processing, storage, and real-time rendering based on HBase, characterized in that, Includes the following steps: S1: Access the raw geospatial data, partition the raw geospatial data based on GeoHash, and load it into a vector to obtain business vector data; S2: Use the pyramid model to split the business vector data into layers, and perform attribute or spatial aggregation on the split vector data; S3: Store the aggregated vector data in HBase; S4: Based on the input loading range, retrieve the aggregated vector data from the HBase and perform real-time rendering.

2. The method for GIS big data processing, storage, and real-time rendering based on HBase according to claim 1, characterized in that, Step S1 includes: S11: Access the original geographic service vector data, which uses the wkb service vector data format and the file uses text data with a common delimiter. S12: The original geographic business vector data received is vectorized based on the Geotools open-source library.

3. The method for GIS big data processing, storage, and real-time rendering based on HBase according to claim 2, characterized in that, Step S12, which describes vectorization based on the Geotools open-source library, includes converting structured coordinate data into different types of Geometry data, including conversions of points, lines, and surfaces based on latitude and longitude, as well as wkb binary data formats. Based on the elements converted into Geometry data, and combined with data attribute data, the Geometry data is transformed into business vector data with business attributes.

4. The method for GIS big data processing, storage, and real-time rendering based on HBase according to claim 1, characterized in that, Step S2 includes: S21: The business vector data is rasterized at different levels according to the multi-level pyramid model, and a certain level is set as the original high-precision data. S22: Based on the input aggregation level and vectorization level parameters, the rasterized data is divided into vector data level and aggregation data level; S23: The business vector data is hierarchically divided by a multi-level pyramid model to establish a LOD hierarchical structure with spatial topological relationships. The original geographic business vector data is aggregated by a pixel aggregation algorithm. Aggregation units are generated based on the aggregation threshold and spatial distribution characteristics. The attribute data of each unit is converted into a set of pixels using a coordinate system transformation algorithm. The X-axis of the pixel corresponds to the Mercator projection longitude coordinate, and the Y-axis corresponds to the latitude coordinate. S24: Perform tile conversion processing. According to the Web Mercator projection specification, the business vector data is spatially divided into triplet blocks. Each tile unit has an embedded quadtree index structure. The converted pixel set is generated into a standardized binary format by the spatial encoding engine and persistently stored in the HBase distributed database according to the xyz structure.

5. The GIS big data processing, storage, and real-time rendering method based on HBase according to claim 4, characterized in that, The coordinate system transformation algorithm calculation formula described in step S23 is as follows: Pixelx = |(X−Xmin) / resolution| Pixely = |(Ymax−Y) / resolution| Wherein, Pixelx is the horizontal axis position of a single spatial feature in the business vector data within the tile; Pixely is the vertical axis position of a single spatial feature in the business vector data within the tile; X is the data longitude; Y is the data latitude; Xmin is the longitude of the top left corner of the tile; and Ymax is the latitude of the top left corner of the tile.

6. The method for GIS big data processing, storage, and real-time rendering based on HBase according to claim 1, characterized in that, Step S3 includes: S31: After the business vector data sharding and density aggregation thinning process is completed, the HBase Connector interface is called through the distributed storage engine to import the business vector data that has been preprocessed by sharding and pixel point set into the HBase columnar database according to the preset row key rules. The row key design integrates spatial partitioning encoding and timestamp information, and multiple version column families are configured to support spatiotemporal data backtracking. S32: When the business vector data is of type point data, GeoHash is used to dynamically shard and aggregate the data volume using GeoHash. The GeoHash encoding bit length is automatically selected according to the data density of the data area where the business vector data is located. The original point cloud data is divided into N spatial subsets, each subset corresponding to an independent geographic grid unit. Then, according to the XYZ rules of the map tiles, HBase RowKey storage is performed.

7. The method for GIS big data processing, storage, and real-time rendering based on HBase according to claim 1, characterized in that, Step S4 includes: S41: Receive and parse the geographic bounding box parameters in the map loading request, the geographic bounding box parameters including the latitude and longitude coordinates of the southwest corner point and the coordinates of the northeast corner point; S42: Based on the Web Mercator projection specification, a spatial coordinate transformation engine is built. Through the tile pyramid hierarchical calculation model, the geographic range of the geographic bounding box parameter is dynamically mapped to the set of tile coordinates under the target level, generating a tile index matrix containing the column number interval {x_start, x_end} and the row number interval {y_start, y_end}. S43: The tile coordinates are recombined into binary according to a preset bit width using a triplet encoding rule to generate a composite row key structure. The query engine converts the tile index matrix into a batch parallel Scan operation based on the row key prefix matching mechanism. S44: Use HBase API to retrieve data based on RowKey, build a business vector data processing pipeline based on GeoTools, parse the binary stream returned by HBase ResultScanner through FeatureIterator, and use SimpleFeatureTypeBuilder to build a vector feature model containing WKT geometry fields; S45: Design a data converter that incorporates the OGC WMS specification protocol, and achieve dynamic data conversion through a query engine. For vector data scenarios, configure a Protobuf binary serialization module to realize WMS image conversion and vector tile PBF format conversion.

8. The method for GIS big data processing, storage, and real-time rendering based on HBase according to claim 7, characterized in that, The expression for the composite row key structure described in step S43 is: RowKey = z << 40 | x << 20 | y, where z is the tile level, x is the column number, and y is the row number. This achieves linear and ordered storage of business vector data in HBase through bitwise operations.

Images