Black soil cultivated land quality space vector big data distributed storage retrieval method

Abstract

The invention discloses a distributed storage and retrieval method for black soil cultivated land quality space vector big data, and relates to the technical field of data storage and retrieval. The method comprises the steps that firstly, according to space information and attribute information of black soil cultivated land quality space vector data, partition coding design is conducted on row keys, column families and Region partitions of a bottom storage system of a distributed non-relational database HBase, and a storage model used for storing the space vector data is obtained; then obtaining a Geohash coding grid of the space vector data in the storage model through a self-adaptive hierarchical division coding method; performing dimension raising conversion on the Geohash coding grid into a Geohash space grid, and constructing a distributed hierarchical spatial index model according to the Geohash space grid and an index unit of the R * tree; and finally, executing the spatial query request in the distributed hierarchical spatial index model through a spatial parallel retrieval method to obtain a spatial retrieval result. According to the method, the mass space vector data of the large-scale black soil cultivated land can be efficiently stored and retrieved.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage and retrieval, and in particular to a distributed storage and retrieval method for black soil arable land quality spatial vector big data. Background Art

[0002] The black soil region in Northeast my country is the "giant panda" among cultivated land. With the in-depth development of information technology and spatial data acquisition technology, the scale of spatial vector data on the quality of black soil cultivated land is increasing. At the same time, the requirements for the accuracy, efficiency, complexity and real-time response of the actual application of massive vector data are also increasing. However, as a traditional means of processing spatial vector data, Geographic Information System (GIS) technology is limited by its data storage mode, index structure and spatial computing capabilities, and can no longer meet the efficiency requirements of high-performance storage and retrieval of large-scale spatial vector data.

[0003] In recent years, the rise of big data and distributed technology has provided a new generation of distributed storage and parallel retrieval methods with high performance for spatial vector big data processing. In this context, the distributed non-relational (Not Only SQL, NoSql) database HBase has become one of the mainstream ways to store and manage massive spatial vector data. However, from the perspective of spatial database, although HBase can directly store spatial vector data, this storage method simply converts spatial vector data into encoding formats such as WKT, WKB and GeoJSON according to object-oriented thinking and then stores it in HBase in the form of byte arrays, without considering the spatial characteristics of the data, which is not conducive to efficient spatial indexing and retrieval operations.

[0004] Therefore, there is an urgent need in the relevant technology for a method that can efficiently perform spatial storage and retrieval of large-scale spatial vector data. Summary of the invention

[0005] Based on this, it is necessary to provide a distributed storage and retrieval method for black soil arable land quality spatial vector big data in response to the above technical problems. This method can perform high-performance storage and retrieval of large-scale spatial vector data.

[0006] The present invention adopts the following technical solutions:

[0007] The present invention provides a black soil arable land quality space vector big data distributed storage and retrieval method, comprising:

[0008] According to the spatial information and attribute information of the black soil arable land quality spatial vector data, the row key, column family and Region partition of the underlying storage system of the distributed non-relational database HBase are designed for partition coding, and the storage model for storing spatial vector data is obtained.

[0009] The Geohash coding grid of the spatial vector data in the storage model is obtained through an adaptive hierarchical division coding method;

[0010] The Geohash coding grid is converted into a Geohash space grid by upgrading its dimension, and a distributed hierarchical spatial index model is constructed based on the Geohash space grid and the index unit of the R* tree.

[0011] The spatial query request is executed in the distributed hierarchical spatial index model through the spatial parallel retrieval method to obtain the spatial retrieval result.

[0012] Preferably, the row key encoding design specifically includes:

[0013] RowKey=GeoHash(Geom)+FeatureID

[0014] Among them, RowKey represents the row key encoding of spatial vector data, Geom represents the spatial information of spatial vector data, GeoHash (Geom) is the Geohash spatial encoding of spatial vector data; FeatureID is the feature ID of the spatial vector data, which uniquely identifies the object in terms of attributes.

[0015] Preferably, the encoding design of the column family specifically includes:

[0016] Design spatial geometry column families and non-geometry attribute column families;

[0017] The spatial information of spatial vector data is stored in the spatial geometry column family, and different types of spatial vector data are uniformly stored under the geom column of the spatial geometry column family;

[0018] The attribute information of the spatial vector data is stored in the non-geometric attribute column family, wherein the field name of the attribute information is defined as the column name under the non-geometric attribute column family, and the attribute value is used as the column value.

[0019] Preferably, the coding design of Region partitions specifically includes:

[0020] Based on the adaptive hierarchical division coding method, the spatial vector data is adaptively divided into hierarchical spaces, and the complex spatial objects are converted into Geohash codes.

[0021] Based on actual application requirements and the natural distribution characteristics of data, perform prefix matching analysis on Geohash codes, customize a reasonable spatial partition range, and calculate the string value of each spatial partition;

[0022] The string value of each spatial partition is used as the delimiter value of the storage model Region pre-partition, that is, the row key range of each Region partition;

[0023] According to the row key of the spatial vector data and the range of the Region partition row key, the spatial vector data is stored in the corresponding Region.

[0024] Preferably, the adaptive hierarchical division encoding method specifically includes:

[0025] Obtain the minimum bounding rectangle of the spatial object to be encoded, and determine the Geohash encoding level of the spatial object to be encoded according to the minimum bounding rectangle;

[0026] According to the Geohash coding level, determine the coding grid of the spatial object to be coded;

[0027] According to the Geohash coding level and the coordinates of the center point of the minimum circumscribed rectangle, the coding grid is binary-coded to obtain the Geohash binary coding string of the spatial object to be coded;

[0028] Based on the Base32 string conversion rule, the Geohash binary code string is converted into a string, and then the string is combined into a Geohash code set to obtain the unique Geohash string code corresponding to the spatial object to be encoded;

[0029] Among them, the Geohash code will be placed in a high position as a component of the row key in the storage model design.

[0030] Preferably, a distributed hierarchical spatial index model is constructed based on the Geohash spatial grid and the index unit of the R* tree, specifically including:

[0031] According to the index units of the spatial grid and the R* tree, the GR*-Tree hybrid spatial index structure is determined;

[0032] According to the GR*-Tree hybrid spatial index structure, a distributed hierarchical spatial index model is constructed.

[0033] Preferably, the distributed hierarchical spatial index model includes a GR*-Tree global spatial index based on the entire table; and according to the GR*-Tree hybrid spatial index structure, a distributed hierarchical spatial index model is constructed, specifically including:

[0034] Using the R* tree in the GR*-Tree hybrid spatial index structure as the basic index structure and the Region related information as the basic index unit, the key-value pair set of the Geohash code and the minimum enclosing rectangle of all Regions in the storage model is obtained;

[0035] Traverse the key-value pair set, take out the key-value pairs in the key-value pair set, and insert them into the R* tree as spatial objects according to the Region spatial rectangular range corresponding to the minimum enclosing rectangle; and based on the R* tree "heuristic" insertion strategy, select the leaf node to import the spatial object;

[0036] Adjust the global spatial index R* tree node structure, determine the root node, and build the GR*-Tree global spatial index structure based on the entire table.

[0037] Preferably, the distributed hierarchical spatial index model includes a GR*-Tree local spatial index structure based on Region; and according to the GR*-Tree hybrid spatial index structure, a distributed hierarchical spatial index model is constructed, specifically including:

[0038] The Geohash spatial grid in the GR*-Tree hybrid spatial index structure is used as the basic unit of the GR*-Tree local spatial index structure of the Region. When the storage model stores spatial vector data, the spatial vector data is stored in the corresponding Region partitions according to the Region pre-partition boundary value, and the Geohash row key code and code grid stored in the spatial vector data uniquely represent the object.

[0039] Automatically insert into the R* tree in the form of <row key encoding, encoding grid> key-value pairs according to the "heuristic" strategy;

[0040] After inserting the spatial vector data, the minimum bounding rectangle of the relevant nodes is updated, and the relevant nodes of the index tree are adjusted according to the R*-Tree index node adjustment strategy;

[0041] When all spatial vector data are inserted into the R*-Tree spatial index structure, the Region-based GR*-Tree local spatial index structure is constructed.

[0042] Preferably, the method further comprises:

[0043] After the distributed hierarchical spatial index model is built, a spatial query request is received;

[0044] According to the spatial query request, the global spatial index or the local spatial index is called from the distributed hierarchical spatial index model to obtain index information; the index information includes the index row key, column family and column;

[0045] According to the index information, load the GR*-Tree serialized byte array into the memory, then deserialize it into a GR*-Tree index instance to complete the construction of the GR*-Tree.

[0046] Preferably, executing a spatial query request in a distributed hierarchical spatial index model by a spatial parallel retrieval method to obtain a spatial retrieval result specifically includes:

[0047] Preprocessing the spatial query request; the spatial query request includes query conditions;

[0048] Based on the preprocessed spatial query request, the GR* tree global spatial index in the distributed hierarchical spatial index model is constructed and executed in memory to filter out the Region set that intersects with the query condition;

[0049] The spatial query request is distributed to all regions in the Region set by remote procedure call protocol batch broadcast. The Region partition receives the spatial query request and query condition parameters.

[0050] The custom query service deployed on the Region loads and executes the GR* tree local spatial index inside each Region, retrieves and filters the spatial objects in multiple Regions that meet the requirements in parallel, and filters the index retrieval set in combination with other query conditions to obtain spatial retrieval results.

[0051] The present invention provides a black soil arable land quality space vector big data distributed storage and retrieval device, comprising:

[0052] A design module is used to design partition coding for row keys, column families and Region partitions of the underlying storage system of the distributed non-relational database HBase according to the spatial information and attribute information of the black soil arable land quality spatial vector data, so as to obtain a storage model for storing spatial vector data;

[0053] A construction module is used to obtain the Geohash coding grid of the spatial vector data in the storage model through an adaptive hierarchical division coding method; the Geohash coding grid is converted into a Geohash spatial grid by upgrading the dimension, and a distributed hierarchical spatial index model is constructed according to the Geohash spatial grid and the index unit of the R* tree;

[0054] The query module is used to execute spatial query requests in a distributed hierarchical spatial index model through a spatial parallel retrieval method to obtain spatial retrieval results.

[0055] The present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned black soil arable land quality spatial vector big data distributed storage and retrieval method is implemented.

[0056] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the above-mentioned distributed storage and retrieval method for black soil arable land quality spatial vector big data is implemented.

[0057] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0058] The present invention proposes an adaptive distributed storage and retrieval method for complex spatial vector data based on the distributed non-relational database HBase. The method first designs the partition coding of the row key, column family and Region partition of the underlying storage system of the distributed non-relational database HBase according to the spatial information and attribute information of the black soil arable land quality spatial vector data, and obtains a storage model for storing spatial vector data. This HBase spatial vector big data storage model obtained based on the characteristics of spatial vector data and combined with the distributed storage principle of the non-relational database HBase enables the non-relational database HBase to directly support spatial vector big data storage and indexing, solving the problems of difficult storage and low retrieval efficiency of traditional planar data.

[0059] In addition, in order to further improve the spatial retrieval efficiency, the Geohash coding grid of the spatial vector data in the storage model is obtained through an adaptive hierarchical division coding method; the Geohash coding grid is converted into a Geohash spatial grid by upgrading the dimension, and combined with the index unit of the R* tree to construct a distributed hierarchical spatial index model. In this way, the spatial retrieval efficiency can be further accelerated in a parallel manner according to the spatial parallel retrieval method. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0061] Figure 1 A schematic diagram of a distributed storage and retrieval method for black soil arable land quality spatial vector big data provided by the present invention;

[0062] Figure 2 A structural schematic diagram of a Region partition and Geohash row key encoding and the spatial relationship represented by the present invention;

[0063] Figure 3 A schematic diagram of a flow chart of Geohash space partition encoding provided by the present invention;

[0064] Figure 4 A schematic diagram of the structure of a single grid coding provided by the present invention;

[0065] Figure 5 A schematic diagram of the structure of a dual Geohash grid code provided by the present invention;

[0066] Figure 6 A schematic diagram of the structure of a multi-Geohash grid coding provided by the present invention;

[0067] Figure 7 A schematic diagram of the structure of a GR*-Tree hybrid spatial index provided by the present invention;

[0068] Figure 8 A schematic diagram of a flow chart of a spatial index loading and construction method provided by the present invention;

[0069] Fig. 9 A schematic diagram of the overall process of spatial query provided by the present invention;

[0070] Fig.10 A technical architecture diagram of a spatial vector big data distributed storage and retrieval method based on HBase provided by the present invention;

[0071] Fig.11 A schematic diagram of the visualization results of the performance comparison with Geomesa under different window sizes provided by the present invention;

[0072] Fig.12 A schematic diagram of the visualization results of the performance comparison with Geomesa under different data amounts provided by the present invention;

[0073] Fig.13 A schematic diagram of a black soil farmland quality space vector big data distributed storage and retrieval device provided by the present invention;

[0074] Fig.14 A schematic diagram of a computer device for implementing a distributed storage and retrieval method for black soil arable land quality spatial vector big data provided by the present invention. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0076] At present, relevant researchers have carried out many studies on storage and management solutions for spatial vector big data around distributed NoSql databases represented by HBase, and have achieved some remarkable research results, such as GeoMesa, MD-HBase, Ganos, VA-HBase, etc.

[0077] In addition to the storage model, spatial index is also the key to the retrieval performance of spatial databases. In existing research, spatial index mechanisms can be roughly divided into two types: Sequence File Control (SFC) index and tree index. SFC index adopts the indexing idea of ​​spatial partition coding, statically divides the geographic space area into grids based on curves such as "Z-Curve" and "Hilbert", and maps multidimensional spatial objects to one-dimensional space through dimensionality reduction coding to achieve linear spatial indexing, which helps to maintain the proximity relationship between multidimensional spatial objects, thereby providing better spatial query performance. As a typical application of SFC index, GeoMesa cleverly reduces, encodes, and maps multidimensional spatial objects into one-dimensional data form through the Z curve algorithm through the geohash coding method, and uses it as the key in the key-value pair of NoSql database. This indexing method not only simplifies the storage structure of spatial data, but also significantly improves the query efficiency of spatial databases. However, since the core organizational principle of SFC is based on the mapping of spatial regions rather than directly targeting the spatial data itself, online data indexing can show relatively efficient performance.

[0078] However, when faced with line and surface data with complex structures and irregular spatial distribution, cross-region object allocation problems may occur during the SFC mapping process, affecting the accuracy and efficiency of the index. Tree-based indexes are represented by the R-tree family index, whose indexing concept is to divide spatial data into a series of spatial units, namely the minimum bounding rectangle (MBR) of spatial objects, and organize and express spatial data through a tree structure. The R* tree has higher indexing performance in the R-tree family. It uses MBR as the basic index unit and corresponds to the nodes in the tree structure. By utilizing the hierarchical characteristics of the tree, it can quickly locate the minimum area containing the target spatial object, thereby reducing unnecessary disk I / O operations and improving indexing efficiency.

[0079] However, when facing the index of multi-dimensional complex spatial objects, the MBRs of the objects may overlap, resulting in waste of spatial areas and low space utilization. In the related technology, a ZPR tree structure is proposed, which uses the prefix characteristics of Z curve encoding to effectively solve the MBR overlap problem of multi-dimensional point data in the indexing process; a distributed R* tree index on the distributed NoSql database HBase is also proposed. Based on the HBase data partition storage principle, a distributed R* tree is constructed by optimizing the STR tree balancing strategy, which provides high-performance spatial index support for the distributed storage of large-scale spatial vector data.

[0080] Based on this, in order to solve the current problems of complex spatial vector big data storage mode, single spatial index structure, insufficient high concurrency support, and improve the spatial query performance of the big data platform, this application proposes an adaptive distributed storage and retrieval method for complex vector data based on the distributed NoSql database HBase. This method first designs the HBase vector big data storage model according to the characteristics of vector data. On the basis of the storage model, by improving the problem that the traditional SFC-based Geohash does not support complex line and surface data, the Geohash method is combined with the R* tree to construct an adaptive distributed hybrid spatial index structure GR*-Tree, and a hierarchical index architecture is designed according to the global spatial index and the local spatial index. Finally, a coprocessor-based spatial parallel retrieval method is designed to further accelerate the efficiency of spatial queries in a parallel manner. Furthermore, this application also takes the black soil arable land quality big data as an example to provide a new method for high-performance storage and retrieval of spatial vector data in various fields.

[0081] The technical solutions provided by various embodiments of the present invention are described in detail below in conjunction with the accompanying drawings.

[0082] Figure 1 This is a schematic diagram of a distributed storage and retrieval method for black soil arable land quality spatial vector big data in the present invention, which specifically includes the following steps:

[0083] S101: Based on the spatial information and attribute information of the black soil arable land quality spatial vector data, the row key, column family and Region partition of the underlying storage system of the distributed non-relational database HBase are partitioned and coded to obtain a storage model for storing spatial vector data.

[0084] Among them, the underlying storage system is the Hadoop big data platform distributed file system. On the distributed non-relational database HBase, this application uses the Hadoop big data platform distributed file system as the underlying storage system. According to the characteristics of the black soil arable land quality spatial vector data, combined with the HBase distributed storage principle, the row key, column family and Region partition of the underlying storage system are designed for spatial information and attribute information, so that HBase can directly support the storage and indexing of spatial vector big data, paving the way for building an efficient distributed hierarchical spatial index model.

[0085] In one embodiment, for HBase, the row key is the key to its storage and retrieval performance. As the unique identifier of the data row in the HBase table, the row key can realize fast access to the specified data row. However, HBase row keys are encoded in lexicographical ascending order by default, without considering the spatial characteristics of vector data, which is not conducive to efficient spatial retrieval. In addition, the row key encoding design also needs to follow basic principles such as uniqueness, orderliness and hashability. In order to meet the needs of efficient retrieval of massive vector data in HBase, this application designs the row key of the storage model based on the adaptive hierarchical Geohash space partitioning encoding method.

[0086] Preferably, the row key encoding design specifically includes:

[0087] RowKey=GeoHash(Geom)+FeatureID

[0088] Among them, RowKey represents the row key encoding of spatial vector data, Geom represents the spatial information of spatial vector data, GeoHash (Geom) is the Geohash spatial encoding of spatial vector data; FeatureID is the feature ID of the spatial vector data, which uniquely identifies the object in terms of attributes.

[0089] The adaptive hierarchical Geohash spatial partitioning encoding method can effectively solve the problem of low efficiency and poor accuracy in encoding complex vector data by traditional SFC-type Geohash methods. The specific encoding method is shown below. This application performs spatial partitioning and encoding of spatial vector data based on Geohash, and uses the encoding as the leftmost bit information of the row key, and then implements Region pre-partitioning based on the row key spatial encoding information, so that newly inserted spatial data is stored in different Region partitions according to its Geohash encoding, achieving high efficiency of the storage model while satisfying load balancing.

[0090] In one embodiment, the encoding design of the column family specifically includes the following steps:

[0091] S201: Design spatial geometry column families and non-geometry attribute column families;

[0092] S202: storing spatial information of spatial vector data in a spatial geometry column family, and uniformly storing different types of spatial vector data under a geom column of the spatial geometry column family;

[0093] S203: Store the attribute information of the spatial vector data in a non-geometric attribute column family, wherein the field name of the attribute information is correspondingly defined as the column name under the non-geometric attribute column family, and the attribute value is used as the column value.

[0094] Specifically, considering the characteristics of vector data containing both spatial information and attribute information, the storage model will design the spatial geometry column family Geometry and the non-geometric attribute column family Attribute respectively.

[0095] Among them, the spatial geometry column family mainly stores the spatial information of spatial vector data, and stores different types of vector data in a unified manner under the geom column of the column family. The storage format is (Geometry:geom), and the spatial information is used as the column value of geom; the non-geometric attribute column family is used to store the attribute information of spatial vector data. The field name of the attribute information is directly defined as the column name under the column family, and the attribute value is used as the column value.

[0096] Optionally, the column family design of the black soil arable land quality data storage model and some sample data are shown in Table 1. Table 1 is a black soil arable land quality spatial vector data table based on HBase.

[0097] Table 1

[0098]

[0099] In one embodiment, data partition storage is the basis for ensuring the load balancing of the storage model, building distributed indexes, executing parallel tasks, and efficient spatial retrieval. Therefore, the present application proposes a Region pre-partitioning method based on Geohash row key coding to design the region partition coding. The coding design of Region partitioning specifically includes: first, adaptive hierarchical spatial division of spatial vector data based on the adaptive hierarchical division coding method, and converting complex spatial objects into Geohash coding; then, based on the actual application requirements and the natural distribution characteristics of the data, the Geohash coding is prefix matched and analyzed to customize a reasonable spatial partition range, and the string value of each spatial partition is calculated, and the string value of each spatial partition is used as the delimiter value of the storage model Region pre-partition, that is, the row key range of each Region partition. Finally, in the data import stage, according to the row key of the spatial vector data and the row key range of the Region partition, the spatial vector data is stored in the corresponding Region.

[0100] In this way, each spatial vector data is accurately stored in the corresponding Region, achieving balanced distribution of data in space and storage. This strategy not only solves the data hotspot problem that HBase may encounter when processing spatial vector big data, but also achieves load-balanced storage of massive data, thereby improving the storage and retrieval efficiency of the HBase spatial vector big data storage model, laying the foundation for subsequent spatial indexing and query. Therefore, we can use the characteristic that the same prefix of the Geohash row key encoding string represents the same spatial range to carry out Region spatial partitioning. The relationship between Region partitioning and Geohash row key encoding and the space it represents is as follows: Figure 2 shown.

[0101] S102: Obtain the Geohash coding grid of the spatial vector data in the storage model through an adaptive hierarchical division coding method, convert the Geohash coding grid into a Geohash spatial grid, and then construct a distributed hierarchical spatial index model based on the Geohash spatial grid and the index unit of the R* tree.

[0102] Among them, the adaptive hierarchical division coding method is an adaptive hierarchical space division coding method obtained by improving Geohash.

[0103] The Geohash method is a space division coding method based on the Z curve, which can map two-dimensional space objects to one-dimensional space through dimensionality reduction coding. Before Geohash encodes spatial objects, it is necessary to divide the space grid in the global geographical scope, and take the entire earth as the original geographical scope of the 0th layer. Then, when dividing the first layer, it is first divided into two parts along the longitude direction, and the left side of the longitude bisector is encoded as 0, and the right side is encoded as 1; then it is divided into two parts along the dimension direction, and the part above the dimension bisector is encoded as 1, and the part below is encoded as 0. Similarly, the space is divided alternately in the longitude and longitude directions, and each division determines a level. The unit grid code in this level only contains "0" and "1", forming the initial binary code of the unit grid. The next higher level recursively divides the longitude and longitude into two parts within the spatial scope after the division of the previous level until the specified coding longitude of the spatial object is met. The flowchart of Geohash space division coding is as follows: Figure 3 shown.

[0104] From the encoding characteristics, we can see that Geohash is more suitable for encoding spatial point data. In theory, as long as the Geohash space division level is high enough, the spatial range represented by its precision is infinitely close to the spatial point object. However, due to the geometric irregularity of spatial line and surface objects themselves, the traditional Geohash method cannot encode quickly and accurately.

[0105] In view of the shortcomings of the traditional Geohash method in encoding spatial line and surface objects, this application proposes an adaptive hierarchical division encoding method, that is, an adaptive hierarchical spatial division encoding method that improves Geohash. The implementation of Geohash adaptive multi-level encoding ensures that each spatial object dynamically determines one or more Geohash coding grids that can completely cover the spatial object according to the size of its own spatial range. This adaptive hierarchical division encoding method enables Geohash encoding to more accurately express the spatial range of the object, while avoiding excessive differences between the encoding area and the object range area, resulting in a large number of blank areas in the encoding area.

[0106] There are generally two methods for adaptive spatial division encoding of spatial line and surface objects. The first is the single grid encoding method, that is, only one Geohash code is used to represent the spatial object, and the encoding grid can just cover the spatial object; the second is the multi-grid encoding method, which uses multiple Geohash codes to represent the spatial object, with a higher division level and a smaller encoding grid, and the boundary of the spatial object is contained in multiple non-intersecting Geohash code grids that can cover the object. Among them, the single grid encoding method is suitable for spatial objects that can just fill most of the area of ​​a certain level of grid without generating a large redundant space. For spatial objects whose spatial range is slightly larger than the size of the grid of the next level and much smaller than the size of the grid of this level, a large amount of overlapping area waste and low precision of spatial encoding will occur in actual storage and query processing. The multi-grid encoding method uses multiple higher-level grids to represent a spatial object. Due to the high grid level and small grid, a spatial object may be represented by too many grids, resulting in too many Geohash codes corresponding to the spatial object, increasing the space demand for storage encoding, and is not conducive to fast spatial retrieval. Therefore, this application will adopt single-grid and multi-grid adaptive coding methods. If the object to be coded can be contained by a grid at a certain level of Geohash and no large irrelevant blank area will be generated, the single-grid coding method at that level is applicable. On the contrary, if the spatial range of the object to be coded involves multiple grids at the next level but is much smaller than the size of the grid at the previous level, the multi-grid coding method is more suitable, and the number of coding grids is controlled to ensure that the length of the spatial object Geohash coding meets the HBase data storage requirements and avoids affecting the storage and retrieval performance of HBase itself.

[0107] The adaptive spatial division coding for the above-mentioned spatial line and surface objects mainly includes two methods: single grid spatial coding and multi-grid spatial coding. The following will improve and optimize the Geohash algorithm to realize the adaptive spatial coding method. The adaptive hierarchical division coding method mainly includes the following steps:

[0108] S301: Obtain a minimum bounding rectangle of a spatial object to be encoded, and determine a Geohash coding level of the spatial object to be encoded according to the minimum bounding rectangle.

[0109] First, the spatial rectangular range corresponding to the minimum bounding rectangle (MBR) of the spatial object to be encoded is obtained, and then the coding level of the spatial object in the spatial grid division of the Geohash global geographic range is adaptively determined according to the MBR. Among them, the basic information of the spatial object MBR is shown in Table 2 below:

[0110] Table 2

[0111]

[0112] In Table 2, the center point mbr_center of the spatial object MBR and the related longitude and latitude information will be used to adaptively determine the optimal Geohash space partition coding level, and also provide spatial coordinate information support for Geohash coding.

[0113] The specific method of determining the Geohash coding level of the spatial object to be encoded based on the minimum circumscribed rectangle includes: Adaptive Geohash spatial coding needs to determine the most suitable spatial division coding level for irregularly shaped line and surface objects. In the spatial division, Geohash space division is performed according to the MBR of the spatial object to be encoded and its related longitude and latitude coordinate information until one or more grids in a certain level can completely cover the object MBR, while avoiding too many irrelevant blank areas in the grid, causing spatial coverage redundancy and affecting spatial indexing and retrieval performance. Therefore, the MBR of the spatial object to be encoded should be slightly smaller than the size of the coding level grid.

[0114] Geohash divides the spatial range along longitude and latitude to generate multiple spatial grids of the same size that completely cover the spatial range. The size of each grid is determined by the spatial division level. For example, the grid at level 0 represents the surface range of the earth, and its size is (360°*180°). Due to the difference in the size of the longitude and longitude of the earth, the longitude span of the coding grid is twice the latitude span. First, the longitude and latitude span values ​​Δlon and Δlat of the MBR of the spatial object to be coded in the spatial range are calculated respectively. The calculation method is as shown in formulas (1) and (2):

[0115] Δlon=r_lon-l_lon (1)

[0116] Δlat=t_lat-b_lat (2)

[0117] When the spatial coding grid just contains or is slightly larger than the spatial object MBR, Geohash coding reaches the optimal coding level. Due to the doubling of the spatial range between longitude and latitude, the horizontal longitude span value of the MBR rectangle is compared with twice the vertical latitude opening value Δlat, and the larger value between the two is taken as the condition for the MBR spatial level division to compare the edge length edge. The edge edge representation formula (3) is as follows:

[0118] edge=max{Δlon,2*Δlat} (3)

[0119] From formula (3), we can see that edge is the maximum value of the MBR edge length, which is used to compare with the grid size edge length determined by the spatial coding level to determine the minimum grid and maximum level level that is slightly greater than or equal to the object MBR. The formula is shown in the following formula (4):

[0120]

[0121] The above formula can be used to determine the optimal coding level level for adaptive spatial coding, where level is a positive integer. After derivation, the level value is as shown in the following formula (5):

[0122]

[0123] The best coding level level for adaptive spatial coding is determined as the Geohash coding level of the spatial object to be coded.

[0124] S302: Determine a coding grid of a spatial object to be coded according to the Geohash coding level.

[0125] After the Geohash coding level of the spatial object to be coded is determined in S301, the specific position of the grid representing the spatial object to be coded needs to be determined on the level. After the Geohash coding level is determined, the coding grid should be slightly larger than the spatial object MBR. At this time, the spatial object and the coding grid may have the following corresponding situations:

[0126] ① Single grid coding: A single grid on the coding level can completely cover the object MBR, and the spatial object can be represented by a Geohash code. The details are as follows Figure 4 As shown, Figure 4 Schematic diagram of the structure of single grid encoding.

[0127] ②Dual grid coding: The MBR space range corresponding to the spatial object to be coded is smaller than the grid size of the level, but its spatial distribution spans two horizontal or vertical grids. The spatial object to be coded can be represented by these two Geohash codes. The specific representation is as follows Figure 5 As shown, Figure 5 Schematic diagram of the structure of dual Geohash grid encoding.

[0128] ③ The MBR space range corresponding to the spatial object to be encoded is smaller than the size of the grid, but its spatial distribution spans four grids at the level, and the MBR intersects with the four adjacent grids. In order to avoid the spatial error between the spatial object entity and its MBR rectangle leading to the encoding of redundant and irrelevant spatial grids, it is necessary to determine whether the spatial object entity has an intersection relationship with all four grids, and finally determine the three or four Geohash codes representing the spatial object. The specific representation is as follows Figure 6 As shown, Figure 6 Schematic diagram of the structure of multiple Geohash grid encoding.

[0129] Based on the above definitions of single grid coding, dual grid coding and multi-grid coding, the MBR spatial range corresponding to the spatial object to be coded is compared with the Geohash coding level size to determine the coding grid of the spatial object to be coded.

[0130] S303: Binary encode the coding grid according to the Geohash coding level and the coordinates of the center point of the minimum circumscribed rectangle to obtain a Geohash binary coding string of the spatial object to be encoded.

[0131] After determining the spatial division Geohash coding level and specific coding grid of the spatial object to be coded, Geohash string coding is required. First, according to the above three different spatial division coding situations, one or more Geohash binary coding strings corresponding to the spatial object to be coded are calculated using the Geohash coding level and the MBR center point coordinates mbr_center obtained in S301.

[0132] S304: Convert the Geohash binary code string into a character string based on the Base32 character string conversion rule, and then combine the character string into a Geohash code set to obtain a unique Geohash string code corresponding to the spatial object to be encoded.

[0133] The Geohash code will be placed in a high position as a component of the row key in the storage model design.

[0134] After obtaining the Geohash binary code string of the spatial object to be encoded, these Geohash binary codes are converted into strings based on the GeohashBase32 string conversion rules. The GeohashBase32 string conversion rules are shown in Table 3. Then the strings are combined into a Geohash code set, and then the elements in the set are connected using the connector "_" to form a unique Geohash string code corresponding to the spatial object to be encoded. The Geohash code will be placed in the high position as a component of the row key in the storage model design, and it is also an adaptive Geohash encoding method in Region spatial partitioning.

[0135] Table 3

[0136]

[0137] The method of obtaining the Geohash coding grid of the spatial object in this embodiment is the same as the method of obtaining the coding grid of the spatial object to be encoded in the above embodiment, and this embodiment will not be repeated here.

[0138] In an exemplary embodiment, based on the index unit according to the Geohash spatial grid and the R* tree in the above embodiment, a distributed hierarchical spatial index model is constructed, which specifically includes the following steps:

[0139] S401: Determine a GR*-Tree hybrid spatial index structure according to the spatial grid and the index unit of the R* tree.

[0140] According to the spatial index structure, the GR*-Tree hybrid spatial index structure includes two parts: one is to convert the Geohash coding grid into a spatial grid, and use the spatial grid as the basic index unit of R*; the other is to build a GR* tree index for the entire data set, using the Geohash coding grid as the node in the R* tree, and continuously splitting and reorganizing the R* tree nodes according to the spatial inclusion relationship between the nodes to achieve the optimal R* tree spatial index structure for efficient filtering and retrieval of spatial objects. Compared with ordinary R* trees, the MBRs between GR* tree nodes rarely overlap, which reduces data retrieval errors and reduces the time cost of adjusting the index structure due to object insertion. The GR*-Tree hybrid spatial index structure is as follows: Figure 7 shown.

[0141] S402: Construct a distributed hierarchical spatial index model based on the GR*-Tree hybrid spatial index structure.

[0142] The distributed hierarchical spatial index model includes the GR*-Tree global spatial index based on the entire table and the GR*-Tree local spatial index structure based on Region.

[0143] In an exemplary embodiment, the distributed hierarchical spatial index model includes a GR*-Tree global spatial index based on the entire table, and a distributed hierarchical spatial index model is constructed, specifically including: using the R* tree in the GR*-Tree hybrid spatial index structure as the basic index structure and the Region-related information as the basic index unit, obtaining the Geohash codes of all Regions in the storage model and the key-value pair set of the minimum enclosing rectangle, and traversing the key-value pair set, taking out the key-value pairs in the key-value pair set, and inserting them into the R* tree as spatial objects according to the Region spatial rectangle range corresponding to the minimum enclosing rectangle; and based on the R* tree "heuristic" insertion strategy, selecting a suitable leaf node to import the spatial object; finally, adjusting the global spatial index R* tree node structure, determining the root node, and completing the construction of the GR*-Tree global spatial index structure based on the entire table.

[0144] Specifically, the global spatial index is built based on the GR*-Tree hybrid spatial index structure. In the design of the storage model, the HBase table row key design is implemented based on the optimized Geohash method to encode spatial objects. The Geohash row key encoding prefix of the specified length is used as the Region pre-partition boundary value, and the encoding prefix is ​​denoted as Prefix_Geohash(S). Prefix_Geohash(S) represents the Geohash code corresponding to the Region partition, which can uniquely identify the Region partition and is denoted as RegionId. At the same time, Prefix_Geohash(S) also represents a Geohash spatial grid, which can completely contain all spatial objects in the Region and is denoted as RegionMBR. The global spatial index uses the Region partition as the basic unit of the R* tree index.

[0145] Based on the above understanding, after spatial partitioning, Region has two important attributes: RegionId and RegionMBR, which represent the Region identifier and spatial range respectively. The construction of GR*-Tree global spatial index uses R* tree as the basic index structure and Region related information as the basic index unit. The basic construction process is as follows:

[0146] (1) Get all Regions in the HBase table<RegionId,RegionMBR> A collection of key-value pairs.

[0147] (2) Take out the key-value pairs in the collection and insert them into the R* tree as spatial objects according to the Region spatial rectangular range represented by RegionMBR.

[0148] (3) Based on the R* tree “heuristic” insertion strategy, select the appropriate leaf node to import the spatial object.

[0149] (4) Repeat steps (2) and (3) until the key-value pair set is traversed.

[0150] (5) Finally, adjust the global spatial index R* tree node structure, determine the root node, and complete the construction.

[0151] In the spatial query stage, the global spatial index needs to be executed first. Spatial queries all have spatial range conditions. In the global spatial index, whether it intersects with the spatial query range is used as a screening condition. In the global spatial index GR*-Tree, the spatial query range is compared from top to bottom to see whether there is an intersection relationship with the node MBR. If it intersects, recursively traverse the next layer of child nodes of the node and continue to compare whether it intersects with the child node MBR. And so on, until it recurses to the leaf node level, filter out the set of leaf nodes that intersect with the spatial query range, that is, determine the set of Regions that meet the query conditions. Then, execute the local spatial index in each Region, finally complete the spatial query process, and output the query result set.

[0152] In an exemplary embodiment, if the distributed hierarchical spatial index model includes a Region-based GR*-Tree local spatial index structure, constructing the distributed hierarchical spatial index model specifically includes the following steps:

[0153] S501: Use the Geohash spatial grid in the GR*-Tree hybrid spatial index structure as the basic unit of the GR*-Tree local spatial index structure of the Region. When the storage model stores spatial vector data, the spatial vector data is stored in the corresponding Region partitions according to the Region pre-partition boundary value, and the Geohash row key code and code grid stored in the spatial vector data uniquely represent the object.

[0154] S502: Automatically insert into the R* tree in the form of <row key code, code grid> key-value pairs according to the "heuristic" strategy.

[0155] S503: After inserting the spatial vector data, update the minimum bounding rectangle of the relevant node to ensure that it contains the minimum bounding rectangles of all child nodes, and adjust the relevant nodes of the index tree according to the R*-Tree index node adjustment strategy.

[0156] S504: When all spatial vector data are inserted into the R*-Tree spatial index structure, a Region-based GR*-Tree local spatial index structure is constructed.

[0157] As the lower index structure in the hierarchical index structure of the hybrid spatial index model based on the HBase distributed database, the local spatial index is an index structure that acts on each spatial object in the Region partition and is constructed using the GR*-Tree hybrid spatial index method. In the storage model, the row key is designed as a combination of the Geohash code and the feature number of the spatial object, where the Geohash code represents the spatial grid of the spatial object after the adaptive Geohash space division. Based on the characteristic that this row key contains the rectangle of the Geohash spatial grid of the spatial object, combined with the index construction principle of R*-Tree, the one-dimensional Geohash code corresponding to the spatial object is spatially upgraded to a Geohash two-dimensional spatial grid, and used as the basic unit of the Region local spatial index R* tree index.

[0158] The construction of the Region local spatial index is performed inside the HBase data partition. When HBase stores data, the spatial data is stored in the corresponding Region partition according to the Region pre-partition boundary value. The Geohash row key code (RowKey) and code grid (GeohashGrid(Object)) stored in the spatial object uniquely represent the object, and<RowKey,GeohashGrid(Object)> The key-value pair form is automatically inserted into the R* tree according to the "heuristic" strategy. After inserting the spatial object, update the relevant node MBR to ensure that it contains all child node MBRs. Then adjust the relevant nodes of the index tree according to the R*-Tree index node adjustment strategy (such as splitting, overlapping, redistribution, etc.) to ensure the balance and performance of the R*-Tree index. Repeat this operation until all spatial objects are inserted into the R*-Tree spatial index structure, and finally build a complete Region-based GR*-Tree local spatial index structure, and adjust the index structure in time during subsequent data updates to achieve stable indexing and efficient retrieval, greatly improving the query performance of massive spatial vector data.

[0159] When a spatial query executes a local spatial index of a Region, first, according to the query conditions, the global spatial index mentioned above is used to filter out the Region set involving the query range, and then the local spatial index is used in parallel inside each Region in the Region set, starting from the root node of the GR* tree, from top to bottom, the node MBR is compared with the query range in turn, and the nodes that have an intersection relationship with the query spatial range are filtered out, and so on, until the leaf nodes that meet the conditions are filtered out in the local spatial index structure, and the row key value stored in the leaf node, that is, the node identifier, is added to the index candidate set to complete the local index. After filtering through the candidate set, the spatial object data record set that fully meets the query conditions in the Region is finally obtained and output.

[0160] After the distributed hierarchical spatial index model is constructed, the persistence of the constructed GR* tree spatial index information is also the key to affecting the spatial retrieval performance, including two aspects: spatial index persistent storage and loading and construction.

[0161] The distributed hierarchical spatial index model consists of a global spatial index and a local spatial index, and the spatial index persistence should also be designed separately.

[0162] The global spatial index is based on the HBase table filtering of Region partitions. When storing it persistently, the global GR* tree index structure is first serialized into a byte array and stored in the HBase data table as a global spatial index row. The row key of the index row needs to be different from the record row key of the spatial object in the table to ensure its uniqueness and irreplaceability. The specific storage location of the index is determined according to the row key of the global spatial index row, so that the HBase API method can extract the index information and load it into the memory to build the global spatial index GR* tree during subsequent spatial query processing.

[0163] The local spatial index is used to filter the spatial objects within the Region partition. A local local index is maintained in each Region partition in the table, and the number of indexes is consistent with the number of partitions. Similar to the global spatial index persistence method, the local index GR* tree is serialized into an index byte array and stored as an index row in the HBase SVD table. The difference is that the local spatial index is an index of the spatial objects in the Region partition to which it belongs. To ensure the clustering of the index and data, the storage location of the index row should also be in the corresponding Region partition. Therefore, the row key design of the local spatial index row should ensure that the row key is within the row key range of the corresponding Region partition while ensuring the uniqueness of the row key. The spatial index persistence storage solution is shown in Table 4 below.

[0164] Table 4

[0165]

[0166] As another part of the hierarchical spatial index persistence solution, how to effectively extract the spatial index information distributed and stored in the HBase table and construct the GR* tree index is also a core problem to be solved. Therefore, this embodiment proposes a memory-based spatial index deserialization loading and construction method based on the above spatial index persistence storage solution, which specifically includes:

[0167] S601: After the distributed hierarchical spatial index model is constructed, a spatial query request is received;

[0168] S602: According to the spatial query request, call the global spatial index or the local spatial index from the distributed hierarchical spatial index model to obtain index information; the index information includes the index row key, column family and column.

[0169] After receiving the spatial query request, call the global spatial index or obtain the local spatial index; specifically, call the global spatial index to obtain index information including: call the HBase get() method to obtain global index information: row key: global_index, column family: geom_etyr; column: rtree; call the local spatial index to obtain index information including: call the HBase get() method to obtain local index information: row key: startkey0000, column family: geom_etyr; column: rtree.

[0170] S603: According to the index information, the GR*-Tree serialized byte array is loaded into the memory, and then deserialized into a GR*-Tree index instance, and the GR*-Tree is constructed.

[0171] In contrast to the serialized storage of spatial index information, deserialization loading refers to extracting the index information byte array stored in the local index row from the HBase table, and converting the byte array into the corresponding spatial index structure through the deserialization mechanism, that is, in this embodiment, it is the process of deserializing into a GR* tree instance. Although the memory space occupied by a single GR* tree index is greatly reduced after the index level division of the distributed hierarchical spatial index model, if the GR*-Tree index is persisted in the server-side memory for a long time, it will also generate high memory resource usage, affecting the spatial index performance. Therefore, the spatial index deserialization loading construction method adopts the principle of "loading on demand". According to the index requirements, the corresponding GR* tree index is loaded and constructed from the HBase table to the server-side memory in real time, and is released immediately after the query processing is completed, reducing the occupation and consumption of memory resources. Figure 8 As shown, Figure 8 It is a spatial index loading and building process.

[0172] exist Figure 8In the , the spatial index construction method based on server-side memory is divided into two steps: index loading and construction. The first is index loading. When processing spatial queries, it is necessary to extract the GR* tree index byte array corresponding to the local index row from the HBase SVD table according to the actual index requirements and load it into the memory. The extraction conditions are the index row key, column family and column. For example, when loading the global spatial index GR* tree, the HBase get method extracts the serialized GR* tree byte array with "global_Index" as the row key, "geometry" as the column family and "rtree" as the column; the loading of the local spatial index GR* tree is mainly reflected in the difference in the row key of the index row. The index row key is "startkey0000", which is the row key of the first row in the Region. The other conditions are the same as the global index. So far, the byte array information of the GR* tree index has been loaded into the memory. The construction of the GR* tree index is carried out in the memory. The loaded GR* tree byte information is constructed into a GR* tree instance through the deserialization method, waiting for the index call in the spatial query processing.

[0173] S103: Execute the spatial query request in the distributed hierarchical spatial index model through the spatial parallel retrieval method to obtain the spatial retrieval result.

[0174] The above are all storage and index optimization at the database level, etc. An efficient and reasonable spatial query method is also the key to speeding up spatial query retrieval. Therefore, the spatial query method in this embodiment of the present application is a distributed spatial parallel retrieval method based on a coprocessor.

[0175] Among them, spatial parallel query usually includes several parts, such as query condition preprocessing, global index retrieval, query request distribution, distributed parallel local indexing and spatial object filtering.

[0176] In an exemplary embodiment, the specific method of obtaining the spatial search result includes the following steps:

[0177] S701: Preprocessing a spatial query request; the spatial query request includes a query condition.

[0178] S702: Based on the preprocessed spatial query request, a GR* tree global spatial index in a distributed hierarchical spatial index model is constructed and executed in memory to filter out a Region set that intersects with the query condition.

[0179] S703: Distribute the spatial query request to all regions in the Region set by remote procedure call protocol batch broadcast, and the Region partition receives the spatial query request and query condition parameters.

[0180] S704: The custom query service deployed on the Region loads and executes the GR* tree local spatial index inside each Region, retrieves and filters the spatial objects in multiple Regions that meet the requirements in parallel, and filters the index retrieval set in combination with other query conditions to obtain the spatial retrieval results.

[0181] Therefore, the overall process of spatial query can be described as follows: Fig. 9 As shown. Fig. 9 In the process, query condition preprocessing is firstly the process of classifying and processing the condition parameters passed in the spatial query request, for example, instantiating the query spatial range condition into a spatial object to facilitate spatial index retrieval. Then, global index retrieval is performed, and the global spatial index GR* tree in the distributed hierarchical spatial index model is built and executed in memory to filter out the Region set that intersects with the query condition. Then, the query request is distributed to all Regions in the Region set through Remote Procedure Call Protocol (RPC) batch broadcast, and the Region partition receives the query request and query condition parameters. Finally, there is the distributed parallel local index retrieval stage, in which the Endpoint custom query service deployed on the Region loads and executes the local spatial index GR* tree in each Region, and retrieves and filters the spatial objects in multiple Regions that meet the requirements in parallel to reduce meaningless data access. At the same time, the index retrieval set is further filtered in combination with other query conditions, and finally the spatial retrieval results are obtained.

[0182] In the process of customizing the Endpoint spatial parallel retrieval service, first define the RPC service, create and implement the custom spatial parallel retrieval method spatialQuery in the Endpoint. Package the Endpoint implementation class into a jar package, deploy it to all the node RegionServers of the HBase cluster, and use the HBaseAPI to register and load the Endpoint coprocessor to the specific Region. Then, according to the spatial query conditions, perform the global spatial index globalIndex on the HBase client, filter out the qualified Region candidate set, and then send the spatial parallel query request and parameters to all Regions in the Region candidate set through the execQueryEndpointCoprocessor() method. Then, in the Region where the Endpoint coprocessor is deployed, execute the spatialQueryService spatial parallel retrieval method in the Endpoint implementation class in parallel, call the Region local spatial index for retrieval, obtain the spatial object query result set and return it to the client. Finally, the client summarizes the spatial retrieval results to complete the spatial query operation. Combining the index characteristics of global index and local index in the self-designed distributed hierarchical spatial index model, spatial query operations can be performed in parallel in the Region collection after global indexing, which minimizes the loss caused by jumping between nodes during data access and effectively improves the spatial query capability of HBase.

[0183] Optionally, the spatial parallel query algorithm is implemented as follows:

[0184]

[0185]

[0186] Based on the above process and method, the Endpoint space parallel retrieval processing mechanism can effectively reduce the system communication overhead, reduce data access and exchange costs, and optimize the system space retrieval processing performance.

[0187] This application provides a distributed storage and retrieval method for black soil arable land quality spatial vector big data. With the help of the distributed NoSql database HBase, a vector data storage model is designed. By improving the traditional Geohash space filling curve (Space-Filling Curve, SFC) indexing method, adaptive hierarchical division encoding of spatial irregular line and surface data is realized. A hybrid spatial index GR*-Tree is constructed by combining Geohash and R* tree indexes, and a distributed hierarchical spatial index model is designed. Finally, based on the spatial parallel query mechanism of the Endpoint coprocessor, the client spatial query performance is further improved.

[0188] In one embodiment, the present application also provides a method for distributed storage and retrieval of black soil arable land quality spatial vector big data. The method carries out relevant research work from three aspects: storage model, spatial index model, and spatial retrieval mechanism. The specific technical architecture is as follows: Fig.10 shown.

[0189] In terms of storage model, the Hadoop big data platform distributed file system is used as the underlying storage system. According to the characteristics of black soil arable land quality spatial vector data and combined with the HBase distributed storage principle, the row key, column family and Region partition of the storage model are designed for spatial information and attribute information respectively, so that HBase can directly support the storage and indexing of spatial vector big data, laying the foundation for building an efficient spatial index model.

[0190] In terms of spatial index model, we first improve Geohash to achieve adaptive hierarchical spatial partition encoding, and propose an adaptive hybrid spatial index GR*-Tree in combination with R* tree, in order to solve the problems of large precision error and difficulty in spatial partitioning in Geohash indexing of spatial line and surface data. We make full use of HBase distributed storage and computing resources to build a distributed hierarchical spatial index model, and finally store spatial index information persistently in server memory to achieve efficient index management and improve spatial retrieval performance at the index layer.

[0191] In terms of spatial retrieval, in order to further speed up spatial retrieval, a spatial parallel retrieval method is designed and implemented based on the above storage model and index model. The distributed hierarchical spatial index model is used to implement data filtering and screening in spatial queries, and then the custom Endpoint coprocessor spatial query service is used to implement concurrent index query in multiple regions, further improving spatial query performance.

[0192] This embodiment constructs an adaptive hierarchical spatial partitioning encoding method based on Geohash, and stores complex spatial objects in HBase according to the corresponding Geohash row key encoding to solve the problem of difficult storage and low retrieval efficiency of traditional planar data. To further improve the efficiency of spatial retrieval, a distributed hierarchical hybrid spatial index model is constructed, including a two-layer architecture of global spatial index and local spatial index. For spatial query, based on the index model, a spatial parallel query mechanism based on coprocessors is implemented to achieve parallel and efficient spatial query at the Region level.

[0193] Furthermore, in order to evaluate the effectiveness of the Geohash-based adaptive hierarchical spatial partitioning coding method provided in this application, this application evaluates the spatial retrieval performance of this method under different data volumes and different query window scenarios based on the black soil arable land quality dataset of Province A.

[0194] This application uses the Java programming language to design and implement an efficient storage and retrieval method for spatial vector big data based on HBase. Relying on the laboratory server environment, a high-availability Hadoop big data cluster is built in a virtual machine environment. The default configuration is a 10-node cluster mode, 1 NameNode master node, and 10 DataNode slave nodes. And it is configured as 1 Master node, 2 Backup Master nodes and 10 RegionServers on the HBase deployment. Based on the Hadoop ecosystem, a cluster environment including Hadoop, HBase, Zookeeper and other components is built and deployed to jointly build a highly available massive data storage and management environment. At the same time, in order to further verify the superiority of this method in spatial query and conduct comparative tests between different methods, this application also deploys the Geomesa spatiotemporal data management tool in a cluster environment, and implements a spatial query method based on GeoMesa. The experimental software and hardware environment configuration information is shown in Table 5.

[0195] Table 5

[0196]

[0197]

[0198] This application is for the research of related methods in the project "Integrated R&D of Multi-scale Big Data Platform for Black Soil Cultivated Land Quality", so the black soil cultivated land quality data of Province A in the third national land survey of my country is selected as the experimental data set. The data set is saved in ShapeFile file format, containing 585,313,000 spatial vector data entries, each of which has more than 40 types of cultivated land indicator data including element ID, soil type, organic matter content and pH value, which are used to describe the basic attribute information of black soil cultivated land quality. The data set sample is shown in Table 6.

[0199] Table 6

[0200] Geom ObjectID Soil Type Organic matter PH ...... K_Mean P_Mean Polygon 2301 Black soil 55.56 5.8 ... 217.07 58.86 Point 2310 Meadow soil 34.53 6.45 ... 211.01 90.18 LineString 2399 Paddy soil 32.01 5.8 ... 245.85 56.85

[0201] It should be noted that in the table, Geom represents spatial data, Polygon, Point, and LineString represent spatial data types, ObjectID represents the spatial element number, PH represents the pH of cultivated land, K_Mean represents the effective potassium content, and P_Mean represents the effective phosphorus content.

[0202] The overall distribution of the data is relatively concentrated, covering the entire Province A in the form of spatial surface data.

[0203] To further verify the superiority of the performance of the GR*-Tree distributed hierarchical spatial index model, this application conducts a comparative test with the GeoMesa spatiotemporal data processing engine (Hughes JN, 2015) through spatial range query, and compares and analyzes the test results. GeoMesa, as a classic spatiotemporal data engine in the current field of Geographic Information System (GIS), is deployed on HBase to jointly build a distributed spatiotemporal data management system, and has good performance for spatial range queries of massive spatial vector data. Therefore, this experiment selects GeoMesa as a comparison method for the GR*-Tree distributed hierarchical spatial index model, and conducts experiments for two control conditions of different spatial query windows and different data volumes. Each group of experiments conducts 25 queries and records the average query time. Under the same conditions, the shorter the query time, the better the performance.

[0204] In order to compare the spatial query performance of GR*-Index (GR*-Tree distributed hierarchical spatial index model) and GeoMesa under different spatial query windows, the experiment selected four continuously expanding spatial query windows R1, R2, R3, and R4. The query window information is shown in Table 6. The experimental data selected the cultivated land quality data of Province A, and the data volume was 585,313 planar spatial vector data. The experimental results are shown in Table 7 below. Table 7 shows the spatial range query time of GR*-Index and GeoMesa under different query windows.

[0205] Table 7

[0206]

[0207] In order to more intuitively show the temporal trend of spatial range query of GR*-Index and GeoMesa under different spatial query windows, the experimental results are visualized as follows: Fig.11 , Fig.11 A diagram showing the visualization of the performance comparison with Geomesa at different window sizes.

[0208] The experimental results show that as the spatial query window continues to increase, the spatial query time of GR*-Index and GeoMesa increases. This is mainly because the larger the query window, the more spatial elements it contains, the larger the amount of returned result data, and the longer the overall query time. However, compared with GeoMesa, GR*-Index has a shorter overall query time, and the query time under multiple windows of different sizes can be basically maintained within 1s. As the window increases, the query time increases slowly, and the performance is stable. In contrast, GeoMesa has a longer overall query time, especially when facing a large-scale R4 query, the query time is longer and the increase is larger. Overall, under the same query range, the average query time of GR*-Index under different windows is only 22.8% of GeoMesa, and the spatial range query performance advantage is significant.

[0209] In order to compare the spatial query performance of GR*-Index and GeoMesa under different data volumes, the experiment selected multiple data sets with different data volumes, namely 10,000, 100,000, 200,000, 300,000 and 500,000 data volumes. The same range was queried separately, and the query range selected a 0.5°*0.5° window. The experimental results are shown in Table 8.

[0210] Table 8

[0211]

[0212]

[0213] In order to more intuitively show the spatial query time changes of GR*-Index and GeoMesa methods under different data volumes, the experimental results are visualized as follows: Fig.12 As shown, Fig.12 The following is a diagram showing the visualization of the performance comparison with Geomesa under different data volumes.

[0214] As the amount of data increases, the query time of GR*-Index and GeoMesa increases. This is mainly because the increase in data volume increases the number of data rows in the HBase table and the data cardinality, so that more irrelevant data needs to be screened during query, increasing the spatial index time overhead. From the above figure, it can be found that the query time of the GR*-Index method is in the range of 399ms-516ms, and the query speed is fast. However, the spatial query speed of GeoMesa's XZ2 index in the same environment is above 1804ms, and the query speed is slow. Overall, in different data volume scenarios, the average spatial query time of GR*-Index is only 23.1% of GeoMesa, and the spatial query efficiency is significantly better than GeoMesa, and the spatial retrieval performance is significantly improved.

[0215] Compared with GeoMesa, with the support of the hierarchical distributed hybrid spatial index and Endpoint coprocessor parallel method of this application, spatial range query will be locally indexed in parallel in multiple regions, and its query speed depends on the slowest region local index time. Therefore, when the data volume increases significantly, the overall spatial query time is still short, with a slow growth rate and small change amplitude, which is suitable for massive spatial vector data query and concurrent scenarios.

[0216] This application aims at the storage and retrieval problem of massive spatial vector data in the black soil arable land quality big data platform. By designing the storage model, index model and query method of spatial vector big data, the efficient storage and retrieval of large-scale spatial vector data is realized. It has been verified by experiments that the method provided in this application is relatively stable for the storage performance of 585,313 arable land quality data in Province A, and has good spatial retrieval performance under different data volumes and different query window scenarios. When the data volume is 580,000, the query time is 333ms and 993ms respectively when the query window is 0.1°*0.1° and 1.0°*1.0°, and the overall average spatial query performance is 4.34 times that of GeoMesa, with obvious performance advantages. The distributed storage and retrieval method of spatial vector big data of black soil arable land quality in this application is practical in the black soil arable land quality big data platform, which can effectively improve the spatial retrieval efficiency of the black soil arable land quality evaluation big data platform, and can also provide a method reference for the storage and retrieval research of spatial vector big data in other fields.

[0217] The above is a distributed storage and retrieval method for black soil farmland quality space vector big data provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding distributed storage and retrieval device for black soil farmland quality space vector big data, such as Fig.13 shown.

[0218] Fig.13 A schematic diagram of a black soil arable land quality space vector big data distributed storage and retrieval device provided by the present invention, the device 1400 includes:

[0219] Design module 1301 is used to design partition coding for row keys, column families and Region partitions of the underlying storage system of the distributed non-relational database HBase according to the spatial information and attribute information of the black soil arable land quality spatial vector data, and obtain a storage model for storing spatial vector data.

[0220] Construction module 1302 is used to obtain the Geohash coding grid of the spatial vector data in the storage model through an adaptive hierarchical division coding method; convert the Geohash coding grid into a Geohash spatial grid, and construct a distributed hierarchical spatial index model based on the Geohash spatial grid and the index unit of the R* tree.

[0221] The query module 1303 is used to execute a spatial query request in a distributed hierarchical spatial index model through a spatial parallel retrieval method to obtain a spatial retrieval result.

[0222] For the specific definition of the black soil arable land quality space vector big data distributed storage and retrieval device, please refer to the definition of the black soil arable land quality space vector big data distributed storage and retrieval method above, which will not be repeated here. Each module in the above-mentioned black soil arable land quality space vector big data distributed storage and retrieval device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0223] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The provided black soil arable land quality spatial vector big data distributed storage and retrieval method.

[0224] The present invention also provides Fig.14 The structural diagram of the computer device shown in FIG. Fig.14 As shown in the figure, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The provided black soil arable land quality spatial vector big data distributed storage and retrieval method.

[0225] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0226] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A distributed storage and retrieval method for black soil arable land quality spatial vector big data, characterized in that: The method comprises: According to the spatial information and attribute information of the black soil arable land quality spatial vector data, the row key, column family and Region partition of the underlying storage system of the distributed non-relational database HBase are designed for partition coding, and the storage model for storing spatial vector data is obtained. The Geohash coding grid of the spatial vector data in the storage model is obtained through an adaptive hierarchical division coding method; The Geohash coding grid is converted into a Geohash space grid by upgrading its dimension, and a distributed hierarchical spatial index model is constructed based on the Geohash space grid and the index unit of the R* tree. The spatial query request is executed in the distributed hierarchical spatial index model through the spatial parallel retrieval method to obtain the spatial retrieval result.

2. The method according to claim 1, characterized in that The encoding design of the row key specifically includes: RowKey=GeoHash(Geom)+FeatureID Among them, RowKey represents the row key encoding of spatial vector data, Geom represents the spatial information of spatial vector data, GeoHash (Geom) is the Geohash spatial encoding of spatial vector data; FeatureID is the feature ID of the spatial vector data, which uniquely identifies the object in terms of attributes.

3. The method according to claim 1 or 2, characterized in that The encoding design of the column family specifically includes: Design spatial geometry column families and non-geometry attribute column families; The spatial information of spatial vector data is stored in the spatial geometry column family, and different types of spatial vector data are uniformly stored under the geom column of the spatial geometry column family; The attribute information of the spatial vector data is stored in the non-geometric attribute column family, wherein the field name of the attribute information is defined as the column name under the non-geometric attribute column family, and the attribute value is used as the column value.

4. The method according to claim 1 or 2, characterized in that: The coding design of the Region partition specifically includes: Based on the adaptive hierarchical division coding method, the spatial vector data is adaptively divided into hierarchical spaces, and the complex spatial objects are converted into Geohash codes. Based on actual application requirements and the natural distribution characteristics of data, perform prefix matching analysis on Geohash codes, customize a reasonable spatial partition range, and calculate the string value of each spatial partition; The string value of each spatial partition is used as the delimiter value of the storage model Region pre-partition, that is, the row key range of each Region partition; According to the row key of the spatial vector data and the range of the Region partition row key, the spatial vector data is stored in the corresponding Region.

5. The method according to claim 4, characterized in that The adaptive hierarchical division coding method specifically comprises: Obtain the minimum bounding rectangle of the spatial object to be encoded, and determine the Geohash encoding level of the spatial object to be encoded according to the minimum bounding rectangle; According to the Geohash coding level, determine the coding grid of the spatial object to be coded; According to the Geohash coding level and the coordinates of the center point of the minimum circumscribed rectangle, the coding grid is binary-coded to obtain the Geohash binary coding string of the spatial object to be coded; Based on the Base32 string conversion rule, the Geohash binary code string is converted into a string, and then the string is combined into a Geohash code set to obtain the unique Geohash string code corresponding to the spatial object to be encoded; The Geohash code will be placed in a high position as a component of the row key in the storage model design.

6. The method according to claim 1 or 2, characterized in that: The distributed hierarchical spatial index model is constructed based on the Geohash spatial grid and the index unit of the R* tree, specifically including: According to the index units of Geohash spatial grid and R* tree, the GR*-Tree hybrid spatial index structure is determined; According to the GR*-Tree hybrid spatial index structure, a distributed hierarchical spatial index model is constructed.

7. The method according to claim 6, characterized in that The distributed hierarchical spatial index model includes a GR*-Tree global spatial index based on the entire table; the distributed hierarchical spatial index model is constructed according to the GR*-Tree hybrid spatial index structure, specifically including: Using the R* tree in the GR*-Tree hybrid spatial index structure as the basic index structure and the Region related information as the basic index unit, the key-value pair set of the Geohash code and the minimum enclosing rectangle of all Regions in the storage model is obtained; Traverse the key-value pair set, take out the key-value pairs in the key-value pair set, and insert them into the R* tree as spatial objects according to the Region spatial rectangular range corresponding to the minimum enclosing rectangle; and based on the R* tree "heuristic" insertion strategy, select the leaf node to import the spatial object; Adjust the global spatial index R* tree node structure, determine the root node, and build the GR*-Tree global spatial index structure based on the entire table.

8. The method according to claim 6, characterized in that The distributed hierarchical spatial index model includes a GR*-Tree local spatial index structure based on Region; the distributed hierarchical spatial index model is constructed according to the GR*-Tree hybrid spatial index structure, specifically including: The Geohash spatial grid in the GR*-Tree hybrid spatial index structure is used as the basic unit of the GR*-Tree local spatial index structure of the Region. When the storage model stores spatial vector data, the spatial vector data is stored in the corresponding Region partitions according to the Region pre-partition boundary value, and the Geohash row key code and code grid stored in the spatial vector data uniquely represent the object. Automatically insert into the R* tree in the form of <row key code, code grid> key-value pairs according to the "heuristic" strategy; After inserting the spatial vector data, the minimum bounding rectangle of the relevant nodes is updated, and the relevant nodes of the index tree are adjusted according to the R*-Tree index node adjustment strategy; When all spatial vector data are inserted into the R*-Tree spatial index structure, the Region-based GR*-Tree local spatial index structure is constructed.

9. The method according to claim 1 or 2, characterized in that: The method further comprises: After the distributed hierarchical spatial index model is built, a spatial query request is received; According to the spatial query request, the global spatial index or the local spatial index is called from the distributed hierarchical spatial index model to obtain index information; the index information includes the index row key, column family and column; According to the index information, load the GR*-Tree serialized byte array into the memory, then deserialize it into a GR*-Tree index instance to complete the construction of the GR*-Tree.

10. The method according to claim 9, characterized in that The spatial parallel retrieval method is used to execute a spatial query request in a distributed hierarchical spatial index model to obtain a spatial retrieval result, specifically including: Preprocessing a spatial query request; the spatial query request includes a query condition; Based on the preprocessed spatial query request, the GR*-Tree global spatial index in the distributed hierarchical spatial index model is constructed and executed in memory to filter out the Region set that intersects with the query condition; The spatial query request is distributed to all regions in the Region set by remote procedure call protocol batch broadcast. The Region partition receives the spatial query request and query condition parameters. The custom query service deployed on the Region loads and executes the GR* tree local spatial index inside each Region, retrieves and filters the spatial objects in multiple Regions that meet the requirements in parallel, and filters the index retrieval set in combination with other query conditions to obtain spatial retrieval results.

Images