Method for generating tiles by mass vector data based on spark big data technology

Abstract

The invention discloses a method for generating tiles by mass vector data based on a spark big data technology. The method comprises the following steps: A, determining a data set or a structured system formed by initial vector data elements; b, reading and merging the original vector data to form a unified RDD model partition; c, solving a minimum enclosing rectangle for the space elements in the multiple RDD model partitions; and D, on the basis of a big data technology, calculating which tiles of the minimum circumscribed matrix corresponding to the space elements intersect with the top layer. Vector graph cutting is realized by adopting a spark big data technology, the characteristic of the spark based on memory calculation is utilized, the memory calculation efficiency is extremely high, multiple servers can be used at the same time based on the spark, deployment is carried out in a cluster mode, the graph cutting efficiency is in direct proportion to the number of server cores and the memory, and expansion is easy so as to adapt to a larger-scale data size. The problem of image cutting efficiency in a big data scene is solved.

Description

Technical Field

[0001] The present invention relates to the field of map development technology, and in particular to a method for generating tiles from massive vector data based on spark big data technology. Background Art

[0002] In the practice of GIS (Geographic Information System) map development, dealing with huge maps that exceed the screen display range is a common challenge. In order to meet this challenge, the industry introduced the concept of Tile Map. The basic idea of ​​Tile Map is to divide the entire world map into multiple small pieces (i.e. tiles). Each tile is an independent small map. By splicing these tiles together in a specific order and rules, a complete map view can be displayed. The construction of Tile Map is based on the pyramid model, which is a multi-scale, multi-resolution data structure. From the bottom to the top of the pyramid, the resolution of the map gradually decreases, but the represented geographic range remains unchanged. In the construction process, first determine the zoom level N required for the map application, and use the map with the highest zoom level (i.e. the largest map scale) as the bottom layer of the pyramid, i.e. layer 0. Then, starting from the upper left corner of the map, follow the order from left to right and from top to bottom. The map is divided into square tiles of the same size to form a tile matrix of the 0th layer. After the 0th layer tile matrix is ​​constructed, the upper layer of the pyramid (i.e., the 1st layer) is constructed. Each tile of this layer is composed of the corresponding four tiles in the 0th layer, so the resolution is reduced by half. The same segmentation operation is performed on these combined tiles to form square tiles of the same size as the next layer, thus forming the tile matrix of the 1st layer. By repeating the above process, the 2nd layer, the 3rd layer, and the N-1th layer can be constructed, and finally a complete tile pyramid model is formed. This structure makes it possible to quickly load and display map tiles of the corresponding level according to the user's zoom requirements, thereby providing an efficient and smooth map browsing experience.

[0003] Vector slicing is a technology that divides large geographic data sets into small pieces based on the principle of the pyramid model for easy storage, transmission and rendering. It slices geographic data according to certain rules so that each slice contains only a part of the geographic information, thereby reducing the data size and improving data processing efficiency. The principle of vector slicing is to divide geographic data according to spatial range and attributes. First, according to the spatial range of geographic data, it is divided into multiple small geographic areas, each area is called a slice, and then, according to the attribute information of geographic data, the data in each slice is further divided, for example, it is divided into different layers according to different data types.

[0004] With the development of the times, the amount of data is growing. The central processing unit (CPU) and memory resources of a single computer device cannot support vector slicing of massive spatial data. At the same time, the vector slicing method commonly used in the industry in the past has the problem of repeated calculation in the vector slicing process. The current method of generating tiles from vector data is mainly based on PostGIS and multi-threading to generate vector tiles. In a big data environment, this method has low efficiency in generating vector tiles. Summary of the invention

[0005] The purpose of the present invention is to provide a method for generating tiles from massive vector data based on spark big data technology, so as to solve the problems raised in the above background technology.

[0006] To achieve the above object, the present invention provides the following technical solution: a method for generating tiles from massive vector data based on Spark big data technology, comprising the following steps: A. Determine which data sets or structured systems are required to be composed of initial vector data elements; B. Read and merge the original vector data to form a unified RDD model partition; C. Solve the minimum enclosing rectangle for the spatial elements in multiple RDD model partitions; D. Based on big data technology, calculate which tiles on the top layer intersect with the minimum circumscribed matrix corresponding to the spatial element, and record the row and column numbers of these tiles; E. Calculate the secondary tile row and column numbers based on the top-level row and column numbers; F. Convert each vector coordinate into pixel coordinate. To display vector data in an electronic map, it is necessary to convert the coordinates of the vector data into pixel coordinates and use Spark parallel computing. G. Encode as vector tiles: Select the appropriate vector tile encoding format.

[0007] Preferably, the initial vector data range in step A includes multiple vector data, and these vector data elements can accurately describe various objects in the geographic space, such as road networks, building outlines, and lake boundaries.

[0008] Preferably, the RDD model partition in step B includes: multiple spatial elements, the spatial elements describe the attribute information corresponding to the vector data in the corresponding model partition, use spark to read the vector data of each layer in parallel, and convert the binary vector data into geometry type, specifically, use appropriate libraries to read the original vector data from various formats, and convert the read vector data into Spark RDD.

[0009] Preferably, the spatial element in step C is a solid polygonal element, and the dotted box is the minimum bounding rectangle of the line element at a certain level. Each element in the RDD partition is an object containing spatial elements and attribute information. The element object contains geometric shapes and related attribute information. An MBR data structure is defined to store the boundary information of the minimum bounding rectangle, which is a four-tuple containing the minimum X coordinate, the minimum Y coordinate, the maximum X coordinate and the maximum Y coordinate. The MBR of each RDD partition is calculated: In each RDD partition, the spatial elements are traversed and a current MBR state is maintained. For the first element of the partition, Initialize the MBR as its boundary. For each subsequent feature in the partition, update the MBR to include the boundary of the feature. Aggregate the partition MBR to obtain the global MBR: Use Spark's reduce or aggregate operation to aggregate all partition MBRs to calculate the global MBR of the entire dataset. The reduce operation accepts a binary function that accepts two MBRs as input and returns their minimum enclosing rectangle. The initial value can be set to an "empty" MBR, which will be replaced immediately when the first partition MBR is processed. Execute the entire operation to get the minimum enclosing rectangle of the spatial feature.

[0010] Preferably, the step D includes detailed disassembly steps: ①, obtaining the upper left and lower right coordinates of the Bbox based on the minimum circumscribed rectangle, wherein the upper left and lower right coordinates of the Bbox are the coordinates of the upper left corner and the lower right corner of the minimum circumscribed rectangle converted into longitude and latitude coordinates under the Mercator projection; ②, obtaining the tile row and column number matrix of the Bbox at the specified level, and the tile row and column numbers are obtained by performing a geographic space conversion on the longitude and latitude coordinates; ③, obtaining the tiling of the row and column number matrix, and expanding the row and column number matrix through a specific algorithm to facilitate subsequent spatial iterative calculations; ④, converting the row and column numbers into tiles with buffers; ⑤, performing spatial operations on spatial elements and tiles and marking the row and column numbers.

[0011] Compared with the prior art, the present invention has the following beneficial effects: The present invention adopts spark big data technology combined with gis vector data map cutting: most of the vector tile map cutting methods based on PostGIS in the past can only be deployed on a single server for operation, which cannot solve the efficiency problem of massive vector data map cutting in big data scenarios, and can only have a relatively good effect on small-scale data. The present invention adopts spark big data technology to realize vector map cutting, and utilizes the characteristics of spark based on memory computing. As we all know, the efficiency of memory computing is extremely high. At the same time, multiple servers can be used at the same time based on spark, and deployed in a cluster manner. The map cutting efficiency is proportional to the number of server cores and memory, and is easy to expand to adapt to larger-scale data volumes, solving the map cutting efficiency problem in big data scenarios; The pyramid model is constructed in reverse order: In the past, most methods for constructing the pyramid model are to construct in forward order, that is, to construct from level 1 to level 18 in sequence. This method requires repeated spatial operations for many times. Each tile at each level will generate 2*2 tiles at the next level, that is, 4 spatial calculations are required, and spatial operations often consume server performance. The present invention adopts a method for constructing the pyramid model in reverse order, and only spatial operations need to be performed when constructing the tiles at level 18. The row and column numbers of the tiles at level 17-1 are obtained from the row and column numbers of the tiles at the previous level. This part of the calculation is optimized from the spatial operation required in the traditional method to only digital operations, which can greatly reduce the amount of spatial calculations, thereby improving the efficiency of the entire vector cutting process. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is an example diagram of the pyramid model of the present invention; Figure 2 It is a technical flow chart of the present invention; Figure 3 This is a schematic diagram of the minimum circumscribed rectangle of the present invention; Figure 4 This is a schematic diagram of the core process of slicing a large data vector slice method of the present invention; Figure 5 It is a reverse pyramid model diagram of the present invention; Figure 6 It is a tile coordinate diagram of a line element of the present invention at different levels; Figure 7 This is a vector data map of buildings in a county in Anhui Province of the present invention. DETAILED DESCRIPTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "provided with", "connected", etc. should be understood in a broad sense. For example, "connected" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0015] See also Figure 1-Figure 7 A method for generating tiles from massive vector data based on spark big data technology includes the following steps: A. Determine which data sets or structured systems are required to be composed of initial vector data elements; B. Read and merge the original vector data to form a unified RDD model partition; C. Solve the minimum enclosing rectangle for the spatial elements in multiple RDD model partitions; D. Based on big data technology (spark), the minimum circumscribed matrix corresponding to the spatial element is calculated to intersect with which tiles on the top layer, and the row and column numbers of these tiles are recorded. The traditional method of cutting the map from level 1 is optimized to cutting the map from level 18. The circumscribed rectangle of each vector data is calculated through spark big data, and it is determined which tiles on level 18 it intersects with, so as to mark the row and column numbers of each vector data. Spark uses memory parallel computing to solve the efficiency problem of generating tiles by processing massive vector data based on traditional PostGIS; E. Calculate the row and column numbers of the secondary tiles based on the top row and column numbers. For the top layer (zoomLevel=3), the traditional slicing starts from zoomLevel=0 for vector slicing. Each layer requires spatial calculation. However, this application can greatly reduce the amount of spatial calculation. It only needs to calculate the row and column numbers of the top layer. The row and column numbers of the lower layers are calculated by the row and column numbers of the higher layers (row / 2, column / 2, zoom - 1). For example, the row and column numbers of a feature at level 7 are (row, column, level) = (105,52 ,6), then its row and column numbers at the 7th level are (row, column, level) = (52,26,6). After the row and column numbers are determined, the vector slice at this level has been completed. In order to vividly illustrate the relationship between the spatial elements in different tile levels in the embodiment of the present disclosure, an example is provided. The spatial elements are represented by solid lines, and their spatial ranges at different levels are defined by the minimum bounding rectangle (MBR) marked by the dotted box. Assuming that the spatial partition level is 2, and the maximum level of the vector slice is set to 4, a sub-quadtree can be constructed with spatial partition 15 as the starting point. The level of its leaf nodes is consistent with maxLevel, and each quadtree node uses an MBR to define the spatial range it covers. Subsequently, the spatial elements will be assigned to multiple quadtree nodes that intersect with their MBRs. According to the correspondence between the quadtree and the vector pyramid, each Each quadtree node corresponds to a vector tile. The spatial elements that need to be encoded in the tile are exactly those that intersect with the node MBR. Taking the line element as an example, it will be encoded in tile No. 15 of level 2, tiles No. 60 and No. 61 of level 3, and tiles No. 241, 243, 244 and 246 of level 4. In levels 3 and 4, due to the difference in tile size and line element length, the line element will span multiple vector tiles and present different row and column numbers in these tiles. In order to simplify the process of calculating these row and column numbers, a reverse pyramid row and column number calculation method is proposed. The specific operation is the maxLevel level (that is, level 4). Taking the first tile (tile No. 241) and the third tile (tile No. 243) in the upper left corner as examples to calculate the row and column numbers of the third level, since the projection coordinates of the line element are fixed, such as the coordinates of a point (182.9, 173.4), then no matter which level it is, its coordinates will not change due to the change of level, so you only need to calculate the row and column numbers of different levels. For example, taking the first tile (Tile No. 241) and the third tile (Tile No. 243) in the upper left corner as an example, the row and column numbers of the third level will all fall on Tile No. 60. This can efficiently manage and calculate the row and column numbers of spatial elements in different tile levels, and innovatively optimize the traditional cutting process, abandoning the step-by-step cutting method starting from the lowest level, and directly performing the cutting operation from the highest level. This improvement significantly improves the processing efficiency, especially when processing massive vector data. Specifically, assuming that the cutting level is 1-18, this application only uses spatial calculations at level 18, through Through the Spark big data processing framework, the circumscribed rectangle of the vector data is efficiently calculated, and the tiles of the 18th level are judged by spatial calculation, so that each vector data is given an accurate row and column number. At the 17th level, the subsequent cutting operation can be completed by digital calculation according to the row and column number of the 18th level, which greatly simplifies the processing flow. By analogy, the same method is adopted from the 16th level to the 1st level. This strategy cleverly avoids the repeated tedious spatial calculations between the 17th level and the 1st level. Through this optimization method, the efficiency bottleneck problem caused by the large number of repeated spatial calculations in the traditional cutting method when processing massive vector data is successfully solved, providing a more efficient and reliable solution for spatial data processing in the big data era. F. Convert each vector coordinate into pixel coordinate. To display vector data in an electronic map, it is necessary to convert the coordinates of the vector data into pixel coordinates and use spark parallel computing to improve computing efficiency. G. Encode as vector tiles: Select an appropriate vector tile encoding format (Mapbox Vector Tile format), which supports efficient transmission and rendering, compresses the extracted vector data to reduce file size and transmission costs, encodes the vector data using the selected encoding format, and generates vector tile files, which usually contain geometry, attribute information, style information, etc.

[0016] The initial vector data range in step A includes multiple vector data, and these vector data elements can accurately describe various objects in geographic space, such as road networks, building outlines, and lake boundaries.

[0017] The RDD model partition in step B includes: multiple spatial elements, which describe the attribute information corresponding to the vector data in the corresponding model partition. Spark is used to read the vector data of each layer in parallel, and the binary vector data is converted into geometry type. Specifically, appropriate libraries (such as GeoTools, PostGIS, etc.) are used to read the original vector data from various formats (such as ERIShapefile, GeoJSON, etc.), and the read vector data is converted into Spark RDD, which involves converting geospatial data into a format that Spark can handle, such as Case Class or custom objects in Scala to represent spatial elements. These classes should contain geometric shapes (such as points, lines, and surfaces) and attribute information associated with them (such as names, IDs, and other metadata).

[0018] The spatial element in step C is a solid polygonal element, and the dotted box is the minimum enclosing rectangle of the line element at a certain level. Each element in the RDD partition is an object containing spatial elements (such as points, lines, and surfaces) and attribute information. The feature object contains geometric shapes (such as Point, LineString, Polygon, etc.) and related attribute information, define an MBR data structure to store the boundary information of the minimum enclosing rectangle, which is a four-tuple containing the minimum X coordinate, the minimum Y coordinate, the maximum X coordinate, and the maximum Y coordinate. Calculate the MBR of each RDD partition: In each RDD partition, traverse the spatial features and maintain a current MBR state. For the first feature of the partition, initialize the MBR as its boundary. For each subsequent feature in the partition, update the MBR to include the boundary of the feature. This usually involves comparing the boundary of the feature with the boundary of the current MBR and updating the minimum and maximum coordinates of the MBR accordingly. Aggregate the partition MBR to obtain the global MBR: Use Spark's reduce or aggregate operation to aggregate all partition MBRs to calculate the global MBR of the entire dataset. The reduce operation accepts a binary function that accepts two MBRs as input and returns their minimum enclosing rectangle (that is, merges the two MBRs). The initial value can be set to an "empty" MBR, which will be replaced immediately when the first partition MBR is processed. Execute the entire operation to get the minimum enclosing rectangle of the spatial feature.

[0019] Step D includes detailed disassembly steps: ①, based on the minimum enclosing rectangle, obtain the upper left and lower right coordinates of the Bbox, where the upper left and lower right coordinates of the Bbox are the coordinates of the upper left corner and the lower right corner of the minimum enclosing rectangle converted into longitude and latitude coordinates under the Mercator projection; ②, obtain the tile row and column number matrix of the Bbox at the specified level, and the tile row and column numbers are obtained by converting the longitude and latitude coordinates into geographic space. This conversion process ensures the accurate correspondence between map tiles and geographic coordinates. In order to efficiently manage and operate these tile data, a matrix data structure is used for encapsulation. Through matrix encapsulation, tiles in a specific area can be easily located, which improves data processing efficiency and enhances scalability and flexibility; ③, obtain the tiling of the row and column number matrix, and expand the row and column number matrix through a specific algorithm to facilitate subsequent spatial iterative calculations; ④, the row and column numbers are converted into tiles with buffers. The tiles in the buffer are used in geographic information systems (GIS) and map applications. ⑤, spatial elements and tiles perform spatial operations and mark row and column numbers. Use spatial analysis libraries (such as GDAL, GeoTools, etc.) to perform spatial operations to determine which spatial elements are within a specific tile range. This is achieved through spatial queries or spatial selection operations. These operations are based on the spatial relationship between the element and the tile boundary (such as intersection, inclusion, etc.) to filter elements, determine the tile row and column numbers of the elements, and attach them to the elements as attribute information. This is achieved by adding new fields to the attribute table of the element and filling in the row and column numbers as field values.

[0020] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for generating tiles from massive vector data based on Spark big data technology, characterized by: The following steps are involved: A. Determine which data sets or structured systems are required to be composed of initial vector data elements; B. Read and merge the original vector data to form a unified RDD model partition; C. Solve the minimum enclosing rectangle for the spatial elements in multiple RDD model partitions; D. Based on big data technology, calculate which tiles on the top layer intersect with the minimum circumscribed matrix corresponding to the spatial element, and record the row and column numbers of these tiles; E. Calculate the secondary tile row and column numbers based on the top-level row and column numbers; F. Convert each vector coordinate into pixel coordinate. To display vector data in an electronic map, it is necessary to convert the coordinates of the vector data into pixel coordinates and use Spark parallel computing. G. Encode as vector tiles: Select the appropriate vector tile encoding format.

2. According to claim 1, a method for generating tiles from massive vector data based on spark big data technology is characterized by: The initial vector data range in step A includes multiple vector data, and these vector data elements can accurately describe various objects in the geographic space, such as road networks, building outlines, and lake boundaries.

3. According to claim 1, a method for generating tiles from massive vector data based on spark big data technology, characterized in that: The RDD model partition in step B includes: multiple spatial elements, the spatial elements describe the attribute information corresponding to the vector data in the corresponding model partition, use spark to read the vector data of each layer in parallel, and convert the binary vector data into geometry type, specifically, use appropriate libraries to read the original vector data from various formats, and convert the read vector data into Spark RDD.

4. The method for generating tiles from massive vector data based on spark big data technology according to claim 1, characterized in that: The spatial element in step C is a solid polygon element, and the dotted box is the minimum bounding rectangle of the line element at a certain level. Each element in the RDD partition is an object containing spatial elements and attribute information. The feature object contains geometric shapes and related attribute information. An MBR data structure is defined to store the boundary information of the minimum bounding rectangle, which is a four-tuple containing the minimum X coordinate, the minimum Y coordinate, the maximum X coordinate and the maximum Y coordinate. The MBR of each RDD partition is calculated as follows: In each RDD partition, the spatial elements are traversed and a current MBR state is maintained. For the first element of the partition, the initial Convert the MBR to its boundaries. For each subsequent feature in the partition, update the MBR to include the boundaries of the feature. Aggregate the partition MBRs to obtain the global MBR: Use Spark's reduce or aggregate operation to aggregate all partition MBRs to calculate the global MBR for the entire dataset. The reduce operation accepts a binary function that accepts two MBRs as input and returns their minimum enclosing rectangle. The initial value can be set to an "empty" MBR, which will be replaced immediately when the first partition MBR is processed. Execute the entire operation to get the minimum enclosing rectangle of the spatial feature.

5. The method for generating tiles from massive vector data based on spark big data technology according to claim 1, characterized in that: The step D includes detailed disassembly steps: ①, obtaining the upper left and lower right coordinates of the Bbox based on the minimum enclosing rectangle, wherein the upper left and lower right coordinates of the Bbox are the coordinates of the upper left corner and the lower right corner of the minimum enclosing rectangle converted into the longitude and latitude coordinates under the Mercator projection; ②、Get the tile row and column number matrix of Bbox at the specified level, and the tile row and column numbers are obtained by converting the longitude and latitude coordinates into geographic space; ③. Obtain the tiling of the row and column number matrix, and expand the row and column number matrix through a specific algorithm to facilitate subsequent spatial iterative calculation; ④. The row and column numbers are converted into tiles with buffers; ⑤. Perform spatial operations on spatial elements and tiles and mark row and column numbers.

Images