Slicing task balanced division method based on dimensionality reduction curve

Abstract

The invention provides a slice task balanced division method based on a dimensionality reduction curve, which comprises the following steps: selecting a Fork / Join framework and a Spark calculation framework of Java as a bottom-layer framework of a slice model, and adopting node-based Spark slice task division and Fork / Join single-machine slice task secondary division. A Fork / Join framework of Java is a computing framework special for solving single-machine parallel tasks and is simple and easy to use, core functions only include a Fork function for decomposing tasks and a join function for combining results, and parallel processing of the tasks is achieved through a single-machine thread pool; the distributed task can be well processed through the Spark framework, the framework provides a concise and clear distributed computing API, the logic structure is clear, and java is well supported. The whole tile map data is preliminarily segmented through a Driver in the spark, each slice is allocated to the Excutor of a single node, and the single node further segments the tile task by using a fork function until the size of the tile data is within the processable range of the node.

Description

Technical Field

[0001] The present invention relates to the technical field of battlefield environment data processing in surveying and mapping geography, meteorology and hydrology, and military geology, and in particular to a slicing task balanced division method based on a dimension reduction curve. Background Art

[0002] For TB-level tile data, the processing time can be as long as several days or even months using the serial slicing method on a single machine, making the release of large amounts of spatial information a bottleneck for the application of geographic information service platforms. It is difficult to meet the needs of work with strong timeliness requirements, such as emergency response, military reconnaissance, and rapid mapping.

[0003] The current tile slicing technology generally uses the ArcGIS Server slicing tool for multi-core servers, and adopts a single-machine multi-threaded parallel approach to improve slicing performance. However, for larger amounts of tile data, the time consumption of single-machine slicing is still unbearable, so distributed slicing is used to process PB-level tile data. Mapnik supports multi-node tile slicing, but this method is very complex to configure. Summary of the invention

[0004] The technical problem to be solved by the technical solution of the present invention is that the configuration of PB-level tile data processed by distributed slicing is complex, and single-machine multi-threading and distributed slicing have performance bottlenecks.

[0005] In order to solve the above technical problems, the technical solution of the present invention provides a method for balanced partitioning of slice tasks based on a dimensionality reduction curve, comprising the following steps:

[0006] The tile coordinates in the vector tile pyramid are reduced in dimension using a one-dimensional filling curve, and are encoded in ascending sequence to obtain one-dimensional coordinates. The one-dimensional coordinates are used as a key and combined with the tile coordinates to be stored in redis or other key-value databases to obtain a one-dimensional encoding sequence;

[0007] Determine whether the tile data corresponding to the one-dimensional encoding sequence is in a cluster single-node dimension or a distributed cluster environment;

[0008] If it is in the single-node dimension of the cluster, a single-machine multi-threaded slicing model is used. According to the granularity set by the slicing task, the one-dimensional coding sequence is evenly divided according to the tile data volume to generate multiple slicing tasks with multiple coding segments. Each slicing task will traverse the corresponding multiple coding segments in turn and retrieve vector data from the mixed index according to the coding information. For all spatial objects retrieved, projection conversion, simplification, and cropping are performed in turn to construct vector tiles. A single vector tile includes all the corresponding vector data within the spatial range of the tile. The single-machine multi-threaded slicing model caches the vector tiles in the output task first, and the task responsible for output uniformly outputs them to the tile storage system.

[0009] If it is in a distributed cluster environment, a distributed slicing model is used. The location of the one-dimensional coding sequence is used as the driving node. According to the granularity set by the slicing task, the one-dimensional coding sequence is evenly divided according to the tile data volume to generate multiple slicing tasks with multiple coding segments. Multiple slicing tasks are distributed to each computing node in the cluster via the network. The slicing task performs a range search on the vector data source through the spatial index based on the inverse-coded tile coordinates and tile spatial information. All retrieved spatial objects are aggregated according to the tile ID, and projection transformation, simplification, and cropping are performed in turn in Spark's "map" class transformation operator to construct vector tiles to obtain the result data set. A single vector tile includes all the corresponding vector data within the spatial range of the tile. The distributed slicing model stores the result data set in the database or converts it into other geo formats and stores it in a file.

[0010] Preferably, the code is filled in the order of tile levels in the tile coordinates from small to large, and the tile planes of each level are spread out in the same direction. The one-dimensional coordinates are obtained by numbering the rows from top to bottom and the same row from left to right according to the column numbers in the tile coordinates.

[0011] Preferably, if when constructing the vector tiles, all the vector data in a single vector tile exceeds a preset processing threshold, the data in the slicing task needs to be divided again according to a size smaller than the granularity, and the vector data is randomly and evenly divided into multiple data slices according to the data volume, and then a data processing subtask is started for each data slice. After the data processing task after the secondary division is completed, the corresponding vector data of the single vector tile is aggregated and output.

[0012] The technical solution of the present invention proposes a method for balanced partitioning of slice tasks based on a dimensionality reduction curve, selects Java's Fork / Join framework and Spark computing framework as the underlying architecture of the slice model, and adopts node-based Spark slice task partitioning and Fork / Join's single-machine slice task secondary partitioning. Java's Fork / Join framework is a computing framework designed to solve single-machine parallel tasks. It is simple and easy to use. The core functions only have the fork function for decomposing tasks and the join function for merging results. It uses a single-machine thread pool to achieve parallel processing of tasks; the Spark framework can handle distributed tasks very well. The framework provides a concise and clear distributed computing API with a clear logical structure and good support for Java. The entire tile map data is preliminarily partitioned through the Driver in spark, and each slice is assigned to the Executor of a single node. The single node uses the fork function to further partition the tile task until the tile data size is within the processable range of the node. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 A schematic diagram of dimensionality reduction and increasing sequence encoding of a one-dimensional filling curve of a vector tile pyramid provided by an embodiment of the present invention;

[0014] Figure 2 A schematic diagram of a slicing model based on a Fork / Join framework provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0015] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0016] like Figure 1 As shown, considering that the vector tile pyramid is in a three-dimensional space, in order to more conveniently divide the task, the tile coordinates (tile level z, row number x, column number y) are reduced in dimension with a one-dimensional filling curve, and the curve is encoded in ascending sequence. The filling method of the filling curve will be filled in the order from small to large according to the tile level z. The tile plane of each level is spread in the same direction, the row number x is from top to bottom, and the columns of the same row are numbered from left to right to obtain a one-dimensional coordinate d. As shown in the figure, the tile number of the second row and second column of the third level is 11. There are three levels of tiles in the figure. The tile numbered 1 is the first level, and the tiles numbered 2, 3, 4, and 5 are in the second level, and so on.

[0017] Store (key: d, val: (x, y, z)) in redis or other key-value databases, so that the level and two-dimensional coordinates of the tile can be quickly found through the reduced-dimensional coordinates.

[0018] According to the granularity set by the slicing task, the one-dimensional coding sequence is evenly divided to generate multiple segments. The tiles are divided four times in the figure, and the range of the slicing tasks is represented by four colored lines. Tiles 1, 2, 3, 4, 5, and 6 are the first slicing task, 7, 8, 9, 10, 11, and 12 are the second slicing task, 13, 14, 15, 16, 17, and 18 are the third slicing task, and 19, 20, and 21 are the fourth slicing task. From the number of tiles, the slicing is not uniform. The last task is only assigned to three slicing tasks, but considering that the amount of data of each tile is not the same, the slicing is balanced according to the amount of tile data. The amount of data of tiles 19, 20, and 21 is roughly equivalent to that of other tasks. Each coding segment is assigned to each slicing task as the input of each slicing task. Each slicing task will traverse all the codes in the input slice in turn and construct vector tiles according to the coding information. Finally, the parallel framework will realize the operation and scheduling of tasks.

[0019] After the one-dimensional code is reversed, the tile coordinates and level information that need to be constructed can be obtained. When constructing a single tile, the data to be processed for each tile includes all the vector data corresponding to the spatial range of the tile. When slicing, all spatial data need to be traversed in sequence. If the amount of data is too large, the data in the slice needs to be divided again, and the data processing subtask is started with a smaller granularity of data slices. The data in the slice is divided evenly directly according to the data in the slice. The vector data is randomly and evenly divided into multiple data slices according to the amount of data, and then a data processing subtask is started for each data slice, thereby realizing the secondary division of the slicing task. After the data processing task after the secondary division is completed, the spatial data of the same tile is aggregated and output.

[0020] Based on the above-mentioned slicing task division method and vector data sharing method, the distributed parallel slicing model can be constructed by slicing the vector tile data once in the distributed cluster environment and slicing it twice by fork / join on a single node of the cluster. The design of the parallel slicing model needs to rely on different development and operating environment dimensions. For example, in the single-node dimension, the model needs to be designed and implemented based on the Java Fork / Join parallel framework, while in the distributed cluster environment, the model slicing is implemented based on the Spark RDD programming paradigm.

[0021] (1) Single-machine multi-threaded slicing model

[0022] Based on the Fork / Join programming concept, the number of vector tiles to be processed by the initial slicing task is determined. If the number of tiles is greater than the set threshold, the slicing task is divided into multiple subtasks. According to the tile sequence slicing method described above, the large-scale tile sequence is evenly segmented and entrusted to the subtask for processing. A single slicing subtask needs to traverse the tile sequence it needs to process, and retrieve vector data from the mixed index according to the tile information obtained after decoding. For all spatial objects retrieved, projection transformation, simplification, and clipping are performed in sequence. Due to the uneven distribution of spatial data, the amount of spatial data contained in each vector tile is different. In the subsequent process of slicing the vector data, the amount of data to be processed by each subtask is also very different. When the task volume of some subtasks is too heavy, the task needs to be decomposed twice, and the vector data is divided into multiple data slices according to the data volume. A new thread task is enabled for each data slice to process it. After the processing is completed, it is merged into a vector tile for output.

[0023] The Java code for the single-node secondary splitting algorithm is as follows:

[0024] / / Main function entry

[0025] static void main(String[]args){

[0026] / / Create a divide-and-conquer task thread pool with 4 threads

[0027] ① ForkJoinPool fjp = new ForkJoinPool(4);

[0028] / / Create a divide and conquer task

[0029] / / Omit reading tile data into memory, ImageTile is the tile metadata after parsing

[0030] …

[0031] ②Task task = new Task(ImageTile);

[0032] / / Start the divide and conquer task

[0033] ③ Result result = fjp.invoke(task);

[0034] / / Output the results and send them back to spark

[0035] System.out.println(result.toString());

[0036] resultHandle(result);

[0037] …

[0038] }

[0039] / / Recursive splitting task

[0040] static class Task extends RecursiveTask <integer>{

[0041] final Task task;

[0042] private static int capacity;

[0043] Task(Task task){

[0044] this.task = task;

[0045] }

[0046] protected Integer compute(){

[0047] / / The slice data size is within the threshold, and the slice data is processed directly.

[0048] ④ if (this.task.size() <= this.capacity)

[0049] return handle(task);

[0050] / / Perform secondary slicing when the slice data is too large

[0051] ⑤ List <task>taskList = partition(task);

[0052] Task t1=new Task(taskList.get(0));

[0053] / / Create subtask

[0054] ⑥t1.fork();

[0055] Task t2=new Task(taskList.get(1));

[0056] / / Wait for subtask results and merge the results

[0057] ⑦return t2.compute()+t1.join();

[0058] }

[0059] }

[0060] The code is commented: ① First, create the ForkJoinPool task thread pool fjp according to the number of CPU cores; ② Then read the tile data into the local memory object; ③ Start the tile segmentation task; ④ Then use the tile segmentation task class Task to perform secondary segmentation on the tile data that exceeds the size limit. When the threshold requirement is met, the handle function directly processes the tile; ⑤ If it does not meet the secondary segmentation requirement, only binary segmentation is performed in the code (the number of elements in the taskList dynamic array is 2). In actual projects, the number of tiles after segmentation is determined according to the threshold limit. If the array length is n, ⑥ Fork function calls asynchronous processing subtasks to realize parallel processing of tasks; ⑦ The final return call is tn.compute()+tn-1.join()+…+t1.join(), which completes the processing result of one step of the task and returns it together.

[0061] The Fork function will start another thread to asynchronously process the subtasks, and the Join function will synchronously wait for the output results of the subtasks, thereby realizing the decomposition and merging of tasks.

[0062] like Figure 2 As shown in the figure, the model divides the overloaded subtasks again through a finer-grained secondary division, and fully utilizes idle computing resources with a finer parallel granularity, thereby avoiding the problem of a single task running for a long time becoming a bottleneck for the entire parallel slicing program.

[0063] Vector tiles will be output to the tile storage system, that is, the vector tiles will be persisted to the disk external storage, such as a file directory with specific organizational rules (hierarchical file directory) and a database system that adapts to the characteristics of vector tiles. The persistence process will inevitably involve a large number of IO operations. If the tiles are output directly in each slice processing subtask, it will cause a lot of IO blocking and easily waste computing resources. Therefore, the final generated tiles will be cached in the output task in the model first, and the task responsible for output will uniformly complete the persistent output of the vector tiles.

[0064] (2) Distributed Slicing Model

[0065] Due to the limited computing resources of a single computer, in order to adapt to the distributed cluster environment and accelerate the construction efficiency of vector tiles, the original program is expanded with the Spark distributed computing framework. According to the vector tile pyramid model, all tile coordinate sets that need to be constructed are generated in the driver node and encoded as an increasing one-dimensional numerical sequence. Then, data is sliced ​​based on the initial task granularity and each slice is distributed to each computing node in the cluster via the network. In the computing node, each slice task corresponds to an input slice. In each slice task, it is necessary to perform a range search on the vector data source through the spatial index based on the decoded tile coordinate information and tile space information. The query result data can be aggregated according to the tile ID, thereby avoiding the network paralysis problem caused by large data volume network transmission. When processing the vector data in each tile, if the data volume is too large, the slice task can be further subdivided in combination with the Fork / Join framework. Since the data exchange in this process does not pass through the network, it will not cause network transmission pressure. At the same time, large tasks can be decomposed, which can well balance the load between tasks.

[0066] In the specific implementation, first traverse the tile sequence and retrieve data in the "flatMapToPair" transformation task, then implement the data processing operations in the slice pipeline (such as projection, selection, simplification, cropping, etc.) in Spark's "map" class transformation operator. When the amount of data retrieved by the tile range is too large, subdivide the tasks in a multi-threaded manner in the transformation task, and finally generate vector tiles in data format through Spark's operation operator and output them to the database.

[0067] The specific algorithm pseudo code is shown in the following table:

[0068] Algorithm 1 Distributed Tile Construction Algorithm

[0069] / / Calculate the spatial range of each tile and obtain the tile shared array for access by nodes in the computing cluster

[0070] Input:GeoJson Output:mvt

[0071] Handle(Box-of-Tile)→Arrays[Box-of-TileN(i,j)]

[0072] / / Preprocess GeoJson to get the tile txt file

[0073] ELT(GeoJson)→txt

[0074] / / Spark reads txt tile data to build RDD data set

[0075] Spark.load(txtPath) → GeoRDD

[0076] / / Use spark map for processing and specify the coordinate system conversion function transform to convert the RDD data format

[0077] GeoRDD.map(transform(GeoRDD))→TransformedGeoRDD

[0078] / / Use map to map the converted RDD into key-value format, where key is the tile number

[0079] TransformedGeoRDD.MapToPair(geo→(id,geo))→PairedRDD

[0080] / / Adjust the number of RDD partitions to the number of cluster computing nodes n, complete the segmentation of the tile set, and all nodes process in parallel

[0081] RePartition(PairedRDD,n)

[0082] / / Perform tile calculations on the partitioned RDD

[0083] PairedRDD.Reducebykey()→resultPairedRDD

[0084] / / Output is stored as mvt file and written to disk or stored in database

[0085] ResultPairedRDD → mvt / db

[0086] The following is the Scala code implementation of the distributed slicing algorithm:

[0087] import org.apache.spark.rdd.RDD

[0088] / / The underscore "_" here is a placeholder, representing the root directory of the txt file after the tile data file is converted

[0089] val rootPath:String=_

[0090] val file:String=s”${rootPath} / ImageTiles.txt”

[0091] / / Read the contents of the tile txt file and extract geometric object points, polygons, etc. from the txt file

[0092] val transformedRDD:RDD[String]=spark.sparkContext.textFile(file)

[0093] / / Separate geometric elements in rows and filter out empty elements

[0094] val geoRDD:RDD[String]=transformedRDD.flatMap(line=>line.split(""))

[0095] val cleanGeoRDD:RDD[String]=wordRDD.filter(word=>!word.equals(""))

[0096] / / Convert the RDD elements into (Key, Value) format

[0097] val pairedRDD:RDD[(Long,GeoData)]=cleanGeoRDD.map(geoData=>(geoToID(geoData),geoData))

[0098] / / Adjust the number of tile partitions to calculate the number of cluster nodes

[0099] pairedRDD.repartition(n)

[0100] / / Perform tile calculation and processing on the partitioned RDD. handle is the processing function. The specific content is omitted.

[0101] val resultPairedRDD:RDD[(Long,GeoData)]=pairedRDD.reduceByKey(handle(geoData))

[0102] The code steps are annotated in detail. The entire process follows the steps of Algorithm 1, and finally the result dataset resultPairedRDD is obtained. You can choose to store it in a database or convert it to other geo formats and store it in a file.

[0103] Traditional tile segmentation only uses Spark distributed segmentation or single-machine segmentation. ArcGIS Server segmentation software can only perform single-machine multi-core tile segmentation tasks. Mapnik only supports distributed multi-node tile segmentation, but its configuration process is very complicated.

[0104] This solution combines the single-machine segmentation of fork / join and the distributed segmentation of spark, further improving the parallelism of a single machine and reducing the time complexity of the serial algorithm, thereby accelerating the efficiency of tile segmentation. It also solves the problem of complex configuration of traditional tile segmentation software. The spark framework defines relevant APIs that can be used out of the box, and the fork / join computing framework only has the core fork and join functions. Developers only need to focus on task segmentation, which greatly simplifies the tile segmentation process.< / task> < / integer>

Claims

1. A slicing task balanced partitioning method based on dimensionality reduction curve, characterized in that: The following steps are involved: The tile coordinates in the vector tile pyramid are reduced in dimension using a one-dimensional filling curve, and are encoded in ascending sequence to obtain one-dimensional coordinates. The one-dimensional coordinates are used as a key and combined with the tile coordinates to be stored in redis or other key-value databases to obtain a one-dimensional encoding sequence; Determine whether the tile data corresponding to the one-dimensional encoding sequence is in a cluster single-node dimension or a distributed cluster environment; If it is in the single-node dimension of the cluster, a single-machine multi-threaded slicing model is used. According to the granularity set by the slicing task, the one-dimensional coding sequence is evenly divided according to the tile data volume to generate multiple slicing tasks with multiple coding segments. Each slicing task will traverse the corresponding multiple coding segments in turn and retrieve vector data from the mixed index according to the coding information. For all spatial objects retrieved, projection conversion, simplification, and cropping are performed in turn to construct vector tiles. A single vector tile includes all the corresponding vector data within the spatial range of the tile. The single-machine multi-threaded slicing model caches the vector tiles in the output task first, and the task responsible for output uniformly outputs them to the tile storage system. If it is in a distributed cluster environment, a distributed slicing model is used. The location of the one-dimensional coding sequence is used as the driving node. According to the granularity set by the slicing task, the one-dimensional coding sequence is evenly divided according to the tile data volume to generate multiple slicing tasks with multiple coding segments. Multiple slicing tasks are distributed to each computing node in the cluster via the network. The slicing task performs a range search on the vector data source through the spatial index based on the inverse-coded tile coordinates and tile spatial information. All retrieved spatial objects are aggregated according to the tile ID, and projection conversion, simplification, and cropping are performed in turn in Spark's "map" class conversion operator to construct vector tiles to obtain the result data set. A single vector tile includes all the corresponding vector data within the spatial range of the tile. The distributed slicing model stores the result data set in the database or converts it into other geo formats and stores it in a file.

2. The method for balancing the partitioning of slice tasks based on a dimension reduction curve according to claim 1, characterized in that: The code is filled in the order of tile levels in the tile coordinates from small to large, and the tile planes of each level are spread out in the same direction. The one-dimensional coordinates are obtained by numbering the same row from top to bottom according to the row number in the tile coordinates and the same row from left to right according to the column number in the tile coordinates.

3. The method for balancing the partitioning of slice tasks based on a dimension reduction curve according to claim 1, characterized in that: If, when constructing the vector tiles, all the vector data in a single vector tile exceeds a preset processing threshold, the data in the slicing task needs to be divided again according to a size smaller than the granularity, and the vector data is randomly and evenly divided into multiple data slices according to the data volume, and then a data processing subtask is started for each data slice. After the data processing task after the secondary division is completed, the corresponding vector data of the single vector tile is aggregated and output.

Images