Multi-level grid tile distributed slicing method based on Kafka

Abstract

According to the Kafka-based multi-level grid tile distributed slicing method provided by the invention, the Kafka is introduced as a distributed message publishing and subscribing system, so that a parallel slicing scheme of a plurality of images can be improved. Firstly, a slicing task request is sent to a main node, then the main node carries out task division, and slicing tasks of different images are distributed to different nodes. Next, a parallel slicing task of a single image is realized on each node by using a queuing polling method and a hierarchical classification and division method; and the parallel slicing tasks of all the images are finally completed until all the nodes inform the main node that the local slicing tasks are completely completed.

Description

Technical Field

[0001] The present invention relates to the field of surveying and mapping geography and image slicing technology, and in particular to a multi-level grid tile distributed slicing method based on Kafka. Background Art

[0002] In the actual image slicing production process, the slicing of a single image can be performed in parallel through the multi-threading method of a single machine. However, it is often necessary to slice multiple images in parallel at the same time, which requires the use of the traditional static allocation method. This method divides the images in advance before the slicing of multiple images begins, and assigns each image to the corresponding machine for parallel slicing of a single image. However, this static allocation method may face the problem of unbalanced task allocation or insufficient performance of a certain machine, thereby affecting the completion efficiency of the overall slicing task. Therefore, it is necessary to find a better way to improve the efficiency of slicing multiple images, such as using a dynamic allocation method to dynamically allocate slicing tasks according to the real-time performance and task conditions of the machine, so as to make full use of machine resources and improve slicing efficiency. Summary of the invention

[0003] The technical problem to be solved by the technical solution of the present invention is that the static allocation method may lead to unbalanced task allocation and increased communication overhead between machines in multiple image slices.

[0004] In order to solve these problems, the technical solution of the present invention provides a multi-level grid tile distributed slicing method based on Kafka, comprising the following steps:

[0005] Select multiple remote sensing images to be sliced, create multiple slicing task lists for single images, and submit slicing requests and slicing task lists corresponding to the multiple remote sensing images to be sliced ​​to the master node of the Kafka cluster;

[0006] The master node of the Kafka cluster receives and caches the slice request and slice task list, and stores them in different partitions according to the balancing strategy;

[0007] According to the allocation rules, different Consumers in the slicing task cluster are assigned to different Partitions. Multiple Consumers continuously pull tasks in parallel through long connections established with multiple corresponding Partitions, and execute tasks in the slicing task list in order according to the order in which slicing requests are received to obtain image slices.

[0008] Preferably, the task is a single image slicing task.

[0009] Preferably, the slicing task cluster implements the parallel continuous pulling task by calling the GDAL library, outputs the image slices corresponding to the generated tiles into empty tile files, and stores them in the distributed file system.

[0010] Preferably, metadata information of the multiple remote sensing images to be sliced ​​is read, including the number of image bands, length, width, and projection coordinates.

[0011] Preferably, it is determined whether the projection coordinates are WGS84, and if not, projection conversion is performed.

[0012] Preferably, the remote sensing image data is loaded according to the projection coordinates, and the image space range and target level are read.

[0013] Preferably, according to the image space range and target level, a queuing polling method and a hierarchical classification division method are performed to divide the slice task list.

[0014] The technical solution of the present invention proposes a multi-level grid tile distributed slicing method based on Kafka, and develops a technology for dynamically allocating slicing tasks, similar to the thread pool method. First, an initial task is assigned to each machine. Once the machine completes the task, it will notify the monitoring machine, and the monitoring machine will dynamically assign new tasks to make full use of machine resources, optimize task allocation, reduce communication overhead, and improve the overall efficiency of slicing multiple images. This method can achieve balanced task allocation, make full use of machines with better performance, and avoid a machine slowing down the overall task execution efficiency. At the same time, in the process of dynamically allocating tasks, ensure that the corresponding image data is available on each machine, which is achieved by dynamically transmitting the image data to be sliced ​​during task allocation or ensuring that all the image data to be sliced ​​is stored on each machine, effectively reducing communication overhead and improving overall slicing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 A schematic diagram of the division of tasks for parallel slicing of multiple images provided in an embodiment of the present invention;

[0016] Figure 2 A distributed parallel slicing process for introducing Kafka provided in an embodiment of the present invention;

[0017] Figure 3 A flow chart of a single image slicing algorithm provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0019] like Figure 1 As shown in the figure, when performing parallel slicing of multiple images, it is necessary to first initiate a slicing task request to the master node, and then the master node divides the task and assigns the slicing tasks of different images to different nodes. Then, the parallel slicing tasks of a single image are implemented on different nodes through queuing polling and hierarchical classification. The parallel slicing tasks of all images are finally completed until each node has notified the master node that all local slicing tasks have been completed.

[0020] Kafka is a distributed message publishing and subscription system, which is widely used in the field of big data. Its performance in daily production environment is not affected by the hardware environment and system platform, so Kafka is introduced in the distributed parallel slicing solution to realize slicing task delivery. The characteristics of the Kafka streaming platform are that it can publish and subscribe to record streams (similar to message queues), store record streams in a fault-tolerant and persistent manner, and process record streams as they occur, that is, it can process the data streams received by this platform in real time. In the process of multi-level grid tile slicing, each slicing task can be recorded as a record and then passed to the Kafka cluster, and the tasks are distributed to each task processing server. The server processes the task data and requests it is responsible for, and completes multi-level grid tile slicing.

[0021] like Figure 2 As shown in the figure, the execution steps of the distributed multi-level grid tile algorithm introduced into the Kafka cluster are described as follows:

[0022] Step 1: Select multiple remote sensing images to be sliced ​​and submit slice requests for multiple images to the master node;

[0023] Step 2: Receive image slicing requests, create a slicing task list, and submit image slicing requests to the master node;

[0024] Step 3: The Kafka cluster receives the task and caches it, stores it in different partitions according to the balancing strategy, and executes the task in order according to the order in which the messages are received. Each task can be a slice task of a single image.

[0025] Step 4: Each Consumer in the slicing task cluster continuously pulls tasks for execution through a persistent connection established with Kafka;

[0026] Step 5: According to the allocation rules, different Consumers are assigned to different Partitions, without interfering with each other, and are executed in parallel until all slice tasks are completed.

[0027] like Figure 3 As can be seen from the algorithm flow chart, multiple images are divided into multiple single-image slicing tasks during the distributed parallel slicing process. The specific steps of the single-image parallel slicing algorithm are described as follows:

[0028] Step 1: Read the metadata information of the image, including the number of image bands, size (length and width), projection coordinates, etc.

[0029] Step 2: Make a judgment based on the projection coordinates obtained in Step 1 to ensure that the coordinate system of the remote sensing image is WGS84. If the projection coordinates are not WGS84, perform projection conversion.

[0030] Step 3: Load remote sensing image data and read image spatial range, band data, target level and other information;

[0031] Step 4: According to the image data range, slice level and other parameters, the slice task units are divided according to the queuing polling method and the hierarchical classification method and executed in parallel;

[0032] Step 5: In the slicing cluster, the parallel slicing task of the image is realized by calling the GDAL library, the generated tiles are output to the empty tile file, and stored in the distributed file system;

[0033] Step 6: After completing all slicing tasks, notify all image slicing tasks that they have been completed.

[0034] The beneficial effects of the embodiments of the present invention are as follows:

[0035] By introducing Kafka as a distributed message publishing and subscription system, the parallel slicing scheme for multiple images can be improved. In this scheme, a slicing task request is first sent to the master node, and then the master node divides the task and assigns the slicing tasks of different images to different nodes. Next, the queuing polling method and hierarchical classification partitioning method are used on each node to implement the parallel slicing task of a single image. The parallel slicing tasks of all images are finally completed until all nodes notify the master node that all local slicing tasks are completed.

[0036] The introduction of Kafka's streaming platform has the following benefits:

[0037] 1. High performance and scalability: Kafka is widely used in the field of big data. It is not restricted by hardware environment and system platform. It has high performance and scalability and can handle large-scale slicing tasks.

[0038] 2. Fault tolerance and persistence: Kafka stores record streams in a fault-tolerant and persistent manner, ensuring reliable delivery and processing of slice tasks and maintaining data integrity even in the event of node failure or network interruption.

[0039] 3. Real-time processing: The Kafka streaming platform can perform real-time processing when the record stream occurs, and can respond to the processing requirements of the slicing task in a timely manner, improving the real-time performance and efficiency of the slicing task.

[0040] By passing each slicing task as a record to the Kafka cluster and distributing it to each task processing server, multi-level grid tile slicing tasks can be implemented. This solution takes advantage of the characteristics of Kafka and transmits and processes slicing tasks in the form of messages, improving the transmission efficiency and processing capacity of slicing tasks.

[0041] In summary, by introducing Kafka as a distributed message publishing and subscription system, the parallel slicing scheme of multiple images can be improved, the transmission efficiency and processing capacity of slicing tasks can be improved, and the overall efficiency of slicing multiple images can be optimized.

[0042] Key technical points and points to be protected in the embodiments of the present invention:

[0043] 1. Dynamic allocation of slicing tasks: By introducing a thread pool-like approach, slicing tasks are dynamically allocated to fully utilize machine resources, optimize task allocation, reduce communication overhead, and improve the overall efficiency of multi-image slicing.

[0044] 2. Task division and allocation: The master node divides tasks according to slicing task requests and allocates slicing tasks of different images to different nodes to ensure the balance and efficiency of task allocation.

[0045] 3. Dynamic slicing task allocation method: involves a method for dynamically allocating slicing tasks, including task division, task allocation and task completion notification mechanism, as well as technology for implementing dynamic task allocation using a thread pool-like method.

[0046] 4. Kafka-based slice task delivery system: Involves using Kafka as a distributed message publishing and subscription system to achieve the delivery and processing of slice tasks, including the publishing, subscription and processing of task records, as well as the application of the characteristics of the Kafka streaming platform.

Claims

1. A multi-level grid tile distributed slicing method based on Kafka, characterized in that: The following steps are involved: Select multiple remote sensing images to be sliced, create multiple slicing task lists for single images, and submit slicing requests and slicing task lists corresponding to the multiple remote sensing images to be sliced ​​to the master node of the Kafka cluster; The master node of the Kafka cluster receives and caches the slice request and slice task list, and stores them in different partitions according to the balancing strategy; According to the allocation rules, different Consumers in the slicing task cluster are assigned to different Partitions. Multiple Consumers continuously pull tasks in parallel through long connections established with multiple corresponding Partitions, and execute tasks in the slicing task list in order according to the order in which slicing requests are received to obtain image slices.

2. A multi-level grid tile distributed slicing method based on Kafka as claimed in claim 1, characterized in that: The task is a single image slicing task.

3. A multi-level grid tile distributed slicing method based on Kafka as claimed in claim 1, characterized in that: The slicing task cluster implements the parallel continuous pulling task by calling the GDAL library, outputs the image slices corresponding to the generated tiles into empty tile files, and stores them in the distributed file system.

4. A multi-level grid tile distributed slicing method based on Kafka as claimed in claim 1, characterized in that: The metadata information of the multiple remote sensing images to be sliced ​​is read, including the number of image bands, length, width, and projection coordinates.

5. A multi-level grid tile distributed slicing method based on Kafka as claimed in claim 3, characterized in that: Determine whether the projection coordinates are WGS84, and if not, perform projection conversion.

6. A Kafka-based multi-level grid tile distributed slicing method as claimed in claim 4, characterized in that: Load remote sensing image data according to projection coordinates, and read the image space range and target level.

7. A multi-level grid tile distributed slicing method based on Kafka as claimed in claim 5, characterized in that: According to the image space range and target level, a queuing polling method and a hierarchical classification division method are used to divide the slice task list.

Images