Multi-thread parallel processing and slice generation method based on geographic image data

Abstract

The invention relates to a multi-thread parallel processing and slice generating method based on geographic image data, which comprises the following steps of: constructing a geographic image data set, acquiring the width, height, projection information and geographic transformation parameters of images in the geographic image data set, accurately converting source projection into a target projection coordinate system, and generating a multi-thread slice; calculating a unit value UPP of each pixel; setting a driver, configuring environment parameters of multi-thread processing, controlling the maximum thread of a thread pool, and creating the thread pool; processing the geographic image in the thread pool to generate a tile; and the generated tiles are stored according to a hierarchical directory structure, so that access and management are facilitated. According to the method, multi-thread parallel processing can be realized, geographic space data operation is carried out, and rapid conversion from a single large geographic image to a multi-stage pyramid tile map is realized.

Description

Technical Field

[0001] The present application relates to the field of geographic image processing technology, and in particular to a multi-threaded parallel processing and slice generation method based on geographic image data. Background Art

[0002] In the field of geographic information systems and remote sensing image processing, processing large geographic image data is a common and very important task. With the rapid development of remote sensing technology, the amount of image data has increased significantly, reaching tens or even hundreds of GB. Therefore, existing image slice processing technology faces huge challenges, including: (1) Coordinate system conversion accuracy: Some existing tools suffer from accuracy loss when performing coordinate conversion, especially when processing non-standard coordinate systems. This may cause the generated tiles to be misaligned or deformed when stitching, greatly affecting the overall quality of the map. (2) Processing efficiency: Most existing slicing software still uses single-threaded processing, which has low processing efficiency; (3) Lack of adaptability: Existing software usually has difficulty in automatically adjusting processing strategies based on different hardware configurations, such as the number of CPU cores and available memory, which results in inability to fully utilize system resources.

[0003] Faced with these challenges, there is an urgent need for a solution that can efficiently process large geographic image data, perform accurate coordinate transformation, flexibly manage system resources, and generate high-quality multi-level tile maps. Summary of the invention

[0004] The purpose of the present invention is to provide a multi-threaded parallel processing and slice generation method based on geographic image data, which can realize multi-threaded parallel processing, perform geographic spatial data operations, and realize rapid conversion from a single large geographic image to a multi-level pyramid tile map.

[0005] The technical solution adopted by the present invention is: a multi-threaded parallel processing and slice generation method based on geographic image data, comprising the following steps: S1: Construct a geographic image dataset, obtain the width, height, projection information and geographic transformation parameters of the images in the geographic image dataset, accurately transform the source projection into the target projection coordinate system, and calculate the unit value UPP per pixel; S2: Set the driver and configure the environment parameters for multi-threaded processing, control the maximum number of threads in the thread pool, and create a thread pool; S3: Processing the geographic image in the thread pool to generate tiles; when generating tiles, it is necessary to calculate the corresponding area of ​​the current tile in the original image, read the data of the corresponding area from the original image, and resample, and the resampling process is based on the per-pixel unit value UPP of the zoom level of different standard maps to ensure the accuracy of the generated tiles in terms of geographical location; S4: Store the generated tiles in a hierarchical directory structure for easy access and management.

[0006] Furthermore, the specific steps of step S3 are: S301: Calculate the number of tiles in each layer and the geographic coordinate range corresponding to each tile according to the minimum zoom level and maximum zoom level input by the user and the pixel resolution of the geographic image data. The specific calculation formula is: hNo=(lon+180.0)×scale / 360.0; vNo= (90.0+lat)×scale / 360.0; Among them, hNo represents the horizontal number of the tile, vNo represents the vertical number of the tile, lon represents the latitude of the upper left corner of the tile, lat represents the longitude of the upper left corner of the tile, and scale represents the scaling ratio; S302: Generate tiles through the thread pool and process tile slices using multi-threading; when each tile is generated, first obtain the semaphore that controls the multi-threading to control the number of concurrency; S303: In the specific slicing method, a mutex lock QMutexLocker is used to protect shared resources. Within the scope of the mutex lock QMutexLocker, any thread that attempts to acquire the same mutex lock will be blocked until the current thread completes the release of the mutex lock; S304: Read the geographic information data such as the latitude and longitude of the corresponding area of ​​the slice in the source dataset, and write the read data into the raster data buffer of the target tile; the corresponding area is based on the unit pixel ratio UPP of each zoom level, and the read area is dynamically adjusted through the zoom adjustment mechanism to ensure consistent geographic accuracy at each zoom level; after the reading is completed, close the source dataset and release the resources; S305: Unlock the mutex and release the semaphore to allow other threads to continue to execute the tile generation task; use atomic operations to update the number of completed tiles and notify the main thread to update the processing progress; S306: Set the correct geographic transformation parameters for each generated tile, and adjust the geographic transformation parameters of the target geographic image data set for the tile to be generated, so that the position of each tile in the target projection coordinate system is accurate. The new geographic transformation parameter calculation formula is as follows: newGeoTransform[0] = lat; newGeoTransform[1] = geoTransform[1]* tileWidth / 256.0; newGeoTransform[3] = lon; newGeoTransform[5] = geoTransform[5]* tileHeight / 256.0; Among them, tileWidth represents the horizontal resolution of a single tile, tileHeight represents the vertical resolution of a single tile, newGeoTransform[0] represents the X coordinate of the upper left pixel of the new tile, newGeoTransform[1] represents the pixel width of the new tile; geoTransform[1] represents the pixel width of the old tile; newGeoTransform[3] represents the Y coordinate of the upper left pixel of the new tile; newGeoTransform[5] represents the pixel height of the new tile, and geoTransform[5] represents the pixel height of the old tile; S307: Setting the projection information of the target geographic image dataset for the tiles to be generated, ensuring that all tiles and the geographic image dataset are in the same projection coordinate system; using the function for setting the projection information to apply the projection information of the geographic image dataset to all target tiles.

[0007] The beneficial effects of the present invention are: (1) The present invention adopts multi-threaded parallel processing technology and fully utilizes the computing power of multi-core processors. Actual measurements show that in a 16-core CPU environment, the processing speed is about 3 to 4 times faster than that of a single-threaded method. Large image data can be effectively processed by reading in blocks. The high-precision coordinate conversion function of the GDAL library is used to significantly improve the positioning accuracy compared to traditional methods. The present invention can automatically adjust the degree of parallelism according to the current hardware configuration and realize accurate progress calculation based on task granularity, with the accuracy of progress display reaching more than 99%; (2) The present invention proposes a high-precision coordinate transformation system that supports complex projection transformations, ensures accurate mapping between different coordinate systems, realizes customized conversion between longitude and latitude and tile coordinates, and improves the efficiency and accuracy of coordinate transformation; at the same time, the present invention adopts an adaptive zoom level calculation method based on map resolution to ensure consistent geographic accuracy at different zoom levels, and realizes the correlation mapping between zoom level and pixel resolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0009] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0010] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. In the following description, many specific details are set forth to facilitate a full understanding of the present invention, but the present invention can also be implemented in other ways different from those described herein, and therefore, the present invention is not limited to the limitations of the specific embodiments disclosed below.

[0011] like Figure 1 As shown, a multi-threaded parallel processing and slice generation method based on geographic image data includes the following steps: S1: Construct a geographic image dataset, obtain the width, height, projection information and geographic transformation parameters of the image in the geographic image dataset, accurately convert the source projection into the target projection coordinate system, and calculate the unit value UPP per pixel. In an embodiment of the present invention, the selected dataset is a geographic image dataset in GeoTIFF format, and the GDALOpen function of the GDAL library is used to open the geographic image dataset in read-only mode to ensure that the format and content of the input file are complete; the GetRasterXSize function and the GetRasterYSize function are used to obtain the width and height of the geographic image respectively, and the geographic transformation parameters of the geographic image are obtained through the GetGeoTransform function. Finally, the projection information of the geographic image is obtained through the GetProjectionRef function, which is used for the consistency maintenance of the projection in the subsequent slice generation process. The target projection coordinate system is the WGS84 coordinate system.

[0012] S2: Set the driver and configure the environment parameters for multi-threaded processing, control the maximum number of threads in the thread pool, and create a thread pool. Specifically, in an embodiment of the present invention, first initialize and register all available GDAL drivers, then configure the environment parameters for multi-threaded processing, and control the maximum number of threads in the thread pool by setting the GDAL_NUM_THREADS parameter in the CPLSetConfigOption function. At the same time, the default configuration is to use all available CPU cores to improve the efficiency of parallel processing. Finally, use the QThreadPool class to build a thread pool, and call the setMaxThreadCount function to set the maximum number of concurrent threads. The setting of the setMaxThreadCount function is consistent with the GDAL_NUM_THREADS parameter.

[0013] S3: Processing the geographic image in the thread pool to generate tiles; when generating tiles, it is necessary to calculate the corresponding area of ​​the current tile in the original image, read the data of the corresponding area from the original image, and resample. The resampling process is based on the per-pixel unit value UPP of the zoom level of different standard maps to ensure the accuracy of the generated tiles in the geographic location; the specific steps are: S301: Calculate the number of tiles in each layer and the geographic coordinate range corresponding to each tile according to the minimum zoom level and maximum zoom level input by the user and the pixel resolution of the geographic image data. The specific calculation formula is: hNo=(lon+180.0)×scale / 360.0; vNo= (90.0+lat)×scale / 360.0; Among them, hNo represents the horizontal number of the tile, vNo represents the vertical number of the tile, lon represents the latitude of the upper left corner of the tile, lat represents the longitude of the upper left corner of the tile, and scale represents the scaling ratio.

[0014] S302: Generate tiles through the thread pool and process tile slices using multithreading; when each tile is generated, first obtain a semaphore that controls the multithreading to control the number of concurrent operations. In the embodiment of the present invention, the semaphore is obtained by calling the acquire function.

[0015] S303: In the specific slicing method, a mutex lock QMutexLocker is used to protect shared resources. Within the scope of the mutex lock QMutexLocker, any thread that attempts to obtain the same mutex lock will be blocked until the current thread completes the release of the mutex lock. The mutex lock QMutexLocker effectively implements exclusive access to shared resources and avoids data competition problems caused by multiple threads modifying shared resources at the same time.

[0016] S304: Read the geographic information data such as the latitude and longitude of the corresponding area of ​​the slice in the source data set, and write the read data into the raster data buffer of the target tile; wherein the corresponding area is based on the per-pixel unit value UPP at each zoom level, and the read area is dynamically adjusted through the zoom adjustment mechanism to ensure consistent geographic accuracy at each zoom level; after the reading is completed, close the source data set and release resources. In an embodiment of the present invention, the RasterIO function of the GDAL library is used to read the specified area of ​​the source data set, and the read data is written into the raster data buffer of the target tile. After the reading is completed, the GDALClose function is used to close the target data set and release resources.

[0017] S305: Unlock the mutex and release the semaphore, allowing other threads to continue to execute tile generation tasks; use atomic operations to update the number of completed tiles and notify the main thread to update the processing progress. Atomic operations allow threads to safely increase an integer value, ensuring that even in a multi-threaded environment, each increase is based on the currently known number of tiles, thereby avoiding race conditions.

[0018] S306: Set the correct geographic transformation parameters for each generated tile, and adjust the geographic transformation parameters of the target geographic image dataset for the tile to be generated, so that the position of each tile in the target projection coordinate system is accurate. Specifically, the SetGeoTransform function can be used to adjust the geographic transformation parameters of the target dataset. The new geographic transformation parameter calculation formula is as follows: newGeoTransform[0] = lat; newGeoTransform[1] = geoTransform[1]* tileWidth / 256.0; newGeoTransform[3] = lon; newGeoTransform[5] = geoTransform[5]* tileHeight / 256.0; Among them, tileWidth represents the horizontal resolution of a single tile, tileHeight represents the vertical resolution of a single tile, newGeoTransform[0] represents the X coordinate of the upper left pixel of the new tile, also known as the minimum longitude or starting longitude, newGeoTransform[1] represents the pixel width of the new tile; geoTransform[1] represents the pixel width of the old tile; newGeoTransform[3] represents the Y coordinate of the upper left pixel of the new tile, also known as the maximum latitude or starting latitude; newGeoTransform[5] represents the pixel height of the new tile, geoTransform[5] represents the pixel height of the old tile. The pixel heights of the old and new tiles are usually negative values ​​because the values ​​on the Y axis increase downward in the geographic coordinate system.

[0019] S307: Setting the projection information of the target geographic image dataset for the tiles to be generated, ensuring that all tiles and the geographic image dataset are in the same projection coordinate system; using a function for setting projection information to apply the projection information of the geographic image dataset to all target tiles. In an embodiment of the present invention, the function for setting projection information is a SetProjection function.

[0020] S4: The generated tiles are stored in a hierarchical directory structure for easy access and management. In an embodiment of the present invention, the Create function in the GDAL library is used to create an output file in GeoTIFF format, and the tile data is written to the output directory, and the file selection function provided by the QFileDialog class is used to allow the user to specify the output path. The generated tiles are stored in the specified directory according to the standard tile naming rules (z / x / y.tif), that is, a hierarchical directory structure is used to store tiles according to the triple structure of zoom level / column number / row number, which is convenient for subsequent quick access and management.

[0021] The experimental statistical results of the embodiment of the present invention are shown in Table 1: Table 1 Experimental statistical results of the embodiments of the present invention The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A multi-threaded parallel processing and slice generation method based on geographic image data, characterized in that: The steps include: S1: Construct a geographic image dataset, obtain the width, height, projection information and geographic transformation parameters of the images in the geographic image dataset, accurately transform the source projection into the target projection coordinate system, and calculate the unit value UPP per pixel; S2: Set the driver and configure the environment parameters for multi-threaded processing, control the maximum number of threads in the thread pool, and create a thread pool; S3: Processing the geographic image in the thread pool to generate tiles; when generating tiles, it is necessary to calculate the corresponding area of ​​the current tile in the original image, read the data of the corresponding area from the original image, and resample, and the resampling process is based on the per-pixel unit value UPP of the zoom level of different standard maps to ensure the accuracy of the generated tiles in terms of geographical location; S4: Store the generated tiles in a hierarchical directory structure for easy access and management.

2. The multi-threaded parallel processing and slice generation method based on geographic image data according to claim 1 is characterized in that: The specific steps of step S3 are: S301: Calculate the number of tiles in each layer and the geographic coordinate range corresponding to each tile according to the minimum zoom level and maximum zoom level input by the user and the pixel resolution of the geographic image data. The specific calculation formula is: hNo=(lon+180.0)×scale / 360.0; vNo= (90.0+lat)×scale / 360.0; Among them, hNo represents the horizontal number of the tile, vNo represents the vertical number of the tile, lon represents the latitude of the upper left corner of the tile, lat represents the longitude of the upper left corner of the tile, and scale represents the scaling ratio; S302: Generate tiles through the thread pool and process tile slices using multi-threading; when each tile is generated, first obtain the semaphore that controls the multi-threading to control the number of concurrency; S303: In the specific slicing method, a mutex lock QMutexLocker is used to protect shared resources. Within the scope of the mutex lock QMutexLocker, any thread that attempts to acquire the same mutex lock will be blocked until the current thread completes the release of the mutex lock; S304: Read the geographic information data such as the latitude and longitude of the corresponding area of ​​the slice in the source dataset, and write the read data into the raster data buffer of the target tile; the corresponding area is based on the per-pixel unit value UPP of each zoom level, and the read area is dynamically adjusted through the zoom adjustment mechanism to ensure consistent geographic accuracy at each zoom level; after the reading is completed, close the source dataset and release the resources; S305: Unlock the mutex and release the semaphore to allow other threads to continue to execute the tile generation task; use atomic operations to update the number of completed tiles and notify the main thread to update the processing progress; S306: Set the correct geographic transformation parameters for each generated tile, and adjust the geographic transformation parameters of the target geographic image data set for the tile to be generated, so that the position of each tile in the target projection coordinate system is accurate. The new geographic transformation parameter calculation formula is as follows: newGeoTransform[0] = lat; newGeoTransform[1] = geoTransform[1] * tileWidth / 256.0; newGeoTransform[3] = lon; newGeoTransform[5] = geoTransform[5] * tileHeight / 256.0; Among them, tileWidth represents the horizontal resolution of a single tile, tileHeight represents the vertical resolution of a single tile, newGeoTransform[0] represents the X coordinate of the upper left pixel of the new tile, newGeoTransform[1] represents the pixel width of the new tile; geoTransform[1] represents the pixel width of the old tile; newGeoTransform[3] represents the Y coordinate of the upper left pixel of the new tile; newGeoTransform[5] represents the pixel height of the new tile, and geoTransform[5] represents the pixel height of the old tile; S307: Setting the projection information of the target geographic image dataset for the tiles to be generated, ensuring that all tiles and the geographic image dataset are in the same projection coordinate system; using the function for setting the projection information to apply the projection information of the geographic image dataset to all target tiles.

Images