Big data remote sensing image efficient slicing method and device and medium

Abstract

The invention discloses a big data remote sensing image efficient slicing method and device and a medium, and the method comprises the steps: employing gdal to preprocess large-scale remote sensing original image data, generating a slicing metadata projection scheme and a slicing task list, and according to the projection scheme and the slicing task list, carrying out the efficient slicing of the large-scale remote sensing original image data; and a multi-server task cluster or a single server adopts a libvips graph cutting function to perform original graph slicing to obtain a bottommost layer tile, resampling is performed to generate each level of upper layer tile, and the hierarchical tiles are packaged, converged and stored according to a star map tile directory rule. According to the method, a task cluster framework and an efficient image processing libvips dynamic link library are introduced, cluster distributed slicing is supported, the concurrent number is not limited by the number of cpu cores of a single machine any more, and the large-scale remote sensing image tiling efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of big data remote sensing image slicing, and in particular to an efficient slicing method, device and medium for big data remote sensing images. Background Art

[0002] With the development of high-tech and the continuous iteration and update of satellite remote sensing image technology, the resolution of remote sensing images obtained by satellites is getting higher and higher, and at the same time, the amount of remote sensing image data is also increasing continuously, from the initial MB level to the current GB level. For this level of data volume, the traditional remote sensing image slicing technology gradually exposes the disadvantages of being unable to efficiently process large-scale image data and having low slicing efficiency.

[0003] Remote sensing image tiling is the main distribution form of current remote sensing image webgis services. Remote sensing image tiling refers to dividing a complete image into multiple small images of the same specification (such as png, webp, etc.) according to a certain zoom level. The user client only needs to load the tiles within the current view range, without loading the entire image. The sliced tiles can be loaded in parallel, reducing the waiting time and improving the response speed. Compared with directly transmitting a whole large remote sensing image over the network, it is often more efficient.

[0004] In the current situation of a vast amount of remote sensing image data, to quickly publish image services, an efficient slicing technology is required. The original remote sensing image slicing technology has the following problems: (1) The traditional tile data organizational structure is stored in the form of.. / z / x / y.png. After dealing with large-scale remote sensing image tiling, there are problems in aspects such as a large number of tile directories, data organization management, update, and data migration. (2) Traditional slicing technologies generally use the gdal slicing technology. When dealing with current large-scale remote sensing images, it has a long slicing time, occupies a large amount of memory, and is limited to running on a single computer or server. The concurrency quantity is limited by the number of cpu cores of a single machine.

[0005] For example, the invention application with the application number 202411178228.X discloses a slicing processing method, device, equipment and storage medium for remote sensing images. Through the solution of this application, the convenience of task division and multi-task processing and the scalability of the system are improved. However, its solution also has the following problems: there is no dask cluster framework, the concurrency quantity is limited; an ordinary slicing function is adopted, and the performance is limited; the storage rule is not conducive to processing fragmented files, etc.

[0006] Therefore, the current remote sensing image tile splitting algorithm is still difficult to meet the rapid processing requirements of large-scale remote sensing image data in practical applications. In reality, an efficient large-scale image slicing technology is needed to improve the efficiency of large-scale remote sensing image tiling. Summary of the Invention

[0007] In view of the above problems, the purpose of the present invention is to provide an efficient slicing method, device and medium for big data remote sensing images, introducing the dask cluster framework and the efficient image processing libvips dynamic link library to improve the efficiency of large-scale remote sensing image tiling.

[0008] An embodiment of the present invention provides an efficient slicing method, device and medium for big data remote sensing images.

[0009] First aspect: An efficient slicing method for big data remote sensing images, including:

[0010] S1. Preprocess the large-scale remote sensing original image data using gdal to generate slice metadata, projection schemes and slice task lists;

[0011] S2. According to the projection scheme and slice task list, use the libvips image cutting function on a multi-server dask cluster or a single server / computer to perform original image slicing to obtain the bottom-layer tiles;

[0012] S3. Generate upper-layer tiles at all levels by stitching and resampling the four lower-layer tiles corresponding to the upper-layer tiles;

[0013] S4. Package and converge the storage of tiles at each level according to the star map tile directory rules.

[0014] Optionally, for the multi-server dask cluster, an nfs mount disk is centrally configured, each server serves as a sub-node, and each sub-node server can access the large-scale remote sensing image data on the nfs mount disk to perform dask cluster distributed slicing processing on the large-scale remote sensing image data.

[0015] Optionally, the preprocessing of the large-scale remote sensing data in S1 includes:

[0016] Read the remote sensing image, use the gdal script, calculate the row and column numbers of the tiles included in each level and various slice parameters according to the input slicing levels, generate a tile task list, write the projection conversion and band nodata into the VRT temporary file in memory, and generate a local tif temporary file.

[0017] Optionally, the step of using the libvips image cutting function to perform original image slicing to obtain the bottom-layer tiles includes:

[0018] S21. Calculate the actual size of the tile relative to the original image resolution and convert it to pixel units;

[0019] S22. Calculate the offset of the upper left corner of the tile relative to the upper left corner of the original image and convert it to pixel units;

[0020] S23. Calculate the range of the original image part sliced into the tile within the tile range and convert it to pixel units;

[0021] S24. Generate a custom - sized tile with blank pixels, and resample the original image part sliced into the tile into this blank tile.

[0022] Optionally, the slicing parameters include reading parameters (X_off, Y_off, R_xsize, R_ysize) and cropping parameters (W_X_off, W_Y_off, W_xsize, W_ysize), where:

[0023] X_off and Y_off are the offset coordinates of the upper - left corner of the original image relative to the upper - left corner of the tile;

[0024] R_xsize and R_ysize are the specifications of the original image sliced into the tile;

[0025] W_X_off and W_Y_off are the starting write positions of the original image part sliced into the tile within the tile;

[0026] W_xsize and W_ysize are the write specifications of the original image within the tile range.

[0027] Optionally: when the four parameters x_off, y_off, tiles_width, and tiles_height are known and the bottom - most tile is a full - frame tile, the formula for obtaining the slicing parameters is:

[0028] W_X_off = 0;

[0029] W_Y_off = 0;

[0030] X_off = int((maxX - geo[0]) / x_res);

[0031] Y_off = int((maxY - geo[3]) / y_res);

[0032] Tiles_width = W_xsize = R_xsize = math.ceil((x - maxX) / x_res);

[0033] Tiles_height = W_ysize = R_ysize = math.ceil((y - maxX) / y_res);

[0034] When the bottom - most tile is a tile that exceeds the left - boundary range of the original image, the formula for obtaining the slicing parameters is:

[0035] W_X_off = int(tiles_width * (abs(X_off) / tiles_width));

[0036] W_Y_off = int(tiles_height * (abs(Y_off) / tiles_height));

[0037] R_xsize = W_xsize = tiles_width - W_X_off;

[0038] R_ysize = W_ysize = tiles_height - W_Y_off;

[0039] The bottom - most tile is the tile that exceeds the upper - boundary range of the original image. The formula for obtaining the segmentation parameters is:

[0040] X_off = int((maxX - geo[0]) / x_res), Y_off = int((maxY - geo[3]) / y_res).

[0041] R_xsize = tiles_width, R_ysize = tiles_height - Y_off.

[0042] W_X_off = 0, W_Y_off = R_ysize.

[0043] W_xsize = R_ysize, W_ysize = R_ysize;

[0044] The bottom - most tile is the tile that exceeds the lower - boundary range of the original image. The formula for obtaining the segmentation parameters is:

[0045] X_off = int((maxX - geo[0]) / x_res), Y_off = int((maxY - geo[3]) / y_res).

[0046] R_xsize = tiles_width, R_ysize = height - Y_off.

[0047] W_X_off = 0, W_Y_off = 0.

[0048] W_xsize = R_ysize, W_ysize = R_ysize.

[0049] The bottom - most tile is the tile that exceeds the right - boundary range of the original image. The formula for obtaining the segmentation parameters is:

[0050] X_off = int((maxX - geo[0]) / x_res), Y_off = int((maxY - geo[3]) / y_res).

[0051] R_xsize = width – X_off, R_ysize = tiles_height – Y_off.

[0052] W_X_off = 0, W_Y_off = Y_off.

[0053] W_xsize = width – X_off, W_ysize = tiles_height – Y_off;

[0054] Among them, maxX, maxY, minx, minY are the four boundaries of the tile, x_res is the width of the tile, y_res is the height of the tile, and geos[0] to geos[5] are the 6-element group of the original image data projection affine matrix geo.

[0055] Optionally:

[0056] In the low zoom level, one tile completely contains the original image, and the formula for obtaining the splitting parameters is:

[0057] X_off = 0, Y_off = 0.

[0058] R_xsize = width, R_ysize = height.

[0059] W_X_off = int((maxX - geo[0]) / x_res), W_Y_off = int((maxY - geo[3]) / y_res).

[0060] W_xsize = R_xsize, W_ysize = R_ysize;

[0061] Among them, maxX, maxY, minx, minY are the four boundaries of the tile, x_res is the width of the tile, y_res is the height of the tile, and geos[0] to geos[5] are the 6-element group of the original image data projection affine matrix geo.

[0062] Optionally, the

[0063] Star map tile directory rule is: at most 1024 (32 * 32) tiles (or folders) are stored in one folder. If more than 32 tiles are browsed continuously horizontally or vertically, different folders will be retrieved;

[0064] The packaging and aggregation rule is as follows: maintain the directory structure of the star map tiles, and use zip compression to package the bottom - most tile directory.

[0065] In a second aspect: an electronic device, including a memory, a processor, and a computer program stored on the memory and operable on the processor, when the processor executes the program, it implements the steps of the method provided in the first aspect.

[0066] In a third aspect: a non - transitory computer - readable storage medium, on which a computer program is stored, when the computer program is executed by a processor, it implements the steps of the method provided in the first aspect.

[0067] Advantages of the present invention:

[0068] 1. For the high - efficiency slicing method of remote - sensing images of the present invention, on the basis of the original slicing process scheme, the dask cluster framework and the efficient image - processing libvips dynamic link library are introduced, replacing the traditional gdal function slicing technology. The efficiency can be increased by 2 times or more. By applying the dask cluster framework, it supports cluster - distributed slicing, and the number of concurrent operations is no longer limited by the number of CPU cores of a single machine, improving the efficiency of large - scale remote - sensing image tiling.

[0069] 2. The present invention consists of multiple servers to form a dask cluster, configured with an nfs mount disk. The nfs mount disk is used for storing large - scale remote - sensing images and outputting tiles. Each sub - node can access the data on the nfs mount disk. When processing large - scale remote - sensing images, it avoids the time overhead of the master node distributing a large amount of data to the sub - nodes. At the same time, through the heartbeat mechanism, retry mechanism, and task re - distribution mechanism of the dask cluster, it ensures that once an exception occurs in the sub - node task, it will be retried. After a certain number of retries, the sub - node task will be distributed to other healthy sub - nodes for execution, improving the reliability of data slicing.

[0070] 3. The present invention couples the efficient image - processing library libvips, so compared with the traditional gdal image - cutting function, the efficiency is increased several times. Especially based on the unique multi - threaded io system of libvips, the effect of multi - process processing is particularly obvious.

[0071] 4. The present invention adopts the self - created star - map tile organization rule, effectively solving the problems in aspects such as the large number of tile directories, data organization management, update, and data migration after large - scale remote - sensing image tiling. Description of the Drawings

[0072] Figure 1 It is the flow structure diagram of a high - efficiency slicing method for big - data remote - sensing images of the present invention;

[0073] Figure 2Structure diagram of software and hardware for the operation of an efficient big data image slicing method according to the present invention;

[0074] Figure 3 Interface diagram of a dask cluster sub-node according to the present invention;

[0075] Figure 4 Preprocessing flow chart of large-scale remote sensing original image data according to the present invention;

[0076] Figure 5 Schematic diagram of the slicing parameter structure according to the present invention;

[0077] Figure 6 Schematic diagram of the original image slicing structure according to the present invention;

[0078] Figure 7 According to the present invention Figure 6 Schematic diagram for calculating the slicing parameters of (0, 0);

[0079] Figure 8 According to the present invention Figure 6 Schematic diagram for calculating the slicing parameters of (1, 0);

[0080] Figure 9 According to the present invention Figure 6 Schematic diagram for calculating the slicing parameters of (2, 0);

[0081] Figure 10 According to the present invention Figure 6 Schematic diagram for calculating the slicing parameters of (0, 1);

[0082] Figure 11 According to the present invention Figure 6 Schematic diagram for calculating the slicing parameters of (2, 1);

[0083] Figure 12 According to the present invention Figure 6 Schematic diagram for calculating the slicing parameters of (0, 2);

[0084] Figure 13 According to the present invention Figure 6 Schematic diagram for calculating the slicing parameters of (1, 2);

[0085] Figure 14 According to the present invention Figure 6 Schematic diagram for calculating the slicing parameters of (2, 2);

[0086] Figure 15 Schematic diagram for calculating the original image slicing parameters included in the low-level scaled tiles according to the present invention;

[0087] Figure 16 Schematic diagram of the traditional tile directory structure;

[0088] Figure 17 Schematic diagram of the tile directory division according to the present invention;

[0089] Figure 18 Schematic diagram of tile data packing and aggregation for the present invention;

[0090] Figure 19 Schematic diagram of the structure of the electronic device of the present invention. Detailed implementation manners

[0091] The embodiments of the present invention will be described in detail below. Examples of the embodiments are shown in the accompanying drawings, where the same or similar symbols represent the same or similar elements or elements with the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as a limitation to the present invention.

[0092] The original remote sensing image slices generally only use the gdal slicing technology. When dealing with large-scale remote sensing images, there are problems such as long slicing time, large memory occupation, and it is only limited to running on a single computer or server. The concurrency number is limited by the number of cpu cores of a single machine. Moreover, after large-scale remote sensing images are tiled, there are problems in aspects such as a large number of tile directories, organization management, update, and data migration of data.

[0093] In view of the above problems, the present invention provides an efficient slicing method for big data remote sensing images. Figure 1 Schematic diagram of the process of the efficient slicing method for big data remote sensing images provided by the embodiment of the present invention. The method includes:

[0094] S1. Preprocess large-scale remote sensing original image data using gdal to generate a slicing metadata projection scheme and a slicing task list.

[0095] As Figure 2 shown, the hardware of the present invention uses multiple servers or a single machine service. When using multiple servers, a dask cluster is deployed, and an nfs mount disk is centrally configured. The nfs mount disk is used for storing large-scale remote sensing images and outputting tiles. As Figure 3 shown, multiple servers are used as sub-nodes, and each sub-node server can access the large-scale remote sensing image data on the nfs mount disk to perform dask cluster distributed slicing processing on the large-scale remote sensing image data, avoiding the time overhead of the master node distributing data to the sub-nodes. With the above structure, the slicing script can cooperate with the multi-server dask cluster framework for distributed slicing, or can be sliced on a single workstation / pc.

[0096] The map slicing script processes large-scale remote sensing data through gdal, performs spatial reference conversion, and generates a slicing metadata projection scheme and a slicing task list.

[0097] Specifically, as Figure 4As shown in the figure, the open-source remote sensing image processing library GDAL is used to read remote sensing images as the cut image program script. The row and column numbers of the tiles and various slicing parameters included in each level are calculated according to the input cut levels, a tile task list is generated, and the projection conversion operation and band nodata are written into the VRT string in memory. Corresponding projection conversions are performed according to the projection coordinate system of the input remote sensing image combined with the spatial reference of the remote sensing image, and local tif temporary files are generated.

[0098] S2. According to the projection scheme and the slice task list, the multi-server Dask cluster or a single server uses the libvips image cutting function to slice the original image to obtain the bottom-layer tiles.

[0099] As Figure 5 shown, to correctly slice remote sensing images, 8 slicing parameters for reading parameters and cropping parameters need to be set. Among them:

[0100] X_off and Y_off are the offset coordinates of the upper left corner of the original image relative to the upper left corner of the tile;

[0101] R_xsize and R_ysize are the specifications of the original image sliced to the tile;

[0102] W_X_off and W_Y_off are the starting positions for writing the part of the original image sliced to the tile within the tile;

[0103] W_xsize and W_ysize are the writing specifications of the original image within the range of the tile.

[0104] According to the projection coordinate system of the input remote sensing image (such as 3857) combined with the spatial reference of the remote sensing image, corresponding projection conversions are performed. The projection affine matrix is geo, and geo is a tuple of 6 elements. The attributes represented by the 6 elements are shown in Table 1:

[0105] Table 1 Affine transformation parameters

[0106] geos[0] The x - coordinate of the upper - left corner geos[1] The pixel resolution in the east - west direction geos[2] The rotation angle, which is 0 when due north is upward geos[3] The y - coordinate of the upper - left corner geos[4] The rotation angle, which is 0 when due north is upward geos[5] The pixel resolution in the north - south direction

[0107] When using the libvips image cutting function to slice the original image, as Figure 6 shown, the original image to be sliced is in the middle, and the box with coordinates is the tile. It can be found that there are the following situations for the sliced tiles:

[0108] Full-frame tiles, such as Figure 6 the tile numbered (1, 1) in the figure.

[0109] The left and right boundaries of the tile exceed the range of the original image, such as Figure 6 the tiles numbered (0, 0), (1, 0), (2, 0), (0, 2), (1, 2), (2, 2) in the figure.

[0110] The upper and lower boundaries of the tile exceed the range of the original image, such as Figure 6 the tiles numbered (0, 1) and (2, 1) in

[0111] In the low zoom level, one tile completely contains the original image, as shown in Figure 15 shown in

[0112] In actual production, the size of the tile results is generally fixed, such as 256 pixels in both width and height. However, the tile range actually used for slicing the original image is not always 256 * 256 pixels. Therefore, the sliced part of the original image must be resampled through an algorithm and stuffed into the 256 * 256 result tiles. As shown in Figure 6 the tiles in

[0113] First, calculate the true size (in meters) of the tile relative to the resolution of the original image and convert it to pixel units;

[0114] Then, calculate the offset of the upper left corner of the tile relative to the upper left corner of the original image and convert it to pixel units;

[0115] Then, calculate the range of the sliced part of the original image within the tile range and convert it to pixel units;

[0116] Finally, generate a blank tile of 256 * 256 pixels and resample the sliced part of the original image into this blank tile.

[0117] Specifically, through the tile (y, x) and the zoom level z, the size of the tile in the projection coordinate system 3857 projection can be calculated. Assume the calculated four - corner ranges are maxX, maxY, minx, and minY.

[0118] For full - size tiles, such as Figure 6 the tile numbered (1, 1) in

[0119] X_off = int((maxX - geo[0]) / x_res)

[0120] Y_off = int((maxY - geo[3]) / y_res)

[0121] Then, use the formula to convert and calculate the size of the tile relative to the original image pixels;

[0122] Tiles_width = W_xsize = R_xsize = math.ceil((x - maxX) / x_res)

[0123] Tiles_height = W_ysize = R_ysize = math.ceil((y - maxX) / y_res)

[0124] Obtain the parameters for cutting the full - scale tiles through the above calculations Figure Eight parameters.

[0125] For tiles whose left boundary exceeds the range of the original image, such as Figure 6 tiles (0, 0), (1, 0), (2, 0); still first use the formula to calculate X_off, y_off, tiles_width, and tiles_height. At this time;

[0126] W_X_off = int(tiles_width * (abs(X_off) / tiles_width))

[0127] W_Y_off = int(tiles_height * (abs(y_off) / tiles_height))

[0128] R_xsize = W_xsize = tiles_width - W_X_off

[0129] R_ysize = W_ysize = tiles_height - W_Y_off

[0130] For tile (0, 0), as Figure 7 shown, X_off and y_off are reset to 0, that is, X_off = y_off = 0. Since the upper - left corner of the original image is completely included in this tile, when reading data, it should be read from the upper - left starting point (0, 0).

[0131] For tile (1, 0), as Figure 8 shown:

[0132] Read the parameters of the original image:

[0133] Starting point coordinates: X_off = 0, y_off = int((maxY - geo[3]) / y_res).

[0134] Reading range: R_ysize = tiles_height, R_xsize = tiles_width - X_off.

[0135] Writing tile parameters:

[0136] Starting point coordinates: W_Y_off = 0, W_X_off = int((maxX - geo[0]) / x_res).

[0137] Writing range: W_ysize = tiles_height, W_xsize = tiles_width – W_X_off.

[0138] For tile (2, 0), as Figure 9 shown:

[0139] Reading original image parameters:

[0140] Starting point coordinates: X_off = 0, y_off = int((maxY - geo[3]) / y_res).

[0141] Reading range: R_ysize = height – y_off, R_xsize = tiles_width - X_off.

[0142] Writing tile parameters:

[0143] Writing starting point coordinates: W_Y_off = 0, W_X_off = int((maxX - geo[0]) / x_res).

[0144] Writing range: W_ysize = tiles_height – y_off, W_xsize = tiles_width - X_off.

[0145] For the case where the upper boundary of the tile exceeds the upper boundary range of the original image, such as Figure 6 tile (1, 0):

[0146] Reading original image parameters:

[0147] Reading starting coordinates:

[0148] X_off = int((maxX - geo[0]) / x_res), Y_off = int((maxY - geo[3]) / y_res).

[0149] Reading range: R_xsize = tiles_width, R_ysize = tiles_height – y_off.

[0150] Tile writing parameters:

[0151] Writing starting point coordinates: W_X_off = 0, W_Y_off = R_ysize.

[0152] Writing range: W_xsize = R_ysize, W_ysize = R_ysize, as Figure 10 shown.

[0153] For the case where the lower boundary of the tile exceeds the lower boundary of the original image, such as Figure 6 tile (2, 1):

[0154] Original image reading parameters:

[0155] Reading starting coordinates:

[0156] X_off = int((maxX - geo[0]) / x_res), Y_off = int((maxY - geo[3]) / y_res).

[0157] Reading range: R_xsize = tiles_width, R_ysize = height – y_off.

[0158] Tile writing parameters:

[0159] Writing starting point coordinates: W_X_off = 0, W_Y_off = 0.

[0160] Writing range: W_xsize = R_ysize, W_ysize = R_ysize, as Figure 11 shown.

[0161] For the case where the right boundary of the tile exceeds the right boundary of the original image, such as Figure 6 tiles (0, 2), (1, 2), (2, 2):

[0162] Still first use the formula to calculate X_off, y_off, R_xsize, R_ysize. Parse height and width from the original image.

[0163] For tile numbered (0, 2):

[0164] Original image reading parameters:

[0165] Reading starting coordinates:

[0166] X_off = int((maxX - geo[0]) / x_res), Y_off = int((maxY - geo[3]) / y_res).

[0167] Reading range: R_xsize = width – X_off, R_ysize = tiles_height – y_off.

[0168] Tile writing parameters:

[0169] Writing starting point coordinates: W_X_off = 0, W_Y_off = y_off.

[0170] Writing range: W_xsize = width – X_off, W_ysize = tiles_height – y_off. As Figure 12 shown.

[0171] For tile number (1, 2):

[0172] Original image reading parameters:

[0173] Reading starting coordinates:

[0174] X_off = int((maxX - geo[0]) / x_res), Y_off = int((maxY - geo[3]) / y_res).

[0175] Reading range: R_xsize = width – X_off, R_ysize = tiles_height.

[0176] Tile writing parameters:

[0177] Writing starting point coordinates: W_X_off = 0, W_Y_off = 0.

[0178] Writing range: W_xsize = width – X_off, W_ysize = tiles_height, as Figure 13 shown. For tile number (2, 2):

[0179] Original image reading parameters:

[0180] Reading starting coordinates:

[0181] X_off = int((maxX - geo[0]) / x_res), Y_off = int((maxY - geo[3]) / y_res).

[0182] Reading range: R_xsize = width – X_off, R_ysize = height - y_off.

[0183] Tile writing parameters:

[0184] Writing start point coordinates: W_X_off = 0, W_Y_off = 0.

[0185] Writing range: W_xsize = R_xsize, W_ysize = R_ysize, as Figure 14 shown.

[0186] In the low zoom level, one tile completely contains the original image, as Figure 15 shown:

[0187] Reading original image parameters:

[0188] Reading start coordinates: X_off = 0, Y_off = 0.

[0189] Reading range: R_xsize = width, R_ysize = height.

[0190] Tile writing parameters:

[0191] Writing start point coordinates:

[0192] W_X_off = int((maxX - geo[0]) / x_res), W_Y_off = int((maxY - geo[3]) / y_res). Writing range: W_xsize = R_xsize, W_ysize = R_ysize.

[0193] By using the above method, all tile slice parameters can be obtained. The tile row and column numbers are used as keys, and the slice parameters are used as values to be encapsulated into a dictionary object for subsequent image slicing programs to perform image slicing.

[0194] Specifically, when using the slicing function provided in the open-source libvips dynamic link library for slicing, the steps include:

[0195] 1. Create a blank tile tiles_new = pyvips.Image.black(256, 256).

[0196] 2. Read the temporarily generated source image file after projection,

[0197] input_image = pyvips.Image.new_from_file(file path).

[0198] 3. Crop the original image with the read original image parameters,

[0199] tiles = input_image.crop(X_off, y_off, R_xsize, R_ysize).

[0200] 4. Resampling of the original image data after cropping: scaled_image = input_image.resize(horizontal scaling ratio, vscale = vertical scaling ratio), where the horizontal scaling ratio = R_xsize / Tiles_width (tile width), and the vertical scaling ratio vscale = R_ysize / Tiles_height (tile height).

[0201] 5. Extracting the data mask after resampling: mask = (scaled_image[1]!= 0).

[0202] 6. Embedding the scaled image into a blank tile,

[0203] out_tiles = tiles_new.insert(scaled_image, W_X_off, W_Y_off);

[0204] 7. Constructing the alpha channel: alpha = tiles_new.insert(mask, W_X_off, W_Y_off).

[0205] 8. Adding the alpha channel to the resulting tile: out_tiles = out_tiles.bandjoin(alpha)

[0206] Writing the tile to disk: out_tiles.write_to_file(output path);

[0207] S3. Generating upper-level tiles at all levels by stitching and resampling the four lower-level tiles corresponding to the upper-level tile.

[0208] When upper-level tiles need to be generated, since the lower-level tiles are all generated by dividing the upper-level tiles into four equal parts, the lower-level tiles can be retrieved by the tile numbers after dividing the upper-level tiles into four equal parts, and the four corresponding lower-level tiles are merged into the upper-level tile using the image stitching interface provided by the open-source libvip.

[0209] S4. Packing and aggregating and storing the tiles at each level according to the star map tile directory rules.

[0210] The traditional tile organization directory is as Figure 16 shown, with each zoom level as the parent directory, the subdirectory as the tile row number (or column number), and the tile file name as the column number (or row number).

[0211] Using the traditional tile-based directory organization, when dealing with a large number of small tiles, this directory organization form will cause waste of disk storage clusters. If the size of a tile is 8 kb and each disk cluster is 4 kb, then three clusters are required for storage. During the data migration process, a large number of small files will also take a very long time.

[0212] The star map tile storage form of the present invention is as Figure 17 shown. At most 1024 (32 * 32) tiles (or folders) are stored in one folder. In the most cases, when browsing 32 + 1 tiles continuously horizontally or vertically, different folders will be retrieved.

[0213] Taking level 18 as an example, the specific solution is as follows: First, taking 2 15 (32768) as the standard, if the row and column numbers are greater than this value, they will be stored in different folders. Calculate the name of the outermost folder intnRow_1 = ix / 2 15 , intnCol_1 = iy / 2 15 . The maximum value of nRow_1 is 8, and the maximum value of nCol_1 is 4. Then introduce the concept of intermediate row and column numbers, and calculate the names of each layer of directories respectively. The intermediate row and column numbers are the relatively unique determined codes of the current tile under the current layer block. Taking level 18 as an example, based on the nRow_1 and nCol_1 obtained in the first step, taking 2 10 as the standard, calculate nRow_2 = middle_ix / 2 10 and nCol_2 = middle_iy / 2 10 . Finally, taking 2 5 as the standard, calculate nRow_3 and nCol_3. At most 1024 tiles are stored in the folder at this level.

[0214] Using the above method, the obtained tile directory structure is:

[0215] 18 / nRow_1nCol_1 / nRow_2nCol_2 / nRow_3nCol_3 / name.suffix

[0216] In the actual data organization and management, a large number of small tile files will be generated at high levels. Excessive small fragmented files will seriously occupy system resources and affect computer performance. For subsequent operations and processing of tiles (such as tile fusion, tile color adjustment, tile migration), etc., so the tiles need to be correspondingly packed and aggregated to facilitate the overall data management work.

[0217] The rules for packing and aggregating are to maintain the star map encoding directory organizational structure, use zip to compress and pack the bottom layer tile directory, and store it as ***.zip. The obtained directory tree is as Figure 18 shown in the example.

[0218] The present invention also provides an electronic device, Figure 19 which is a schematic structural diagram of the electronic device provided by an embodiment of the present invention. As Figure 19 shown, the electronic device may include: a processor, a communications interface, a memory, and a communication bus. Among them, the processor, the communication interface, and the memory complete mutual communication through the communication bus. The processor can call logical instructions in the memory to execute the following methods, for example:

[0219] S1. Use gdal to preprocess large-scale remote sensing original image data to generate tile metadata, projection schemes, and tile task lists;

[0220] S2. According to the projection scheme and the tile task list, use the libvips tile cutting function on a multi-server dask cluster or a single server / computer to cut the original image and obtain the bottom-layer tiles;

[0221] S3. Generate upper-layer tiles at all levels by splicing the four lower-layer tiles corresponding to the upper-layer tiles and resampling;

[0222] S4. Package and converge and store the tiles at each level according to the star map tile directory rules.

[0223] In addition, when the logical instructions in the above-mentioned memory are implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical discs that can store program codes.

[0224] An embodiment of the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is implemented to execute the methods provided in the above-mentioned various embodiments, for example, including:

[0225] S1. Use gdal to preprocess large-scale remote sensing original image data to generate tile metadata, projection schemes, and tile task lists;

[0226] S2. According to the projection scheme and the slice task list, the multi-server dask cluster or a single server / computer uses the libvips slicing function to slice the original image and obtain the bottom-layer tiles.

[0227] S3. Generate each level of upper-layer tiles by splicing the four lower-layer tiles corresponding to the upper-layer tiles and resampling.

[0228] S4. Package and converge the storage of the tiles at each level according to the star map tile directory rules.

[0229] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative work.

[0230] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0231] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments or equivalently replace some of the technical features. These modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An efficient slicing method for big data remote sensing images, characterized in that, Including: S1. Preprocess large-scale remote sensing original image data using gdal to generate slice metadata, projection schemes, and slice task lists; S2. According to the projection scheme and slice task list, use the libvips slicing function on a multi-server dask cluster or a single server / computer to slice the original image and obtain the bottom-layer tiles; S3. Generate upper-layer tiles at all levels by stitching the four lower-layer tiles corresponding to the upper-layer tiles and resampling; S4. Package and converge the storage of tiles at each level according to the star map tile directory rules.

2. The efficient slicing method for big data remote sensing images according to claim 1, wherein For the multi-server dask cluster, centrally configure an nfs-mounted disk. Each server serves as a sub-node, and each sub-node server can access the large-scale remote sensing image data on the nfs-mounted disk to perform dask cluster distributed slicing processing on the large-scale remote sensing image data.

3. The efficient slicing method for big data remote sensing images according to claim 1, characterized in that, In the preprocessing of large-scale remote sensing data in S1, it includes: Read the remote sensing image, use the gdal script, calculate the row and column numbers of tiles and various slice parameters included in each level according to the input slice levels, generate a tile task list, write the projection conversion and band nodata into a VRT temporary file in memory, and generate a local tif temporary file.

4. The efficient slicing method for big data remote sensing images according to claim 1, characterized in that, In S2, perform original image slicing to obtain the bottom-layer tiles. The steps include: S21. Calculate the actual size of the tile relative to the original image resolution and convert it to pixel units; S22. Calculate the offset of the upper left corner of the tile relative to the upper left corner of the original image and convert it to pixel units; S23. Calculate the range of the part of the original image sliced by the tile within the tile range and convert it to pixel units; S24. Generate a custom-sized tile with blank pixels, and resample the part of the original image sliced by the tile into this blank tile.

5. The efficient slicing method for big data remote sensing images according to claim 4, characterized in that The slice parameters include reading parameters (X_off, Y_off, and R_xsize, R_ysize) and cropping parameters (W_X_off, W_Y_off, and W_xsize, W_ysize), where: X_off and Y_off are the offset coordinates of the upper left corner of the original image relative to the upper left corner of the tile; R_xsize and R_ysize are the specifications of the original image sliced by the tile; W_X_off and W_Y_off are the starting positions for writing the part of the original image sliced by the tile within the tile; W_xsize and W_ysize are the writing specifications of the original image within the tile range.

6. A method for efficient slicing of big data remote sensing images according to claim 5, characterized in that: When the bottom-layer tile is a full-frame tile, the formula for obtaining the slicing parameters is: W_X_off = 0; W_Y_off = 0; X_off = int((maxX - geo[0]) / x_res); Y_off = int((maxY - geo[3]) / y_res); Tiles_width = W_xsize = R_xsize = math.ceil((x - maxX) / x_res); Tiles_height = W_ysize = R_ysize = math.ceil((y - maxX) / y_res); The bottom - most tile is a tile that extends beyond the left - hand boundary of the original image. The formula for obtaining the splitting parameters is as follows: W_X_off = int(tiles_width * (|X_off| / tiles_width)); W_Y_off = int(tiles_height * (|Y_off| / tiles_height)); R_xsize = W_xsize = tiles_width - W_X_off; R_ysize = W_ysize = tiles_height - W_Y_off; The bottom - most tile is a tile that extends beyond the upper - hand boundary of the original image. The formula for obtaining the splitting parameters is as follows: X_off = int((maxX - geo[0]) / x_res), Y_off = int((maxY - geo[3]) / y_res); R_xsize = tiles_width, R_ysize = tiles_height – Y_off; W_X_off = 0, W_Y_off = R_ysize; W_xsize = R_ysize, W_ysize = R_ysize; The bottom - most tile is a tile that extends beyond the lower - hand boundary of the original image. The formula for obtaining the splitting parameters is as follows: X_off = int((maxX - geo[0]) / x_res), Y_off = int((maxY - geo[3]) / y_res); R_xsize = tiles_width, R_ysize = height – Y_off; W_X_off = 0, W_Y_off = 0; W_xsize = R_ysize, W_ysize = R_ysize; The bottom - most tile is a tile that extends beyond the right - hand boundary of the original image. The formula for obtaining the splitting parameters is as follows: X_off = int((maxX - geo[0]) / x_res), Y_off = int((maxY - geo[3]) / y_res); R_xsize = width – X_off, R_ysize = tiles_height – Y_off; W_X_off = 0, W_Y_off = Y_off; W_xsize = width – X_off, W_ysize = tiles_height – Y_off; Where maxX, maxY, minx, minY are the four - corner ranges of the tile, x_res is the width of the tile, y_res is the height of the tile, and geos[0] to geos[5] are the 6 - element group of the original - image data projection affine matrix geo.

7. An efficient slicing method for big - data remote - sensing images according to claim 5, characterized in that: In the low - zoom level, one tile completely contains the original image. The formula for obtaining the splitting parameters is as follows: X_off = 0, Y_off = 0; R_xsize = width, R_ysize = height; W_X_off = int((maxX - geo[0]) / x_res), W_Y_off = int((maxY - geo[3]) / y_res); W_xsize = R_xsize, W_ysize = R_ysize; Among them, maxX, maxY, minx, minY are the four boundaries of the tile, x_res is the width of the tile, y_res is the height of the tile, and geos[0] to geos[5] are the 6 - element group of the original image data projection affine matrix geo.

8. A method for efficiently slicing big data remote sensing images according to claim 1, characterized in that The rules for the star map tile directory are as follows: at most 1024 (32 * 32) tiles or folders are stored in one folder. When continuously browsing more than 32 tiles horizontally or vertically, different folders will be retrieved; The packaging and aggregation rules are: keep the star map tile directory structure and use zip compression to package the bottom - layer tile directory.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of a method for efficiently slicing large - data remote - sensing images as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of a method for efficiently slicing large - data remote - sensing images as described in any one of claims 1 to 8.

Images