Cloud-optimized geological raster data processing method and system

Abstract

The invention discloses a geological raster data processing method and system for cloud optimization. The method comprises the following steps: S1, performing format conversion on original geological raster data to obtain target format raster data with cloud optimization characteristics; s2, carrying out optimization processing before data service release on the converted raster data in the target format, wherein the optimization processing comprises pyramid structure construction and raster data recoding and compression processing; and S3, performing data service release on the optimized raster data in the target format. The invention relates to a raster data conversion, compression and sharing publishing method based on cloud optimization, which improves the data storage and transmission efficiency and reduces the cost on the premise of ensuring the data quality by combining the characteristics of a cloud optimization technology and raster data storage.

Description

Technical Field

[0001] The present invention relates to the technical field of raster data processing, and in particular to a cloud-optimized geological raster data processing method and system. Background Art

[0002] Raster data is a common data type in geography and geology informatization, used to describe surface features, terrain, remote sensing images, etc. The processing and storage of raster data have always been one of the key challenges in the GIS field. Traditional raster data formats have many limitations in storage and transmission. For example, high-resolution images on the Web require the large file to be completely downloaded to the client before it can be analyzed or visualized, resulting in problems such as large data files, low access efficiency, and high storage costs. To solve these problems, an efficient, scalable raster data conversion, compression, and sharing and publishing solution is needed. Summary of the Invention

[0003] This application provides a cloud-optimized geological raster data processing method and system to solve the problems of large file size, slow download speed, and high data transmission cost in the traditional raster data storage method.

[0004] According to a first aspect, in one embodiment, a cloud-optimized geological raster data processing method is provided. The method includes:

[0005] Step S1, converting the original geological raster data into target format raster data with cloud-optimized characteristics;

[0006] Step S2, performing optimization processing on the converted target format raster data before data service publishing, including constructing a pyramid structure, raster data recoding, and compression processing;

[0007] Step S3, performing data service publishing on the optimized target format raster data.

[0008] Further, the step S1 specifically includes:

[0009] Step S11, converting GeoTIFF format raster data into COG format data.

[0010] Further, the step S1 specifically further includes:

[0011] Step S12, adopting a block strategy to ensure the feasibility of data processing, including:

[0012] Step S121, dividing the target format raster data into a grid to obtain a plurality of raster data blocks;

[0013] Step S122: Encode the obtained multiple raster data blocks in sequence to obtain an encoded sequence, and process the raster data blocks successively according to the formed encoded sequence.

[0014] Further, the step S2 specifically includes:

[0015] Step S21: Construct a pyramid structure, including:

[0016] Step S211: Obtain the raster data band information, and define the levels and the lowest resolution of the pyramid;

[0017] Step S212: Downsample the raster data to obtain the next layer of the pyramid;

[0018] Step S213: Repeat the sampling until the specified number of levels or the lowest resolution is reached;

[0019] Step S214: Generate a pyramid structure by arranging the raster data at different levels according to the principle of "placing the larger resolution at the bottom and the smaller resolution at the top".

[0020] Further, the step S2 specifically includes:

[0021] Step S22: Raster data re-encoding and compression processing, including:

[0022] Step S221: Divide the raster data into blocks of equal size;

[0023] Step S222: Convert the raster spatial domain data into frequency domain data;

[0024] Step S223: Quantize the converted frequency domain data;

[0025] Step S224: Encode and compress the quantized data.

[0026] Further, the step S22 specifically further includes:

[0027] Step S225: Decode and inverse-quantize the compressed data, and reconstruct the image data through inverse transformation.

[0028] Further, the step S222 specifically includes:

[0029] Convert the spatial domain data into frequency domain data by using the discrete cosine transform algorithm.

[0030] Further, the step S224 specifically includes:

[0031] Encode and compress the quantized data by using the Huffman entropy coding technique.

[0032] According to a second aspect, in one embodiment, a cloud-optimized geological raster data processing system is provided. The system includes:

[0033] A format conversion module for converting the original geological raster data into target format raster data with cloud-optimized characteristics;

[0034] An optimization processing module for performing optimization processing on the converted target format raster data before data service publishing, including constructing a pyramid structure, raster data re-encoding, and compression processing;

[0035] A service publishing module for publishing data services for the optimized target format raster data.

[0036] According to a third aspect, in one embodiment, an electronic device is provided. The device includes: a processor and a memory;

[0037] The memory is used to store one or more program instructions;

[0038] The processor is used to run one or more program instructions to execute the steps of a cloud-optimized geological raster data processing method as described in any one of the above.

[0039] According to a fourth aspect, in one embodiment, a computer-readable storage medium is provided. A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, it implements the steps of a cloud-optimized geological raster data processing method as described in any one of the above.

[0040] This application provides a cloud-optimized geological raster data processing method and system, including: Step S1, converting the original geological raster data into target format raster data with cloud-optimized characteristics; Step S2, performing optimization processing on the converted target format raster data before data service publishing, including constructing a pyramid structure, raster data re-encoding, and compression processing; Step S3, publishing data services for the optimized target format raster data. The present invention relates to a method for cloud-optimized raster data conversion, compression, and shared publishing, aiming to improve problems existing in traditional raster data storage methods, such as large file size, slow download speed, and high data transmission cost. It has cloud-optimized characteristics, and the final processing result supports high-speed data reading and distributed computing. Compared with traditional GeoTIFF raster data storage and analysis applications, the file size compression rate is below 75%. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of a cloud-optimized geological raster data processing method provided by an embodiment of the present invention;

[0042] Figure 2Example of raster data block division and encoding scheme (5×5) in a cloud-optimized geological raster data processing method provided by an embodiment of the present invention;

[0043] Figure 3 Schematic diagram of raster data block division in a cloud-optimized geological raster data processing method provided by an embodiment of the present invention;

[0044] Figure 4 Schematic diagram of run-length encoding in a cloud-optimized geological raster data processing method provided by an embodiment of the present invention;

[0045] Figure 5 Array matrix reconstructed by run-length encoding in a cloud-optimized geological raster data processing method provided by an embodiment of the present invention;

[0046] Figure 6 Input data of an experimental example in a cloud-optimized geological raster data processing method provided by an embodiment of the present invention;

[0047] Figure 7 Output data of an experimental example in a cloud-optimized geological raster data processing method provided by an embodiment of the present invention;

[0048] Figure 8 Example of a published COG raster data service in a cloud-optimized geological raster data processing method provided by an embodiment of the present invention;

[0049] Figure 9 Schematic diagram of the structure of a cloud-optimized geological raster data processing system provided by an embodiment of the present invention. Detailed implementation manners

[0050] The present invention will be further described in detail below in conjunction with the accompanying drawings through specific implementation manners. Similar elements in different implementation manners are labeled with related similar element numbers. In the following implementation manners, many detailed descriptions are provided to enable a better understanding of the present application. However, those skilled in the art can easily recognize that some of the features can be omitted in different situations, or can be replaced by other elements, materials, or methods. In some cases, some operations related to the present application are not shown or described in the specification to avoid overwhelming the core part of the present application with excessive descriptions. For those skilled in the art, it is not necessary to describe these related operations in detail, and they can fully understand the related operations based on the descriptions in the specification and general technical knowledge in the art.

[0051] In addition, the features, operations, or characteristics described in the specification can be combined in any suitable manner to form various embodiments. At the same time, the steps or actions in the method description can also be reordered or adjusted in a manner obvious to those skilled in the art. Therefore, the various sequences in the specification and drawings are only for clearly describing a certain embodiment and do not mean a necessary sequence, unless it is stated otherwise that a certain sequence must be followed.

[0052] Cloud-Optimized GeoTIFF (COG), as an emerging raster data storage format, aims to overcome the deficiencies of traditional GeoTiff raster data storage and application. It has advantages such as supporting block storage of data, multi-resolution data access, and flexible data streaming processing. Its essence is to host a regular GeoTIFF file on an HTTP file server, and through re-encoding and re-organizing, realize on-demand data reading in the cloud, and try to achieve cloud processing of data without downloading the entire file, thereby reducing the download and upload time and improving processing efficiency.

[0053] For this reason, the present invention proposes a method for cloud-optimized raster data conversion, compression, and sharing and publishing, aiming to combine cloud optimization technology and the characteristics of raster data storage, improve data storage and transmission efficiency, and reduce costs on the premise of ensuring data quality.

[0054] A cloud-optimized geological raster data processing method provided by the first embodiment of the present invention will be described in detail below in combination with Figure 1 for detailed description.

[0055] As Figure 1 shown, in step S1, the original geological raster data is converted into target format raster data with cloud optimization characteristics.

[0056] The above steps specifically include:

[0057] Step S11, convert the GeoTIFF format raster data into COG format data.

[0058] Specifically as follows:

[0059] S111, first create a C++ project in the Visual Studio 2019 environment, configure the GDAL3.5.2 conversion development library, and obtain the driver for converting raster data into COG format.

[0060] The GDAL library (Geospatial Data Abstraction Library) is an open source C++ converter library for raster and vector geospatial data formats. It is released by the Open Geospatial Foundation under an MIT (Massachusetts Institute of Technology) style open source license. Currently, almost all GIS and RS software use GDAL to read and write spatial data.

[0061] S112 uses the COG format conversion driver provided by GDAL to convert the regular GeoTIFF raster data into the Cloud Optimized GeoTIFF (COG) format and output the COG file. The COG format has cloud-optimized features that support high-speed data reading and distributed computing, thereby improving data access efficiency.

[0062] Furthermore, in order to avoid the risk of being unable to process raster data due to insufficient memory resources, the present invention adopts a blocking strategy to ensure the feasibility of raster data processing.

[0063] Step S12, adopting a block strategy to ensure data processing feasibility, including:

[0064] Step S121, gridding the target format raster data to obtain a plurality of raster data blocks;

[0065] Step S122 , encoding the obtained multiple raster data blocks in sequence to obtain a coding sequence, and processing the raster data blocks in sequence according to the generated coding sequence.

[0066] Raster data block is mainly used to decompose an image (several GB to hundreds of GB) for processing when hardware resources are limited. Specifically, by defining the height and width of the raster data block (h, x), the number of raster data blocks is calculated by dividing the height and width of the raster data (H, X) by the height and width of the raster data block. The block coding uses the first data block from the upper left corner of the grid sheet (i.e. the northwest direction of the grid data) as the starting coding block, and the reverse S-shaped sequential coding scheme (such as Figure 2 ). By the number of blocks The calculation times of raster data block processing can be obtained, and the raster data blocks are processed in sequence according to the coding sequence formed by the coding scheme, and the processed results are written to the corresponding positions in the COG dataset. Figure 3 shown.

[0067] like Figure 1 As shown, in step S2, the converted target format raster data is optimized before data service release, including building a pyramid structure, recoding and compressing the raster data.

[0068] The above steps specifically include:

[0069] Step S21, constructing a pyramid structure, including:

[0070] Step S211, obtaining raster data band information and defining the levels and the lowest resolution of the pyramid.

[0071] Specifically, read the COG raster dataset to obtain information such as the bands, spatial reference, and metadata of the COG raster dataset. Among them, a certain raster dataset may contain multiple bands, and each band represents a raster data layer. By user-defined pyramid scale level n - 1 to be constructed (i.e., the number of pyramid layers, generally not exceeding 20 levels, and the original raster data is recorded as the 0th layer), and the lowest resolution of the raster data (e.g., 1024×1024), when the sampling calculation reaches either the highest scale level or the lowest resolution condition, the pyramid structure index starts to be constructed.

[0072] Step S212, downsampling the raster data to obtain the next layer of the pyramid.

[0073] Specifically, select interpolation methods such as nearest neighbor interpolation, bilinear interpolation, and cubic convolution interpolation to downsample the original COG raster data to obtain the next layer of the pyramid. [[ID=ID=19]]

[0074] For example, read the spatial range of the COG raster dataset and the image resolution of the original raster data (i.e., raster height and width H×X). Further, use the quadtree indexing mechanism to slice the original raster dataset, that is, take the original raster data as the 0th layer, downsample and merge 2×2 pixels into 1 pixel to generate the 1st layer, and so on until reaching the user-defined scale level or resolution range.

[0075] As another example, the original COG raster data can be downsampled using bilinear interpolation to obtain the next layer of the pyramid, and its working principle is as follows:

[0076] Assume that G(x, y) is the interpolation point for raster image downsampling, and points A(0, 0), B(0, 1), C(1, 0), and D(1, 1) are the four nearest points to this point. Through first-order linear interpolation, f(x, 0) is obtained:

[0077] f(x, 0) = f(0, 0) + x[f(1, 0) - f(0, 0)]

[0078] Similarly, f(x, 1) can be obtained:

[0079] f(x, 1) = f(0, 1) + x[f(1, 1) - f(0, 1)]

[0080] Finally, perform first-order linear interpolation in the X direction to obtain f(x,y):

[0081] f(x,y) = f(x,0) + y[f(x,1) - f(x,1)]

[0082] = x[f(1,0) - f(0,0)] + y[f(0,1) - f(0,0)]

[0083] + xy[f(1,1) + f(0,0) - f(0,1) - f(1,0)] + f(0,0)

[0084] Step S213, perform cyclic sampling until the specified number of layers or the lowest resolution is reached.

[0085] Step S214, generate a pyramid structure for the raster data at different levels according to the principle of "placing the larger resolution at the bottom and the smaller resolution at the top".

[0086] Step S22, perform raster data recoding and compression processing. In this embodiment, the BASISU format UASTC compression technology provided by GDAL is used to compress the COG raster data to obtain a compressed raster data block. BasisUniversal is a "super compression" GPU texture data exchange system that supports two highly compressed intermediate file formats (basis or the KTX2 open standard of the Khronos Group) and can be quickly transcoded into various GPU compressed and uncompressed pixel formats. Specifically, it includes:

[0087] Step S221, divide the raster data into blocks of equal size.

[0088] Specifically, according to the technical characteristics of UASTC, the COG raster data is divided into blocks of equal size, and the size of each block is 8×8 pixels.

[0089] Step S222, convert the raster spatial domain data to frequency domain data.

[0090] Specifically, use the discrete cosine transform (DCT) algorithm to convert the COG spatial domain data to frequency domain data, where: g(i,j) is the read original band signal amount, G(u,v) is the transformed coefficient, and N×N is the matrix of the original raster image. Then:

[0091]

[0092]

[0093] After the DCT transformation, the COG data is decomposed into a series of frequency components, still an 8×8 matrix, that is, it can contain 64 coefficients, where the low-frequency components represent the smooth changes in the image.

[0094] Step S223: Quantize the converted frequency-domain data.

[0095] Specifically, to quantize the transformed frequency-domain data, the precision of the data is reduced through rounding and quantization tables. That is, divide the 64 DCT coefficients calculated in step S222 by the quantization step size and then round. Among them, the quantization step size is determined by the quantization tables such as Table 1 and Table 2, and the 8×8 matrix of the quantization table corresponds one by one to the 64 DCT coefficients. The quantization processing formula is:

[0096]

[0097] where: The value of round() is the nearest integer to c(u,v), that is, the integer obtained by rounding. G(u,v) is the DCT coefficient obtained by calculation in step S222, and Q(u,v) is the quantization step size. The reference values of Table 1 and Table 2 are used by the user in combination with the specific calculation scenario.

[0098] Table 1 Common Luminance Quantization Table (8*8)

[0099] 16 11 10 16 24 40 51 61 12 12 14 19 26 58 60 55 14 13 16 24 40 57 69 56 14 17 22 29 51 87 80 62 18 22 37 56 68 109 103 77 24 35 55 64 81 104 113 92 49 64 78 87 103 121 120 101 72 92 95 98 112 100 103 99

[0100] Table 2 Common Grayscale Quantization Table (8*8)

[0101]

[0102]

[0103] The low-frequency components obtained through calculation are mainly concentrated in the upper left corner.

[0104] Step S224: Encode and compress the quantized data.

[0105] Specifically, the Huffman entropy coding technique is used to encode and compress the quantized data. Specifically as follows:

[0106] Step S2241: First, use run-length encoding to re-encode the DCT coefficients quantized in S222, convert from a two-dimensional array to a one-dimensional array, and organize similar frequencies together to obtain a re-encoded matrix. The run-length encoding is as Figure 4 shown, and the array matrix reconstructed through run-length encoding is as Figure 5 shown.

[0107] Step S2242: Based on S2241, find the last non-zero array, represent the subsequent arrays with value 0 by EOF, and then use the Huffman entropy coding technique to compress the data for the updated array.

[0108] Step S225: Decode and inverse-quantize the compressed data, and reconstruct the image data through inverse transformation.

[0109] Specifically:

[0110] Step S2251: The decoding of the compressed data is the inverse operation of Huffman coding;

[0111] Step S2252: The inverse quantization is the inverse operation of S223, and its calculation formula is:

[0112]

[0113] Experimental analysis:

[0114] (1) Experimental environment: The algorithm test environment is Visual Studio 2019, and the programming language is C++. The pyramid level is 3, the interpolation method is nearest neighbor interpolation, and the data compression option is BASISU (Basis Universal).

[0115] (2) The data input is a digital elevation GeoTIFF format file of the whole Sichuan Province (such as Figure 6 ), and the original file size is 8.18 GB; the output is a file in COG format (such as Figure 7 ).

[0116] (3) Experimental results: ① The output data area is complete, and no obvious missing or shadow is found; ② The quality of the output data has not decreased significantly, the map zooming, panning and rotation are still smooth, the data rendering details are still clear, and the attribute table data is complete; ③ The size of the output data has decreased significantly, the original data file size has decreased from the original 8.19 GB to 6.08 GB, and the data compression ratio is 74.23%.

[0117] As Figure 1 shown, in step S3, data service publishing is performed on the optimized target format raster data.

[0118] Specifically, for the COG raster data service publishing: Based on GIS Server platform software such as GeoServer, configure the data service publishing for the compressed COG raster data obtained in S1 to S2, and publish the raster data service. The published COG raster data service is as Figure 8 shown.

[0119] Corresponding to the above-disclosed method for processing cloud-optimized geological raster data, an embodiment of the present invention also discloses a system for processing cloud-optimized geological raster data, as Figure 9 shown, which specifically includes:

[0120] A format conversion module for converting the original geological raster data into raster data in a target format with cloud optimization characteristics;

[0121] An optimization processing module for performing pre - optimization processing on the raster data in the target format after conversion, including constructing a pyramid structure, raster data recoding, and compression processing;

[0122] A service publishing module for publishing data services for the raster data in the target format after optimization processing.

[0123] It should be noted that for a detailed description of a cloud - optimized geological raster data processing system provided by an embodiment of the present invention, reference can be made to the relevant description of a cloud - optimized geological raster data processing method provided by an embodiment of the present application, which will not be elaborated here.

[0124] In addition, an embodiment of the present invention also provides an electronic device, which includes: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of a cloud - optimized geological raster data processing method as described in any one of the above.

[0125] It should be noted that for a detailed description of an electronic device provided by an embodiment of the present invention, reference can be made to the relevant description of a cloud - optimized geological raster data processing method provided by an embodiment of the present application, which will not be elaborated here.

[0126] In addition, an embodiment of the present invention also provides a computer - readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of a cloud - optimized geological raster data processing method as described in any one of the above.

[0127] It should be noted that for a detailed description of a computer - readable storage medium provided by an embodiment of the present invention, reference can be made to the relevant description of a cloud - optimized geological raster data processing method provided by an embodiment of the present application, which will not be elaborated here.

[0128] Those skilled in the art can understand that all or part of the functions of the various methods in the above embodiments can be implemented in a hardware manner or in a computer program manner. When all or part of the functions in the above embodiments are implemented in a computer program manner, the program can be stored in a computer-readable storage medium. The storage medium can include: read-only memory, random access memory, magnetic disk, optical disk, hard disk, etc. The above functions can be realized by a computer executing the program. For example, the program is stored in the memory of the device, and when the processor executes the program in the memory, the above all or part of the functions can be realized. In addition, when all or part of the functions in the above embodiments are implemented in a computer program manner, the program can also be stored in a storage medium such as a server, another computer, magnetic disk, optical disk, flash drive or mobile hard disk, and saved to the memory of the local device by downloading or copying, or the system of the local device is updated. When the processor executes the program in the memory, all or part of the functions in the above embodiments can be realized.

[0129] The above uses specific examples to elaborate on the present invention, which is only used to help understand the present invention and is not intended to limit the present invention. For those skilled in the art of the present invention, according to the idea of the present invention, several simple deductions, deformations or substitutions can also be made.

Claims

1. A cloud-optimized geological grid data processing method, characterized in that, The method includes: Step S1, converting the format of the original geological raster data into target format raster data with cloud optimization characteristics; Step S2, performing optimization processing before data service publishing on the converted target format raster data, including constructing a pyramid structure, raster data recoding, and compression processing; Step S3, performing data service publishing on the optimized target format raster data.

2. The cloud-optimized geological grid data processing method according to claim 1, characterized in that The said Step S1 specifically includes: Step S11, converting GeoTIFF format raster data into COG format data.

3. A cloud-optimized geological grid data processing method according to claim 2, characterized in that, The said Step S1 specifically further includes: Step S12, adopting a chunking strategy to ensure feasible data processing, including: Step S121, dividing the target format raster data into grids to obtain multiple raster data chunks; Step S122, encoding the obtained multiple raster data chunks in sequence to obtain an encoding sequence, and processing the raster data chunks in sequence according to the formed encoding sequence.

4. A cloud-optimized geological grid data processing method as described in claim 1, characterized in that, The said Step S2 specifically includes: Step S21, constructing a pyramid structure, including: Step S211, obtaining raster data band information, defining the levels and the lowest resolution of the pyramid; Step S212, performing downsampling on the raster data to obtain the next layer of the pyramid; Step S213, performing cyclic sampling until the specified number of levels or the lowest resolution is reached; Step S214, generating a pyramid structure by placing raster data at different levels according to the principle of "placing larger resolution at the bottom and smaller resolution at the top".

5. A cloud-optimized geological grid data processing method according to claim 1, characterized in that, The said Step S2 specifically includes: Step S22, raster data recoding and compression processing, including: Step S221, dividing the raster data into equal-sized chunks; Step S222, converting raster spatial domain data into frequency domain data; Step S223, quantizing the converted frequency domain data; Step S224, encoding and compressing the quantized data.

6. The cloud-optimized geological grid data processing method according to claim 5, wherein, The said Step S22 specifically further includes: Step S225, decoding and inverse quantizing the compressed data, and reconstructing the image data through inverse transformation.

7. The cloud-optimized geological grid data processing method according to claim 5, characterized in that The said Step S222 specifically includes: Converting spatial domain data into frequency domain data using the discrete cosine transform algorithm.

8. A cloud-optimized geological grid data processing method according to claim 5, characterized in that The said Step S224 specifically includes: Encoding and compressing the quantized data using Huffman entropy coding technology.

9. A cloud-optimized geological grid data processing system, characterized in that, The system includes: A format conversion module, used to convert the format of the original geological raster data into target format raster data with cloud optimization characteristics; An optimization processing module, used to perform optimization processing before data service publishing on the converted target format raster data, including constructing a pyramid structure, raster data recoding, and compression processing; A service publishing module, used to perform data service publishing on the optimized target format raster data.

10. An electronic device, characterized in that, The said device includes: a processor and a memory; The said memory is used to store one or more program instructions; The said processor is used to run one or more program instructions to execute the steps of a method for processing cloud-optimized geological raster data as described in any one of claims 1 to 8.

Images