A method for compressing ocean grid data

Abstract

The application discloses a compression method of marine grid data, and relates to the field of data processing.The method first performs mask calculation on target grid data, extracts a mask matrix and performs compression, so that position information of marine and land grids is separately stored, and the compression process of marine data is prevented from being disturbed by land data.Marine grid data is screened out according to the mask matrix, integer wavelet transformation is performed on the preprocessed data, and the wavelet transformation result is twice compressed by using run-length encoding and entropy encoding, so that the best compression ratio is achieved.Finally, the generated secondary compression file and the mask compression file are merged to form a final compression file.The application can achieve a higher compression ratio, meanwhile, ensures the quality and integrity of the data in the recovery process, and significantly improves the data storage and transmission efficiency.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of data compression, and particularly relates to a marine grid data compression method. BACKGROUND

[0002] Grid data is one of commonly used data representation formats and is the most common format used for storage and exchange of natural science data. Grid form is widely used in marine scientific research and application fields. Whether in the data collection stage, the processing stage or the later stage of providing a model for prediction simulation, marine grid data occupies a dominant position. With the rapid development of marine observation and assimilation, marine grid data is increasingly abundant, and the explosive growth of data volume brings great pressure on data storage management, especially network transmission.

[0003] Although transmission and storage technologies are also developing rapidly, they still cannot keep up with the growth momentum of data volume. The importance of data compression technology is self-evident. At present, there are various compression methods for grid data, such as run-length encoding, entropy encoding and wavelet transform compression. However, in the field of marine science, due to the existence of land, the above lossless compression schemes cannot be directly applied. The existing marine meteorological research field mainly uses the network communication data format (netCDF). This scheme uses the binary compression encoding technology of the "minimum value elimination method", which can achieve a compression ratio of about two times within the allowable precision range. This is still insufficient for large data volume transmission and storage tasks.

[0004] At present, there is a lossless compression method for scientific data. The scientific data is first subjected to wavelet transform and then subjected to secondary compression using lz77 and Huffman coding. Although a higher compression ratio is achieved, it cannot achieve a high compression ratio for marine grid data with frequent continuous land data.

[0005] At present, there is a new lossless compression method for marine grid data. Run-length encoding is used to compress marine grid data, which can compress multiple continuous land data points to two values. Entropy encoding is used for secondary compression of the run-length encoding result, which achieves a higher compression ratio. However, due to the large changes in marine grid data, even using incremental encoding cannot achieve a high compression ratio. SUMMARY

[0006] In view of the above problems, the present application provides a marine grid data compression method, which aims to significantly improve the compression ratio of marine grid data through advanced data processing and compression technology, and to ensure the quality and integrity of the data in the recovery process, thereby effectively solving the problem of low compression ratio of marine grid data in the prior art.

[0007] The marine grid data compression method of the present application comprises the following specific steps:

[0008] Step 1: Mask calculation is performed on the marine grid point data to obtain a mask matrix, and marine data and land data are distinguished therein.

[0009] Step 2: The mask matrix obtained in step 1 is rearranged to form one-dimensional data, and compression is performed through run-length encoding to obtain a mask compressed file.

[0010] Step 3: The grid point data matrix obtained in step 1 is preprocessed, and land points are removed, rearranged into one-dimensional data, incrementally encoded and integerized in sequence.

[0011] Step 4: Wavelet transform is performed on the preprocessed marine data to obtain a wavelet coefficient matrix.

[0012] Step 5: The wavelet coefficient matrix is thresholded using a hard thresholding method, and high-frequency coefficients with absolute values less than the threshold are set to zero, and high-frequency coefficients with absolute values greater than the threshold remain unchanged.

[0013] Step 6: The wavelet coefficient matrix after thresholding is compressed multiple times, and the mask compressed file is merged to generate the final compressed file.

[0014] First, the wavelet coefficient matrix is rearranged to obtain one-dimensional wavelet coefficient data.

[0015] Subsequently, the one-dimensional wavelet coefficient data after rearrangement is compressed for the first time using run-length encoding.

[0016] Further, the wavelet coefficient data after the first compression is compressed for the second time using entropy encoding operation.

[0017] Finally, the generated two-level compressed file is merged with the mask compressed file obtained in step 2 to form the final compressed file.

[0018] The advantages of the present application are:

[0019] 1. The marine grid point data compression method of the present application introduces the application of wavelet transform in the direction of marine grid point data, converts marine data into a wavelet coefficient matrix for secondary compression, and combines run-length encoding and entropy encoding to achieve higher compression ratio while ensuring data accuracy, thereby significantly improving data storage and transmission efficiency.

[0020] 2. The marine grid point data compression method of the present application separates the mask file from the data file and uses multi-layer compression to improve data security and confidentiality.

[0021] 3、The marine grid data compression method of the present application can freely select the wavelet function, the number of decomposition layers and the size of the high-frequency matrix threshold in the compression process, so as to freely select lossless or lossy compression, and control the precision of lossy compression, thereby autonomously controlling the compression size and compression precision. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 The overall flowchart of the marine grid data compression method of the present application is shown in the figure.

[0023] Figure 2 The mask matrix compression flowchart is shown in the figure.

[0024] Figure 3 The mask matrix rearrangement method schematic diagram is shown in the figure.

[0025] Figure 4 The result diagram of three-level wavelet transform is shown in the figure.

[0026] Figure 5 The decompression flowchart of the marine grid data compressed by the present application is shown in the figure. DETAILED DESCRIPTION

[0027] The present application will be further described in detail below with reference to the accompanying drawings.

[0028] The marine grid data compression method of the present application is shown in the figure, and the specific steps are as follows: Figure 1

[0029] The present application comprises the following steps:

[0030] Step 1: distinguishing the land grid points and the marine grid points in the original data, and calculating the mask matrix;

[0031] (1) Data extraction

[0032] The target grid point data of the required single element is extracted from the original multi-element file to form a grid point data matrix.

[0033] (2) Calculation of mask matrix

[0034] The mask matrix is calculated according to the value of the grid point data matrix. The mask matrix has the same size as the grid point data matrix, but each element of the mask matrix has a value of 0 or 1; wherein 0 represents a marine grid point and 1 represents a land grid point. The basis for judging the value of each element of the mask matrix is: if the data in the corresponding position of the grid point data matrix is nan or the nan value after being converted into a floating point number results in a very small value, then the value is 1, indicating a land grid point. If it is a normal value other than this, it is a marine grid point, and the value is 0.

[0035] ​Step 2: The mask matrix obtained in step 1 is rearranged in a specific order to form one-dimensional data, and compression is performed by run-length encoding to obtain a mask compression file, as shown in Figure 2

[0036] As shown in Figure 3 , the selection method for rearranging the mask matrix is to splice the mask matrix in row, column, and interlaced arrangement modes, and reorganize it into one-dimensional data. Here, multiple rearrangement methods can be selected for mask matrix rearrangement, and then one-dimensional data obtained by the multiple rearrangement methods is respectively run-length encoded and compressed to obtain a mask compression file. The rearrangement method with the largest compression ratio is selected as the best rearrangement scheme for this compression. The above run-length encoding is a traditional one-dimensional run-length encoding, which is a lossless compression method that can highly compress continuous identical values. The main compression principle is to use two numbers to represent continuous identical values, where the first number is the continuous identical value, and the second number is the number of continuous occurrences.

[0037] Step 3: The grid data matrix obtained in step 1 is preprocessed according to the precision of the marine data, and is sequentially removed from the land points, rearranged into one-dimensional data, incrementally encoded, and integerized to make the data suitable for subsequent integer wavelet transform.

[0038] The land points in the grid data matrix are removed using the mask matrix, and then the land point-free grid data is rearranged into one-dimensional marine grid data according to the rearrangement method determined in step 2.

[0039] The obtained marine grid data is incrementally encoded, i.e., the difference between the current data and the previous data is calculated in reverse order of the data sequence, and stored in the current data position.

[0040] The incrementally encoded marine data is integerized. The integerization method used is to add the maximum value of the negative increment to each increment, so that all increments are positive to facilitate subsequent processing. Since the increment encoding of marine data usually does not change much and the precision is usually not more than 10 -3 , the decimal point can be directly moved to the right by the number of bits of the data precision to directly integerize the increment data.

[0041] Step 4: The preprocessed and integerized marine data is subjected to wavelet transform to obtain a wavelet coefficient matrix. Wavelet transform is a time-frequency domain transform of the preprocessed and integerized marine data according to the selected specified wavelet function and decomposition level. Wavelet transform has the characteristic that most of the useful information is concentrated in the low-frequency coefficients, and a small amount of detailed information is concentrated in the high-frequency coefficients, as shown in Figure 4 ​The result of three-level wavelet transform is shown, wherein L and H represent low-frequency or high-frequency parts, and the number represents the i-th level wavelet transform, and the data size of high-frequency coefficients is greater than that of low-frequency coefficients. The wavelet coefficient matrix obtained after the wavelet transform is an integer value, which is more convenient for the next step of processing.

[0042] Step 5: Threshold processing is performed on the wavelet coefficient matrix.

[0043] Since the precision requirement of marine data is usually not more than 10 negative cubes, the threshold value can be appropriately adjusted to reduce the range of high-frequency wavelet coefficients or to allow the same value to appear more times in succession, so that the storage space can be further compressed while ensuring the accuracy of data recovery. There are two main methods for threshold processing, namely hard threshold method and soft threshold method. Since the hard threshold method can greatly increase 0 after output processing, it is convenient to improve the compression rate, and the calculation is simpler in solving the problem of mean square error, so the hard threshold method is used here. The high-frequency coefficients with an absolute value less than the threshold value are set to zero, and the high-frequency coefficients with an absolute value greater than the threshold value remain unchanged. The expression is:

[0044]

[0045] In the formula, u j,k is the wavelet coefficient of the kth point on the scale j; t j is the threshold value set on the scale j.

[0046] Step 6: The wavelet coefficient matrix after threshold processing is compressed multiple times using a compression model combining run-length encoding and entropy encoding, and is merged with the mask compression file to generate the final compression file.

[0047] First, the wavelet coefficient matrix is rearranged, and the wavelet coefficient matrix of different scales is rearranged in the same arrangement manner as the mask matrix to form a one-dimensional wavelet coefficient data.

[0048] Subsequently, the one-dimensional wavelet coefficient data after rearrangement is compressed for the first time using run-length encoding. Here, the run-length encoding process is the same as the run-length encoding compression of the mask matrix in step 2.

[0049] Further, the wavelet coefficient data after the first compression is compressed for the second time using entropy encoding operation. The entropy encoding operation is a traditional lossless compression data method, which can compress the input data through arithmetic encoding or Huffman encoding to obtain a series of restorable symbols. In this invention, Huffman encoding is selected to form a secondary compression file by Huffman encoding the wavelet coefficient data after the first compression.

[0050] Finally, the secondary compression file generated is merged with the mask compression file obtained in step 2 to form the final compression file.

[0051] The obtained compressed file can be recovered by the same recovery model as the data compression described above, as shown in the following specific process: Figure 5

[0052] 1) Separate the compressed file into a mask compressed file and a secondary compressed file.

[0053] 2) Use run-length encoding to decode the mask matrix in reverse, that is, repeat the first number of the number pair n times, and n is the size of the second number of the number pair.

[0054] 3) Rearrange the mask matrix to form the grid points of the original area.

[0055] 4) Huffman decode the secondary compressed file according to the Huffman tree generated in step 6 to form the primary compressed file;

[0056] 5) Use run-length encoding to decode the primary compressed file in reverse to obtain the wavelet coefficient matrix;

[0057] 6) Perform inverse wavelet transform on the wavelet coefficient matrix and perform inverse preprocessing to obtain the data corresponding to the ocean grid points;

[0058] 7) Fill the obtained ocean grid point data into the 0 value of the mask matrix according to the position order, and set the 1 value of the mask matrix to NAN, indicating the land value. Thus, the compression is completed, and the restored ocean grid point data containing land is obtained.

[0059] Embodiment:

[0060] The ocean grid point data compression of the present application is performed using python, and the specific steps are as follows:

[0061] (1) Select the ocean grid point data to be compressed:

[0062] Take the oisst data provided by NOAA satellite as an example:

[0063] Select the daily data of oissts data on March 1, 2022, and select the data area from west longitude 180° (-180°) to east longitude 180° (+180°), north latitude 90° (North Pole) to south latitude 90° (South Pole). The spatial resolution of this data is 0.25° x 0.25°, and the time resolution is 1 day. Extract the target sea surface temperature sst data to obtain a 720*1440 data matrix.

[0064] (2) Acquisition and compression of mask matrix:

[0065] ​Read the data matrix of sea surface temperature, according to whether the grid data is NAN value or small negative value, judge whether each grid is land. If it is land, the mask value at the same position is 1, if it is ocean, the mask value at the same position is 0, and a 720*1440 mask matrix is obtained.

[0066] (3) the obtained 720*1440 mask matrix is pretreated, and then run length coding compression is carried out, that is, the same value is represented by a number pair, and the number pair consists of two numbers, the first number represents the value, and the second number represents the number of continuous occurrence of the value. After run length coding of the mask matrix, the length of the target area is input at the end of the matrix to facilitate recovery in later period. The result of this step is the mask compression file.

[0067] (4) preprocessing of ocean data mainly includes:

[0068] According to the mask matrix, the original data matrix is processed, and the grid points of ocean data are selected and rearranged by row. Then the data is integerized. Since the precision of the sst data is 0.01, multiplying all the data by 100 can be integerized.

[0069] For the integerized ocean grid data, wavelet transform is carried out. By selecting appropriate wavelet function and decomposition level, wavelet coefficient matrix which is more convenient for subsequent compression can be obtained. Here we use haar wavelet and set the decomposition level to 3. After wavelet transform, four wavelet coefficient matrices are obtained. Threshold processing is carried out on the high frequency matrix to facilitate compression and not to lose accuracy. Since the precision of the sst data is 0.01 Celsius, the threshold value can be set to 0 after pre experiment when the threshold value is set to 0, which still meets the precision. The wavelet coefficient matrix after threshold processing is rearranged into a matrix in the form of row arrangement.

[0070] The wavelet coefficient matrix is compressed, mainly including:

[0071] For the rearranged matrix, run length coding compression is first carried out, and then entropy coding is used for secondary compression of the compression result. Specifically, Huffman coding is used for secondary compression.

[0072] The mask matrix compression file and the wavelet coefficient compression file are written into the same npz file in order, and the compression work is completed.

[0073] (5) evaluate the compression effect

[0074] Since the threshold selection process of wavelet transform has certain influence on the precision, after compression, MAE, MSE and RMSE indexes need to be selected to evaluate the overall compression effect, and the formulas are as follows:

[0075]

[0076] wherein n is the total number of ocean data grids, x i is the value of the grid data matrix before compression, y i is the value after decompression. The above indexes are all smaller, indicating that the prediction effect is better.

[0077] The compression method of the ocean grid data, which is from the perspective of separating and compressing the ocean and land grids, utilizes the feature that the wavelet coefficient matrix after the integer wavelet transform has more repeated coefficients, and compresses the ocean grid data with higher efficiency. The method can also freely select the wavelet function, the number of decomposition layers, and the size of the high-frequency matrix threshold, so as to freely select lossless or lossy compression, and control the precision of lossy compression, thereby autonomously controlling the compression size and the compression precision.

Claims

1. A method of ocean grid data compression, characterized by: The specific steps are as follows: Step 1: Mask calculation is performed on the marine grid data to obtain a mask matrix, and marine data and land data in the mask matrix are distinguished; Step 2: The mask matrix obtained in step 1 is rearranged to form one-dimensional data, and compression is performed through run-length encoding to obtain a mask compression file; Step 3: The grid data matrix obtained in step 1 is preprocessed, and the land points are removed, the one-dimensional data is rearranged, the incremental encoding is performed, and the integer type is formed; wherein the incremental encoding method is to calculate the difference between the current data and the previous data in reverse order according to the data sequence of the rearranged one-dimensional data, and store it in the current data position; Step 4: Wavelet transform is performed on the preprocessed marine data to obtain a wavelet coefficient matrix; Step 5: The wavelet coefficient matrix is thresholded using a hard thresholding method, and the high-frequency coefficients with an absolute value less than the threshold are set to zero, and the high-frequency coefficients with an absolute value greater than the threshold are kept unchanged; Step 6: The wavelet coefficient matrix after thresholding is compressed multiple times, and the mask compression file is merged to generate the final compression file; First, the wavelet coefficient matrix is rearranged to obtain one-dimensional wavelet coefficient data; Then, the one-dimensional wavelet coefficient data after rearrangement is compressed for the first time using run-length encoding; Further, the wavelet coefficient data after the first compression is compressed for the second time using entropy encoding operation; Finally, the generated two-level compression file and the mask compression file obtained in step 2 are merged to form the final compression file.

2. The method of claim 1, wherein: In step 1, the elements in the mask matrix are represented by 0 or 1, and the method is as follows: if the data in the corresponding position of the grid data matrix is nan or the minimum value caused by taking the floating point number after nan, then assign 1, indicating a land grid; if it is a normal value other than this, it is a marine grid, and the value is 0; in this way, the marine and land data in the mask matrix are distinguished.

3. The method of claim 1, wherein: In step 2 The mask matrix is arranged in row, column and interlaced arrangement, and then the one-dimensional data obtained is compressed by run-length encoding to obtain a mask compression file; Select the rearrangement mode with the largest compression ratio and the mask compression file.

4. The method of claim 1, wherein: The rearrangement mode in steps 3 and 6 is the same as the rearrangement mode determined in step 2.

5. The method of claim 1, wherein: The integer type in step 3 uses the rounding method, which adds the maximum value of the negative increment to each increment and directly moves the decimal point to the right by the number of data precision bits, so that all increments are positive.

6. The method of claim 1, wherein: In step 6, Huffman encoding is used to perform Huffman encoding on the wavelet coefficient data after the first compression to form a two-level compression file.

7. The method of claim 1, wherein: The decompression method of the final compression file obtained in step 6 is as follows: 1) Separate the compression file into a mask compression file and a two-level compression file; 2) Use run-length encoding to decode the mask matrix in reverse; 3) Rearrange the mask matrix to form the grid points of the original area; 4) Perform Huffman decoding on the two-level compression file according to the Huffman tree generated in step 6 to form a one-level compression file; 5) Use run-length encoding to decode the one-level compression file in reverse to obtain a wavelet coefficient matrix; 6) Perform inverse wavelet transform on the wavelet coefficient matrix, and perform anti-preprocessing to obtain the data corresponding to the marine grid points; 7) The obtained marine grid data is filled into the 0 value of the mask matrix in position order, and the 1 value of the mask matrix is set to NAN, indicating the land value, and thus the decompression is completed, and the restored marine grid data containing land is obtained.

8. The method of claim 1, wherein: After obtaining the compressed file in step 6, the compression effect is evaluated by the following method: The MAE, MSE and RMSE indicators are selected to evaluate the overall compression effect: Wherein, n is the total number of marine data grids, is the value of the grid data matrix before compression, is the decompressed value; the smaller the value, the better the prediction effect.

Images