Ocean lattice point data compression method

Abstract

The invention discloses an ocean lattice point data compression method, and relates to the field of data processing. According to the method, firstly, mask calculation is carried out on target grid point data, a mask matrix is extracted and compressed, so that position information of sea and land grid points is independently stored, and the situation that land data disturbs the compression process of ocean data is avoided. Meanwhile, ocean lattice point data are screened out according to a mask matrix, integer wavelet transformation is carried out after data preprocessing, and a wavelet transformation result is subjected to secondary compression by using run length coding and entropy coding, so that the optimal compression ratio is achieved. And finally, combining the generated secondary compressed file with the mask compressed file to form a final compressed file. According to the method, a higher compression ratio can be achieved, meanwhile, the quality and the integrity of the data in the recovery process are ensured, and the data storage and transmission efficiency is remarkably improved.

Description

Technical Field

[0001] The invention belongs to the technical field of data compression, and in particular relates to an ocean grid data compression method. Background Art

[0002] Grid data is one of the commonly used data representation formats and is the most common format for storing and exchanging natural science data. Grid data is widely used in marine scientific research and application fields. Whether it is the data collection stage, processing stage, or the later stage of providing models for prediction and simulation, marine grid data occupies a dominant position. Nowadays, with the rapid development of marine observation, assimilation and other hormones, while marine grid data is becoming more and more abundant, its data volume is exploding, which brings huge pressure to data storage management, especially network transmission.

[0003] Although transmission and storage technologies are also developing rapidly, they are still far behind the growth momentum of data volume. The importance of data compression technology is self-evident. At present, there are many compression methods for grid data, such as run-length coding, entropy coding, and wavelet transform compression. However, in the field of marine science, due to the existence of land, the above lossless compression schemes cannot be directly applied. The existing marine meteorological research field is mainly based on the network communication data format (netCDF), which uses the binary compression coding technology of the "minimum elimination method" and can achieve a compression rate of about twice within the range of accuracy. This is still insufficient for the transmission and storage of large amounts of data.

[0004] There is currently a lossless compression method for scientific data. The scientific data is first subjected to wavelet transform and then compressed twice using lz77 and Huffman coding. Although a higher compression ratio is achieved, a high compression rate cannot be achieved for ocean grid data where continuous land data frequently appears.

[0005] There is currently a new lossless compression method for ocean grid data. It uses run-length encoding to compress ocean grid data, which can compress multiple consecutive land data points into two values, and uses entropy coding to perform secondary compression on the result of run-length encoding, achieving a higher compression ratio. However, due to the large variations in ocean grid data, even using incremental encoding cannot achieve a very high compression rate. Summary of the invention

[0006] In view of the above problems, the present invention proposes a method for compressing ocean grid data, which aims to significantly improve the compression ratio of ocean grid data through advanced data processing compression technology; at the same time, ensure the quality and integrity of the data during the recovery process, thereby effectively solving the problem of low compression rate of ocean grid data existing in the prior art.

[0007] The ocean grid data compression method of the present invention comprises the following specific steps:

[0008] Step 1: Perform mask calculation on the ocean grid data to obtain a mask matrix and distinguish the ocean data from the land data.

[0009] Step 2: Rearrange the mask matrix obtained in step 1 to form one-dimensional data, and obtain a mask compression file through run-length encoding compression.

[0010] Step 3: Preprocess the grid data matrix obtained in step 1 by removing land points, rearranging it into one-dimensional data, and performing incremental encoding and integerization.

[0011] Step 4: Perform wavelet transform on the preprocessed ocean data to obtain the wavelet coefficient matrix.

[0012] Step 5: Use the hard threshold method to perform threshold processing on the wavelet coefficient matrix, set the high-frequency coefficients with absolute values ​​less than the threshold to zero, and keep the high-frequency coefficients with absolute values ​​greater than the threshold unchanged.

[0013] Step 6: Compress the wavelet coefficient matrix after threshold processing multiple times and merge it with the mask compression file to generate the final compressed file.

[0014] First, the wavelet coefficient matrix is ​​rearranged to obtain one-dimensional wavelet coefficient data.

[0015] Subsequently, the rearranged one-dimensional wavelet coefficient data is compressed for the first time using run-length coding.

[0016] Furthermore, the wavelet coefficient data after the primary compression is compressed for the second time using an entropy coding operation.

[0017] Finally, the generated secondary compressed file is merged with the mask compressed file obtained in step 2 to form the final compressed file.

[0018] The advantages of the present invention are:

[0019] 1. The ocean grid data compression method of the present invention introduces the application of wavelet transform in the direction of ocean grid data, converts the ocean data into a wavelet coefficient matrix for secondary compression, and achieves a higher compression rate while ensuring data accuracy by combining run-length coding and entropy coding, thereby achieving a significant improvement in data storage and transmission efficiency.

[0020] 2. The ocean grid data compression method of the present invention separates the mask file from the data file and uses multi-layer compression to improve the security and confidentiality of the data.

[0021] 3. The ocean grid data compression method of the present invention can freely select the wavelet function, the number of decomposition layers, and the size of the high-frequency matrix threshold during the compression process to achieve the purpose of freely selecting lossless or lossy compression and controlling the lossy compression accuracy, thereby independently controlling the compression size and compression accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 This is an overall flow chart of the method for compressing ocean grid data of the present invention;

[0023] Figure 2 is the mask matrix compression flow chart;

[0024] Figure 3 Schematic diagram of the mask matrix rearrangement method;

[0025] Figure 4 This is the result of the third-level wavelet transform;

[0026] Figure 5 The figure is a flow chart for decompressing the ocean grid data after being compressed by the present invention. DETAILED DESCRIPTION

[0027] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0028] The ocean grid data compression method of the present invention is as follows: Figure 1 As shown, the specific steps are:

[0029] The present invention comprises the following steps:

[0030] Step 1: Distinguish the land grid points and ocean grid points in the original data and calculate the mask matrix;

[0031] (1) Data extraction

[0032] The target grid point data of the required single element is extracted from the original multi-element file to form a grid point data matrix.

[0033] (2) Calculation of the mask matrix

[0034] Calculate the mask matrix based on the values ​​of the grid data matrix. The size of the mask matrix is ​​the same as that of the grid data matrix, but each element is either 0 or 1, where 0 represents an ocean grid and 1 represents a land grid. The value of each element of the mask matrix is ​​determined based on the following criteria: if the data at the corresponding position in the grid data matrix is ​​a nan value or a minimum value resulting from a floating point representation of a nan value, then a value of 1 is assigned to indicate a land grid. If it is a normal value other than this, then it is an ocean grid and a value of 0 is assigned.

[0035] Step 2: Rearrange the mask matrix obtained in step 1 in a specific order to form one-dimensional data, and obtain the mask compression file through run-length encoding compression, such as Figure 2 shown.

[0036] like Figure 3 As shown in the figure, the selection method of mask matrix rearrangement is: splicing the mask matrix in rows, columns and staggered arrangement, and reorganizing it into one-dimensional data; here, multiple rearrangement methods can be selected to rearrange the mask matrix, and then the one-dimensional data obtained by the multiple rearrangement methods are run-length encoded and compressed to obtain a mask compressed file; the rearrangement method with the largest compression ratio is selected as the best rearrangement scheme for this compression. The above-mentioned run-length encoding is a traditional one-dimensional run-length encoding, which is a lossless compression method that can highly compress consecutive identical values. Its main compression principle is to represent consecutive identical values ​​with two numbers, the first number is the consecutive identical value, and the second number is the number of consecutive occurrences.

[0037] Step 3: Preprocess the grid data matrix obtained in step 1 according to the accuracy of the ocean data, remove the land points, rearrange it into one-dimensional data, perform incremental encoding and integerization, so that the data is suitable for subsequent integer wavelet transform.

[0038] The land points in the grid data matrix are removed using a mask matrix, and then the grid data without land points is rearranged into one-dimensional ocean grid data according to the rearrangement method determined in step 2.

[0039] The obtained ocean grid data is incrementally encoded, that is, the difference between the current data and the previous data is calculated in reverse order according to the data sequence, and stored in the current data position.

[0040] The incrementally encoded ocean data is converted to integers. The rounding method used is to add the maximum value of the negative increment to each increment, so that all increments are positive numbers for subsequent processing. Since the incremental encoding of ocean data usually does not change much and the accuracy is usually no more than 10 -3 , you can directly move the decimal point to the right by the number of digits of data precision and directly convert the incremental data into an integer.

[0041] Step 4: Perform wavelet transform on the preprocessed integer ocean data to obtain the wavelet coefficient matrix. Wavelet transform is to select the specified wavelet function and decomposition level according to the data characteristics to transform the preprocessed integer ocean data from time domain to frequency domain. Wavelet transform has the characteristics that most of the useful information is concentrated in the low-frequency coefficients, while a small part of the detail information is concentrated in the high-frequency coefficients, such as Figure 4The figure shows the results of the three-level wavelet transform, where L and H represent the low-frequency or high-frequency part, and the number represents the i-th level wavelet transform. The data scale of the high-frequency coefficient is larger than that of the low-frequency coefficient. The wavelet coefficient matrices obtained after the above wavelet transform are all integer values, which is more convenient for the next step of processing.

[0042] Step 5: Perform threshold processing on the wavelet coefficient matrix.

[0043] Since the accuracy requirement for ocean data is usually no more than 10 to the negative cube, the threshold can be adjusted appropriately to narrow the range of high-frequency wavelet coefficients or allow the same value to appear more times in succession, so that the storage space can be further compressed while ensuring the accuracy of data recovery. There are currently two main methods for threshold processing, namely the hard threshold method and the soft threshold method. Since the hard threshold method can significantly increase the output after processing by 0, it is easy to improve the compression rate and is simpler to calculate in solving the problem of mean square error, the hard threshold method is used here to set the high-frequency coefficients with absolute values ​​less than the threshold to zero, and the high-frequency coefficients with absolute values ​​greater than the threshold remain unchanged. Its expression is:

[0044]

[0045] In the formula, u j,k is the wavelet coefficient of the kth point on scale j; t j is the threshold set on scale j.

[0046] Step 6: Use a compression model combining run-length coding and entropy coding to compress the wavelet coefficient matrix after threshold processing multiple times, and merge it with the mask compression file to generate the final compressed file;

[0047] First, the wavelet coefficient matrix is ​​rearranged, and the wavelet coefficient matrices of different scales are rearranged into one-dimensional wavelet coefficient data in the same arrangement as the mask matrix.

[0048] Subsequently, the rearranged one-dimensional wavelet coefficient data is compressed for the first time using run-length coding. Here, the run-length coding process is the same as the run-length coding compression of the mask matrix in step 2.

[0049] Furthermore, the wavelet coefficient data after the primary compression is compressed for the second time using entropy coding operation. Entropy coding operation is a traditional lossless data compression method, which can compress the input data through arithmetic coding or Huffman coding to obtain a series of restorable symbols. In the present invention, Huffman coding is selected to perform Huffman coding on the wavelet coefficient data after the primary compression to form a secondary compression file.

[0050] Finally, the generated secondary compressed file is merged with the mask compressed file obtained in step 2 to form the final compressed file.

[0051] The compressed file can be restored using the same recovery model as the above data compression, such as Figure 5 As shown, the specific process is:

[0052] 1) Separate the compressed file into a mask compressed file and a secondary compressed file.

[0053] 2) Use run-length coding to reversely decode the mask matrix, that is, repeat the first number of the pair n times, where n is the size of the second number of the pair.

[0054] 3) Rearrange the mask matrix to form the grid points of the original region.

[0055] 4) Perform Huffman decoding on the secondary compressed file according to the Huffman tree generated in step 6 to form a primary compressed file;

[0056] 5) Use run-length encoding to reversely decode the first-level compressed file to obtain the wavelet coefficient matrix;

[0057] 6) Perform inverse wavelet transform on the wavelet coefficient matrix and perform inverse preprocessing to obtain the corresponding data of the ocean grid points;

[0058] 7) The obtained ocean grid data is filled into the 0 value of the mask matrix in order of position, and the 1 value of the mask matrix is ​​set to NAN, indicating the land value. At this point, the decompression is completed, and the restored ocean grid data containing the land is obtained.

[0059] Example:

[0060] The ocean grid data compression of the present invention is performed using Python, and the specific steps are as follows:

[0061] (1) Select the ocean grid data to be compressed:

[0062] Take the OISST data, a sea surface temperature analysis product provided by NOAA satellite, as an example:

[0063] Select the daily data of OISSTS data on March 1, 2022. The data area is selected from 180°W (-180°) to 180°E (+180°), 90°N (North Pole) to 90°S (Antarctic). The spatial resolution of the data is 0.25°×0.25° and the temporal resolution is 1 day. Extract the target sea surface temperature SST data and obtain a 720*1440 data matrix.

[0064] (2) Obtaining and compressing the mask matrix:

[0065] Read the data matrix of sea surface temperature, and judge whether each grid point is land based on whether the grid point data is NAN value or extremely small negative value. If it is land, the mask value of the same position is set to 1, and if it is ocean, the mask value of the same position is set to 0, and a 720*1440 mask matrix is ​​obtained.

[0066] (3) After preprocessing the obtained 720*1440 mask matrix, run-length encoding compression is performed on it, that is, the method of using: consecutive identical values ​​are represented by a number pair, which consists of two numbers, where the first number represents the value and the second number represents the number of consecutive occurrences of the value. After run-length encoding is performed on the mask matrix, the length of the target area is input at the end of the matrix for later recovery. The result of this step is the mask compression file.

[0067] (4) Preprocessing of ocean data, mainly including:

[0068] According to the mask matrix, the original data matrix is ​​processed, the grid points of the ocean data are screened out, and the data are rearranged by row. Then the data is integerized. Since the precision of this SST data is 0.01, all data can be integerized by multiplying them by 100.

[0069] For the integerized ocean grid data, wavelet transform is performed on it. By selecting the appropriate wavelet function and the number of decomposition layers, a wavelet coefficient matrix that is more convenient for subsequent compression can be obtained. Here we use the Haar wavelet and set the number of decomposition layers to 3. After wavelet transform, 4 wavelet coefficient matrices are obtained. Threshold processing is performed on the high-frequency matrix to facilitate compression without losing accuracy. Since the accuracy of this SST data is 0.01 degrees Celsius, when the threshold is set to 0 after preliminary experiments, the accuracy is still met, so this threshold can be set to 0. The wavelet coefficient matrix after threshold processing is rearranged into a matrix in a row-by-row manner.

[0070] The compression of wavelet coefficient matrix mainly includes:

[0071] The re-arranged matrix is ​​firstly compressed by run-length coding, and then the compressed result is compressed again by entropy coding, specifically Huffman coding.

[0072] Write the mask matrix compressed file and the wavelet coefficient compressed file into the same npz file in sequence, and the compression work is completed.

[0073] (5) Evaluate the compression effect

[0074] Since the threshold selection process of wavelet transform has a certain impact on the accuracy, after compression is completed, it is necessary to select MAE, MSE and RMSE indicators to evaluate the overall compression effect. The formula is as follows:

[0075]

[0076] Where n is the total number of ocean data grids, x i is the value of the grid data matrix before compression, y i is the decompressed value. The smaller the value of the above indicators is, the better the prediction effect is.

[0077] The present invention provides a method for compressing ocean grid data. The method starts from the perspective of separating and compressing the ocean and land grids, and utilizes the characteristic that the wavelet coefficient matrix after integer wavelet transformation has a large number of repeated coefficients to compress the ocean grid data with high efficiency. The method can also freely select the wavelet function, the number of decomposition layers, and the size of the high-frequency matrix threshold to achieve the purpose of freely selecting lossless or lossy compression, and controlling the accuracy of lossy compression, thereby independently controlling the compression size and compression accuracy.

Claims

1. A method for compressing ocean grid data, characterized in that: The specific steps are: Step 1: Perform mask calculation on the ocean grid data to obtain a mask matrix and distinguish the ocean data from the land data; Step 2: Rearrange the mask matrix obtained in step 1 to form one-dimensional data, and obtain a mask compression file through run-length encoding compression; Step 3: Preprocess the grid data matrix obtained in step 1 by removing land points, rearranging into one-dimensional data, and performing incremental coding and integerization; Step 4: Perform wavelet transform on the preprocessed ocean data to obtain the wavelet coefficient matrix; Step 5: Use the hard threshold method to perform threshold processing on the wavelet coefficient matrix, set the high-frequency coefficients whose absolute values ​​are less than the threshold to zero, and keep the high-frequency coefficients whose absolute values ​​are greater than the threshold unchanged; Step 6: compress the wavelet coefficient matrix after threshold processing multiple times and merge it with the mask compression file to generate the final compression file; First, the wavelet coefficient matrix is ​​rearranged to obtain one-dimensional wavelet coefficient data; Then, the rearranged one-dimensional wavelet coefficient data is compressed for the first time using run-length coding; Further, the wavelet coefficient data after the primary compression is compressed for the second time using an entropy coding operation; Finally, the generated secondary compressed file is merged with the mask compressed file obtained in step 2 to form the final compressed file.

2. The method for compressing ocean grid data according to claim 1, characterized in that: In step 1, the value of each element in the mask matrix is ​​represented by 0 or 1. The method is: if the data at the corresponding position in the grid data matrix is ​​a nan value or a minimum value caused by a floating point representation of the nan value, then a value of 1 is assigned to indicate a land grid point; if it is a normal value other than this, then it is an ocean grid point and a value of 0 is assigned to distinguish between ocean and land data in the mask matrix.

3. The method for compressing ocean grid data according to claim 1, characterized in that: In step 2, the mask matrix is ​​arranged in various rearrangement modes, and then the obtained one-dimensional data is run-length encoded and compressed to obtain a mask compression file; Select the rearrangement method with the highest compression ratio and the mask compression file.

4. The method for compressing ocean grid data according to claim 1, characterized in that: The rearrangement method described in step 3 and step 6 is the same as the rearrangement method determined in step 2.

5. The method for compressing ocean grid data according to claim 1, characterized in that: The incremental encoding method in step 3 is to calculate the difference between the current data and the previous data in reverse order according to the data sequence for the data obtained after the rearrangement, and store it in the current data position.

6. The method for compressing ocean grid data according to claim 1, characterized in that: The integer conversion described in step 3 adopts the rounding method, adding the maximum value of the negative increment to each increment, and directly moving the decimal point to the right by the number of digits of data precision, so that all increments are positive numbers.

7. The method for compressing ocean grid data according to claim 1, characterized in that: In step 6, Huffman coding is selected to perform Huffman coding on the wavelet coefficient data after the first compression to form a secondary compression file.

8. The method for compressing ocean grid data according to claim 1, characterized in that: The decompression method of the final compressed file obtained in step 6 is: 1) Separate the compressed file into a mask compressed file and a secondary compressed file; 2) Use run-length coding to inversely decode the mask matrix; 3) Rearrange the mask matrix to form the grid points of the original area; 4) Perform Huffman decoding on the secondary compressed file according to the Huffman tree generated in step 6 to form a primary compressed file; 5) Use run-length encoding to reversely decode the first-level compressed file to obtain the wavelet coefficient matrix; 6) Perform inverse wavelet transform on the wavelet coefficient matrix and perform inverse preprocessing to obtain the corresponding data of the ocean grid points; 7) The obtained ocean grid data is filled into the 0 value of the mask matrix in order of position, and the 1 value of the mask matrix is ​​set to NAN, indicating the land value. At this point, the decompression is completed, and the restored ocean grid data containing the land is obtained.

9. The method for compressing ocean grid data according to claim 1, characterized in that: After obtaining the compressed file in step 6, the compression effect is evaluated as follows: Select MAE, MSE and RMSE indicators to evaluate the overall compression effect: Where n is the total number of ocean data grids, x i is the value of the grid data matrix before compression, y i is the decompressed value; the smaller the value of the above indicators is, the better the prediction effect is.

Images