A novel gridded meteorological data encoding and compression method based on sparse matrices

Abstract

This invention discloses a novel gridded meteorological data encoding and compression method based on sparse matrices, belonging to the field of data processing technology. The method includes: reading a gridded meteorological data matrix Matrix0 with a resolution of {m×n} from a gridded meteorological data field numerical model; multiplying Matrix0 by a scaling factor to convert it into matrix Matrix1; extracting the first value at index [0, 0] of the gridded meteorological data field as a baseline value; creating matrix Matrix2 and assigning values ​​point by point; converting each vertex value of matrix Matrix2 into a gridded meteorological data field matrix Matrix3, encoding it using p bits; describing the resolution and values ​​of the gridded meteorological data field using a 64-byte file; and saving matrix Matrix3 as a {m×n×p / 8+64}-byte binary file, completing the data encoding and converting it into the original sparse matrix field.

Description

Technical Field

[0001] This invention belongs to the field of data processing technology, specifically relating to a novel gridded meteorological data encoding and compression method based on sparse matrices. Background Technology

[0002] Numerical forecasting plays a crucial role in weather forecasting. During the use of numerical forecasting, a large amount of gridded meteorological data needs to be transmitted over the network. Therefore, the encoding and storage format of numerical forecast gridded data has a significant impact on the storage and transmission of massive amounts of meteorological data.

[0003] The China Meteorological Administration has defined the MICAPS4 text format for encoding and storing gridded meteorological data, while the World Meteorological Organization has defined the binary GRIB format for encoding and compressing numerical weather prediction gridded meteorological data. Both are commonly used encoding formats for storing gridded meteorological data.

[0004] However, as numerical weather prediction accuracy increases and the spatial resolution of gridded meteorological data becomes increasingly higher, the original data storage encoding formats are increasingly unable to meet the requirements for storing and transmitting high-resolution gridded meteorological data over networks. The MICAPS4 encoding format developed by the China Meteorological Administration records gridded meteorological data as text files. With the increase in gridded meteorological data resolution, the file size becomes extremely large; a single high-precision global gridded meteorological field can reach nearly 200MB, which is highly unfavorable for data transmission over networks. The GRIB2 data format defined by the World Meteorological Organization incorporates data compression mechanisms. Although the file size is smaller, this format is very complex to encode, has poor encoding and decoding performance, and places high demands on the decoding program, hindering the rapid application of gridded meteorological data. Summary of the Invention

[0005] This invention provides a novel gridded meteorological data encoding and compression method based on sparse matrices, the improvement of which lies in that the method includes...

[0006] (1) Read the gridded meteorological data matrix Matrix0 with a resolution of {m×n} from the gridded meteorological data field numerical model;

[0007] (2) Multiply matrix Matrix0 by the scaling factor and convert it into a new matrix Matrix1;

[0008] (3) Extract the first value of the gridded meteorological data field [0, 0] as the baseline value;

[0009] (4) Create a new matrix Matrix2 and assign values ​​to each point. The value is the difference between the grid point and the previous neighboring point.

[0010] (5) Convert each vertex value of matrix Matrix2 into a grid meteorological data field matrix Matrix3, and encode it using p bits;

[0011] (6) Describe the resolution and values ​​of the entire gridded meteorological data field in a 64-byte file;

[0012] (7) Save matrix Matrix3 as a binary file of {m×n×p / 8+64} bytes, complete the data encoding, calculate and convert it into the original sparse matrix field.

[0013] Further, step (1) includes analyzing and processing the gridded meteorological data field, reading the gridded meteorological data matrix Matrix0 with a resolution of {m×n} from the numerical model of the gridded meteorological data field, and defining the scale factor of the data.

[0014] Further, step (2) includes

[0015] Multiply the gridded meteorological data matrix Matrix0 by the scale factor and convert it into a new matrix Matrix1;

[0016] Matrix1 ≈ |Matrix0 × scaleFactor + 0.5|;

[0017] The scale factor is set to 10.

[0018] Further, step (3) includes extracting the data in the first row and first column of the gridded meteorological data field [0, 0] from the new matrix Matrix1 as the data encoding reference value:

[0019] Value0 = Matrix1[0,0].

[0020] Further, step (4) includes constructing a new matrix Matrix2 and assigning values ​​point by point. In matrix Matrix2, the value of each grid point is determined according to the following rules:

[0021] The first value of the vertex, [0,0], is set as the base value for the whole field encoding, and the default value is 0.

[0022] Starting from the second row, the value in the first column of each row is the difference between the current position and the position in the first column of the previous row;

[0023] Starting from the second column of each row, the value of each column is the difference between the current value and the previous value;

[0024] The formulas for calculating the values ​​of matrix Matrix2 are as follows:

[0025]

[0026] Each Matrix2 value is recorded as the difference between adjacent grid points.

[0027] Further, step (5) includes traversing matrix Matrix2, finding the maximum value mx and the minimum value mn in the array, and obtaining the overall range of the data upper and lower limits as follows:

[0028] range = mx - mn;

[0029] Number of bits required to calculate the range of data:

[0030]

[0031] Here, the value p indicates that the value between mx and mn can be expressed using p bits;

[0032] Subtract mn from the entire matrix Matrix2 to obtain a new {m×n} resolution gridded meteorological data field matrix Matrix3:

[0033] Matrix3 = Matrix2 - mn;

[0034] The new gridded meteorological data field matrix Matrix3 is a positive integer gridded meteorological data field with a minimum of 0 and a maximum of range. All data is represented by p bits of binary data.

[0035] Convert each data in matrix Matrix3 into binary data identified by p bits;

[0036] After converting each data in matrix Matrix3 to binary, store it in a byteArray of p bits in sequence.

[0037] Furthermore, step (6) includes defining a 64-byte file header to describe the resolution and data segments of the entire gridded meteorological data field.

[0038] Further, step (7) saves the matrix Matrix3 as a binary file of {m×n×p / 8+64} bytes and completes data encoding. It reads p bits sequentially from the data segment, converts the binary bits into integer data vi, and calculates the original sparse matrix field.

[0039] Matrix_rev=(vi+mn)×scaleFactor

[0040] The entire field is successively added to the values ​​of its neighboring grid points to obtain the original matrix field. The calculation method is as follows:

[0041]

[0042] The matrix Matrix_ori is the same as the original matrix Matrix0, and the encoded matrix field is restored to the original matrix.

[0043] Beneficial effects:

[0044] Based on the accuracy requirements of gridded meteorological data, this invention converts the entire gridded meteorological data field into an integer and utilizes the continuity characteristic of adjacent grid points to record only the difference between adjacent grid points, thus converting the entire meteorological gridded meteorological data field from data into a sparse matrix. For this sparse matrix, the data encoding is used to convert each number into a number described by p numbers, which greatly enhances the data compression ratio.

[0045] Meanwhile, due to the sequential read / write characteristics of this data, the encoding and decoding algorithm for each number depends on the value of the preceding adjacent grid point. During the data encoding and decoding process, the data is read and written sequentially, and the adjacent grid points have already undergone the encoding and decoding process. The processing algorithm for the next grid point is already prepared, so the encoding and decoding speed of the entire data field is also very fast, which is beneficial for the transmission and application of gridded meteorological data files encoded in this format on the Internet.

[0046] It should be understood that the above general description and the following specific embodiments are merely exemplary and illustrative, and do not limit the scope of the claims made in this application. Attached Figure Description

[0047] Figure 1 This is a flowchart of a novel grid-based meteorological data encoding and compression method based on sparse matrices according to the present invention.

[0048] It should be understood that the accompanying drawings are not necessarily drawn to scale and present slightly simplified representations of various features illustrating the basic principles of this disclosure. Specific design features of the invention as disclosed herein, including, for example, particular dimensions, orientations, positions, and shapes, will be determined in part by the specifically intended application and usage environment.

[0049] In the figures, throughout the several figures, reference numerals refer to the same or equivalent parts of the invention. Detailed Implementation

[0050] Reference will now be made in detail to various embodiments of the invention, examples of which are illustrated in the accompanying drawings and described below. Although the invention will be described in conjunction with exemplary embodiments thereof, it should be understood that this specification is not intended to limit the invention to those exemplary embodiments. On the other hand, the invention is intended to cover not only the exemplary embodiments thereof, but also various alternatives, modifications, equivalents and other embodiments that may be included within the spirit and scope of the invention as defined by the appended claims.

[0051] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The specific structures and functions described in the exemplary embodiments of the present invention are for illustrative purposes only. Embodiments of the present invention can be implemented in various forms, and it should be understood that they should not be construed as limited to the exemplary embodiments described in the exemplary embodiments, but include all modifications, equivalents, or substitutions included within the spirit and scope of the present invention.

[0052] Throughout this specification, the technical terms used are for the purpose of describing various exemplary embodiments only and are not intended to be limiting. It will be further understood that the terms "comprising," "including," "having," etc., when used in the exemplary embodiments, specifically refer to the presence of the stated components, steps, operations, or elements, but do not exclude the presence or addition of one or more other components, steps, operations, or elements.

[0053] Based on the spatial continuity of meteorological data, this invention processes gridded meteorological data into a sparse matrix, and then encodes the gridded meteorological data based on the sparse matrix. This greatly reduces the data storage capacity, which is beneficial for data storage and transmission. Furthermore, this method has simple encoding and fast decoding speed, which is conducive to the transmission of gridded meteorological data over the Internet and to the rapid decoding and application of the data by the application end.

[0054] like Figure 1 As shown, this invention analyzes and processes gridded meteorological data fields, converting them into sparse matrices while retaining a certain level of accuracy. These sparse matrices are then encoded to reduce data storage capacity. The specific steps of this invention include:

[0055] S11, For the gridded meteorological data Matrix0 with a resolution of {m×n}, analyze and process the gridded meteorological data field, read the gridded meteorological data matrix Matrix0 with a resolution of {m×n} from the numerical model of the gridded meteorological data field, and define the scale factor of the data.

[0056] S12, multiply the entire gridded meteorological data matrix Matrix0 by the scale factor, round off the integer part, and convert it into a new {m×n} resolution integer gridded meteorological data field matrix Matrix1:

[0057] Matrix1 ≈ |Matrix0 × scaleFactor + 0.5|;

[0058] The value of this scaleFactor can be specified specifically based on the characteristics of the corresponding meteorological elements. According to meteorological industry practice, this value is generally set to 10, which is equivalent to retaining 0.1% precision in the original data.

[0059] S13, in the new matrix field Matrix1, extract the data in the first row and first column as the base value for subsequent data encoding:

[0060] Value0 = Matrix1[0,0];

[0061] S14, construct a new matrix Matrix2 and assign values ​​point by point. The rules for assigning values ​​to each grid point in matrix Matrix2 are as follows:

[0062] The first value of the vertex, [0,0], is set as the base value for the whole field encoding, and the default value is 0.

[0063] Starting from the second row, the value in the first column of each row is the difference between the current position and the position in the first column of the previous row;

[0064] Starting from the second column of each row, the value of each column is the difference between the current value and the previous value.

[0065] The formulas for calculating each value of the entire two-dimensional field matrix Matrix2 are as follows:

[0066]

[0067] The algorithm above yields a new matrix, Matrix2. Each value in Matrix2 records the difference between adjacent grid points. Due to the continuity of adjacent grid points in the gridded meteorological data, each value is a positive or negative integer with a small absolute value.

[0068] S15, traverse matrix Matrix2, find the maximum value mx and minimum value mn in the array, and obtain the overall range of the data upper and lower limits:

[0069] range = mx - mn;

[0070] Calculate the number of bits required to fully represent this data range:

[0071]

[0072] The value p indicates that p bits are sufficient to fully express all values ​​between mx and mn.

[0073] S16, subtract mn from the entire matrix Matrix2 to transform it into a new {m×n} resolution gridded meteorological data field matrix Matrix3:

[0074] Matrix3 = Matrix2 - mn;

[0075] The new gridded meteorological data field is a positive integer gridded meteorological data field with a minimum of 0 and a maximum of range. All data can be represented by p bits of binary data.

[0076] Allocate a byte array byteArray of length len.

[0077]

[0078] Convert each data in the Matrix3 matrix into binary data identified by p bits;

[0079] After converting each data in the Matrix3 matrix into binary, store each data in a byteArray in p-bit order.

[0080] S17 defines a 64-byte file header to describe the data segment. The format of each byte in the data header is designed as follows:

[0081] 1-8 bytes: 8-byte file marker, used as a file format marker, with the default content being "WIZGRD01";

[0082] 9-12: 4 bytes, floating-point number, starting longitude;

[0083] 13-16: 4 bytes, floating-point number, ending longitude;

[0084] 17-20: 4 bytes, floating-point number, starting latitude;

[0085] 21-24: 4 bytes, floating-point number, end dimension;

[0086] 25–28: 4 bytes, floating-point number, longitude interval;

[0087] 29–32: 4 bytes, floating-point number, latitude interval;

[0088] 33-36: 4 bytes, integer, number of grid points in the latitude direction;

[0089] 37-40: 4 bytes, integer, number of grid points in the longitude direction;

[0090] 42-44: 4 bytes, floating-point number, data base value, Value 0;

[0091] 45-48: 4 bytes, floating-point number, data scaling factor coefficient;

[0092] 49-52: 4 bytes, unsigned short type, the number of bits occupied by each number;

[0093] 53-56: 4 bytes, integer, the length of each number in bits, p;

[0094] 57-58: 2 bytes, short type, minimum value mn;

[0095] 59-60: 2 bytes, short type, maximum value mx;

[0096] 61-64: 4 bytes, integer, data segment length len;

[0097] The data segment begins at byte 65, and its length and content are defined as follows:

[0098] 65~(64+len): len bytes, the encoded data content byteArray;

[0099] S18, after encoding, the data size of the entire gridded meteorological data field matrix Matrix3 is reduced from {m×n×4} bytes to {m×n×p÷8+64} bytes while maintaining data accuracy. This is generally about 1 / 8 of the original data size, and in extreme cases such as precipitation phases, it is 1 / 16 of the original data size.

[0100] After encoding, the data is decoded sequentially according to its storage order. After reading p bits sequentially from the data segment, the binary bits are converted into integer data vi, and then converted back into the original sparse matrix field through the following calculation:

[0101] Matrix_rev=(vi+mn)×scaleFactor

[0102] The original matrix field can be obtained by adding the entire field to the values ​​of its neighboring grid points sequentially. The calculation method is as follows:

[0103]

[0104] The matrix Matrix_ori is the same as the original matrix Matrix0. After this decoding process, the encoded matrix field is restored to the original matrix.

[0105] The embodiments of this application described above can be implemented in various hardware, software codes, or combinations thereof. For example, embodiments of this application may also represent program code executing the above methods in a data signal processor. This application may also relate to various functions performed by a computer processor, digital signal processor, microprocessor, or field-programmable gate array. The processor described above can be configured to perform specific tasks according to this application, which are accomplished by executing machine-readable software code or firmware code defining the specific methods disclosed in this application. The software code or firmware code can be developed to represent different programming languages ​​and different formats or forms. It can also represent software code compiled for different target platforms. However, the different code styles, types, and languages ​​of the software code performing tasks according to this application and other types of configuration code do not depart from the spirit and scope of this application.

[0106] The foregoing description of specific exemplary embodiments of the invention has been presented for purposes of illustration and description. It is not intended to exclude or limit the invention to the precise forms disclosed, and it will be apparent that many modifications and alterations are possible in light of the foregoing teachings. Exemplary embodiments were chosen and described to explain certain principles of the invention and their practical application, so that others skilled in the art can make or utilize various exemplary embodiments of the invention, and their various alternatives and modifications. The purpose is that the scope of the invention will be defined by the appended claims and their equivalents.

[0107] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A novel gridded meteorological data encoding and compression method based on sparse matrices, characterized in that, The method includes: (1) In the data signal processor, a grid meteorological data matrix Matrix0 with a resolution of {m×n} is read from the grid meteorological data field numerical model containing meteorological element features, wherein the adjacent grid points of the grid meteorological data are specifically continuous. Step (1) includes: analyzing and processing the gridded meteorological data field, reading the gridded meteorological data matrix Matrix0 with a resolution of {m×n} from the numerical model of the gridded meteorological data field, and defining the scale factor of the data. The value of the scale factor is specifically specified according to the characteristics of the corresponding meteorological elements. (2) Multiply matrix Matrix0 by the scaling factor and convert it into a new matrix Matrix1; (3) Extract the first value of the gridded meteorological data field [0, 0] as the baseline value; (4) Create a new matrix Matrix2 and assign values ​​to each point. The value is the difference between the grid point and the previous neighboring point. Step (4) involves constructing a new matrix Matrix2 and assigning values ​​point by point. The rules for assigning values ​​to each grid point in matrix Matrix2 are as follows: The first value of the vertex, [0,0], is set as the base value for the whole field encoding, and the default value is 0. Starting from the second row, the value in the first column of each row is the difference between the current position and the position in the first column of the previous row; Starting from the second column of each row, the value of each column is the difference between the current value and the previous value; (5) Convert each vertex value of matrix Matrix2 into a grid meteorological data field matrix Matrix3, and encode it using p bits; (6) Describe the resolution and values ​​of the entire gridded meteorological data field in a 64-byte file; (7) Save the matrix Matrix3 as a binary file of {m×n×p / 8+64} bytes, complete the data encoding, calculate and convert it into the original sparse matrix field, so as to reduce the storage capacity of the grid meteorological data in the data signal processor.

2. The novel gridded meteorological data encoding and compression method based on sparse matrices according to claim 1, characterized in that, Step (2) includes: Multiply the gridded meteorological data matrix Matrix0 by the scale factor and convert it into a new matrix Matrix1; Matrix1 ≈ |Matrix0 × scaleFactor + 0.5|; The scale factor is set to 10.

3. The novel gridded meteorological data encoding and compression method based on sparse matrices according to claim 1, characterized in that, Step (3) includes extracting the data from the first row and first column of the gridded meteorological data field [0, 0] in the new matrix Matrix1 as the data encoding reference value: Value0 = Matrix1[0,0].

4. The novel gridded meteorological data encoding and compression method based on sparse matrices according to claim 1, characterized in that, The calculation formulas for each value of matrix Matrix2 in step (4) are as follows: Each Matrix2 value is recorded as the difference between adjacent grid points.

5. The novel gridded meteorological data encoding and compression method based on sparse matrices according to claim 1, characterized in that, Step (5) involves traversing matrix Matrix2, finding the maximum value mx and the minimum value mn in the array, and obtaining the overall range of the data upper and lower limits as follows: range = mx - mn; Number of bits required to calculate the range of data: Here, the value p indicates that the value between mx and mn can be expressed using p bits; Subtract mn from the entire matrix Matrix2 to obtain a new {m×n} resolution gridded meteorological data field matrix Matrix3: Matrix3 = Matrix2 - mn; The new gridded meteorological data field matrix Matrix3 is a positive integer gridded meteorological data field with a minimum of 0 and a maximum of raget. All data is represented by p bits of binary data. Convert each data in matrix Matrix3 into binary data identified by p bits; After converting each data in matrix Matrix3 to binary, store it in a byteArray of p bits in sequence.

6. The novel gridded meteorological data encoding and compression method based on sparse matrices according to claim 1, characterized in that, Step (6) includes defining a 64-byte file header that describes the resolution and data segments of the entire gridded meteorological data field.

7. The novel gridded meteorological data encoding and compression method based on sparse matrices according to claim 1, characterized in that, Step (7) Save matrix Matrix3 as a binary file of {m×n×p / 8+64} bytes and complete the data encoding. Read p bits sequentially from the data segment, convert the binary bits into integer data vi, and calculate the original sparse matrix field: Matrix_rev=(vi+mn)×scaleFactor The entire field is successively added to the values ​​of its neighboring grid points to obtain the original matrix field. The calculation method is as follows: The matrix Matrix_ori is the same as the original matrix Matrix0, and the encoded matrix field is restored to the original matrix.

Images