Novel lattice point meteorological data coding compression method based on sparse matrix

Abstract

The invention discloses a novel lattice point meteorological data coding compression method based on a sparse matrix, and belongs to the technical field of data processing. The method comprises the following steps: reading a lattice point meteorological data matrix Matrix0 with {m * n} resolution from a numerical mode of a lattice point meteorological data field; multiplying the matrix Matrix0 by a proportionality coefficient, and converting the matrix Matrix0 into a matrix Matrix1; extracting a first value of a grid point meteorological data field [0, 0] index as a reference value; creating a matrix Matrix2, and performing assignment point by point; each vertex value of the matrix Matrix2 is converted into a lattice point meteorological data field matrix Matrix3, and p bits are used for coding; carrying out 64-byte file description on the resolution and the numerical value of the grid point meteorological data field; and storing the matrix Matrix3 as a binary file of {m * n * p / 8 + 64} bytes, completing data coding, and converting into an original sparse matrix field.

Description

Technical Field

[0001] The invention belongs to the technical field of data processing, and in particular relates to a novel grid meteorological data encoding and compression method based on a sparse matrix. Background Art

[0002] Numerical forecasting plays an important role in the process of weather forecasting. During the use of numerical forecasting, it is necessary to transmit a large amount of numerical forecast grid meteorological data field data through the network. Therefore, the encoding and storage format of numerical forecast grid data has a huge impact on the storage and transmission of massive meteorological data.

[0003] The China Meteorological Administration defines the MICAPS4 text format to encode and store grid meteorological data, and the World Meteorological Organization defines the binary GRIB format to encode and compress numerical forecast grid meteorological data field data. Both are commonly used storage encoding formats for grid meteorological data.

[0004] However, as the accuracy of numerical forecasts increases, the spatial resolution of grid meteorological data increases, and the original data storage encoding format is increasingly unable to meet the requirements of storing high-resolution grid meteorological data and transmitting it over the network. The MICAPS4 encoding format developed by the China Meteorological Administration records grid meteorological data in the form of text files. As the resolution of grid meteorological data increases, the file size becomes very large. The capacity of a high-precision global grid meteorological field is nearly 200M, which is very unfavorable for data transmission over the network. The GRIB2 data format defined by the World Meteorological Organization adds a data compression mechanism. Although the file capacity is small, the encoding of this format is very complex, the encoding and decoding performance is poor, and the decoding program has high requirements, which is not conducive to the rapid application of grid meteorological data. Summary of the invention

[0005] The present invention provides a novel grid meteorological data encoding and compression method based on a sparse matrix, the improvement of which is that the method comprises

[0006] (1) Read the grid meteorological data matrix Matrix0 with a resolution of {m×n} from the grid meteorological data field numerical model;

[0007] (2) Multiply the matrix Matrix0 by the scaling factor and convert it into a new matrix Matrix1;

[0008] (3) Extract the first value of the grid meteorological data field [0, 0] index as the reference value;

[0009] (4) Create a new matrix Matrix2 and assign values ​​point by point, with the value being the difference between the grid point and the previous adjacent point;

[0010] (5) Convert each vertex value of matrix Matrix2 into grid meteorological data field matrix Matrix3 and encode it using p bits;

[0011] (6) A 64-byte file describing the resolution and value of the entire grid meteorological data field;

[0012] (7) Save the matrix Matrix3 as a binary file of {m×n×p / 8+64} bytes, complete the data encoding, calculate and convert it into the original sparse matrix field.

[0013] Furthermore, the step (1) includes analyzing and processing the grid meteorological data field, reading the grid meteorological data matrix Matrix0 with a resolution of {m×n} from the numerical model of the grid meteorological data field, and defining a scale factor scaleFactor of the data.

[0014] Furthermore, the step (2) comprises

[0015] Multiply the grid meteorological data matrix Matrix0 by the scale factor scaleFactor and convert it into a new matrix Matrix1;

[0016] Matrix1≈|Matrix0×scaleFactor+0.5|;

[0017] Among them, the value of the scale factor scaleFactor is 10.

[0018] Furthermore, the step (3) includes extracting the first row and first column data of the grid meteorological data field [0, 0] in the new matrix Matrix1 as the data encoding reference value:

[0019] Value0=Matrix1[0,0].

[0020] Furthermore, the step (4) includes constructing a new matrix Matrix2 and assigning values ​​point by point. In the matrix Matrix2, the value of each grid point position is as follows:

[0021] Set the first value of the vertex [0,0] as the reference value of the full-field encoding, which defaults to 0;

[0022] Starting from the second row, the value of the first column of each row is the difference between the current position and the first column position of the previous row;

[0023] Starting from the second column of each row, the value of each column is the difference between the current value and the previous value;

[0024] The calculation formulas for each value of matrix Matrix2 are as follows:

[0025]

[0026] The value of each Matrix2 is recorded as the difference between adjacent grid points.

[0027] Furthermore, the step (5) includes traversing the matrix Matrix2, searching for the maximum value mx and the minimum value mn in the array, and obtaining the overall range of the data upper and lower limits as follows:

[0028] range = mx-mn;

[0029] Calculate the number of bits required for the range data range:

[0030]

[0031] The value p indicates that p bits are used to express a value between mx and mn.

[0032] Subtract mn from the matrix Matrix2 as a whole and convert it into a new {m×n} resolution grid meteorological data field matrix Matrix3:

[0033] Matrix3 = Matrix2-mn;

[0034] The new grid meteorological data field matrix Matrix3 has a minimum value of 0 and a maximum value of range, which is a positive integer grid meteorological data field. All data are represented by p-bit binary data.

[0035] Convert each data in the matrix Matrix3 into binary data marked by p bits;

[0036] After converting each data in the matrix Matrix3 into binary, it is stored in byteArray bytes sequentially according to p bits.

[0037] Furthermore, the step (6) includes defining a file header of 64 bytes in length to describe the resolution and data segments of the entire grid meteorological data field.

[0038] Furthermore, the step (7) saves the matrix Matrix3 as a binary file of {m×n×p / 8+64} bytes, completes data encoding, reads p bits in sequence in the data segment, converts the binary bits into integer data vi, and calculates the original sparse matrix field:

[0039] Matrix_rev=(vi+mn)×scaleFactor

[0040] The entire field is added to the adjacent grid point values ​​in turn to obtain the original matrix field. The calculation method is as follows:

[0041]

[0042] The matrix Matrix_ori is the same as the original matrix Matrix0, and restores the encoded matrix field to the original matrix.

[0043] Beneficial effects:

[0044] According to the accuracy requirements of grid meteorological data, the present invention converts the entire grid meteorological data field into an integer, and utilizes the continuity characteristics of adjacent grid points of the grid meteorological data to only record the difference between adjacent grid points, and converts the entire meteorological grid meteorological data field from data to a sparse matrix. For the sparse matrix, the data encoding is utilized to convert each number into a number described by p numbers, thereby greatly enhancing the data compression ratio.

[0045] At the same time, due to the sequential read and write characteristics of the data, the encoding and restoration algorithm of each number depends on the value of the previous adjacent grid point. In the data encoding and decoding process, the data is read and written sequentially, and the dependent adjacent grid points have already gone through the encoding and decoding process, and the processing algorithm for the next grid point has been prepared, so the encoding and decoding speed of the entire data field is also very fast, which is conducive to the transmission and application of grid meteorological data files encoded in this format on the Internet.

[0046] It should be understood that the above general description and the following specific embodiments are only exemplary and illustrative and do not limit the scope of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 The present invention is a flowchart of a novel grid meteorological data encoding and compression method based on a sparse matrix.

[0048] It should be understood that the drawings are not necessarily drawn to scale, presenting somewhat simplified representations of various features illustrative of the basic principles of the present disclosure. The specific design features of the present invention as disclosed herein, including, for example, specific dimensions, orientations, locations, and shapes will be determined in part by the particular intended application and use environment.

[0049] In the drawings, reference numbers refer to the same or equivalent parts of the present invention throughout the several figures of the drawing. DETAILED DESCRIPTION

[0050] Reference will now be made in detail to various embodiments of the present invention, examples of which are illustrated in the accompanying drawings and described below. Although the present invention will be described in conjunction with exemplary embodiments of the present invention, it should be understood that this specification is not intended to limit the present invention to those exemplary embodiments. On the other hand, the present invention is intended to cover not only the exemplary embodiments of the present invention, but also various alternatives, modifiers, equivalents and other embodiments, which may be included in the spirit and scope of the present invention as defined by the appended claims.

[0051] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The specific structures and functions described in the exemplary embodiments of the present invention are for illustrative purposes only. Embodiments according to the concepts of the present invention can be implemented in various forms, and it should be understood that they should not be interpreted as being limited to the exemplary embodiments described in the exemplary embodiments, but include all modifications, equivalents or substitutes included in the spirit and scope of the present invention.

[0052] Throughout the specification, the professional terms used herein are only for describing various exemplary embodiments and are not intended to be limiting. It will be further understood that the terms "include", "comprising", "having", etc., when used in the exemplary embodiments, specifically refer to the presence of the stated parts, steps, operations or elements, but do not exclude the presence or addition of one or more other parts, steps, operations or elements.

[0053] Based on the spatial continuity of meteorological data, the present invention processes grid meteorological data and converts it into a sparse matrix, and then encodes the grid meteorological data based on the sparse matrix, which greatly reduces the storage capacity of the data and is beneficial to the storage and transmission of the data. In addition, the method has simple encoding and fast decoding speed, which is beneficial to the transmission of grid meteorological data through the Internet and the rapid decoding and application of the data by the application end.

[0054] like Figure 1 As shown, the present invention analyzes and processes the grid meteorological data field, converts it into a sparse matrix while retaining a certain accuracy, and then encodes the sparse matrix to reduce the data storage capacity. The specific steps of the present invention include:

[0055] S11, for the grid meteorological data Matrix0 with a resolution of {m×n}, the grid meteorological data field is analyzed and processed, the grid meteorological data matrix Matrix0 with a resolution of {m×n} is read from the numerical mode of the grid meteorological data field, and the scale factor scaleFactor of the data is defined;

[0056] S12, multiply the entire grid meteorological data matrix Matrix0 by the scale factor scaleFactor, round it off to retain the integer part, and convert it into a new {m×n} resolution integer grid meteorological data field matrix Matrix1:

[0057] Matrix1≈|Matrix0×scaleFactor+0.5|;

[0058] The value of the scaleFactor factor can be specified according to the characteristics of the corresponding meteorological elements. According to the practice of the meteorological industry, the value can generally be 10, which is equivalent to retaining 0.1 accuracy of the original data.

[0059] S13, in the new matrix field Matrix1, extract the data of the first row and the first column as the reference value for subsequent data encoding:

[0060] Value0 = Matrix1[0,0];

[0061] S14, construct a new matrix Matrix2, and assign values ​​point by point. In the matrix Matrix2, the value of each grid point position is as follows:

[0062] Set the first value of the vertex [0,0] as the reference value of the full-field encoding, which defaults to 0;

[0063] Starting from the second row, the value of the first column of each row is the difference between the current position and the first column position of the previous row;

[0064] Starting from the second column of each row, the value of each column is the difference between the current value and the previous value.

[0065] The calculation formulas for the values ​​of the entire two-dimensional field matrix Matrix2 are as follows:

[0066]

[0067] By the above algorithm, a new matrix Matrix2 is obtained. Each value of the matrix Matrix2 records the difference between adjacent grid points. Due to the continuity of adjacent grid points of grid meteorological data, each value is a positive or negative integer with a small absolute value.

[0068] S15, traverse the matrix Matrix2, find the maximum value mx and the minimum value mn in the array, and get the overall range of the upper and lower limits of the data:

[0069] range = mx-mn;

[0070] Calculate the number of bits required to fully express the range of data:

[0071]

[0072] The value p means that all values ​​between mx and mn can be fully expressed using p bits.

[0073] S16, subtract mn from the matrix Matrix2 as a whole, and convert it into a new {m×n} resolution grid meteorological data field matrix Matrix3:

[0074] Matrix3 = Matrix2-mn;

[0075] The new grid meteorological data field is a positive integer grid meteorological data field with a minimum of 0 and a maximum of range. All data can be represented by p-bit binary data.

[0076] Apply for a byte array byteArray of length len, the length len is:

[0077]

[0078] Convert each data in the matrix Matrix3 into binary data marked by p bits;

[0079] After converting each data in the matrix Matrix3 into binary, it is stored in byteArray bytes sequentially according to p bits.

[0080] S17, define a 64-byte file header to describe the data segment. The format of each byte in the data header is designed as follows:

[0081] 1 to 8 bytes: 8-byte file mark, used as a file format mark, the default content is "WIZGRD01";

[0082] 9-12: 4 bytes, floating point number, starting longitude;

[0083] 13-16: 4 bytes, floating point number, end longitude;

[0084] 17-20: 4 bytes, floating point number, starting latitude;

[0085] 21-24: 4 bytes, floating point number, end latitude;

[0086] 25-28: 4 bytes, floating point number, longitude interval;

[0087] 29~32: 4 bytes, floating point number, latitude interval;

[0088] 33-36: 4 bytes, integer, number of grid points in the latitude direction;

[0089] 37-40: 4 bytes, integer, number of grid points in the longitude direction;

[0090] 42-44: 4 bytes, floating point number, data base value, Value0;

[0091] 45-48: 4 bytes, floating point number, data scale factor coefficient, scaleFactor;

[0092] 49-52: 4 bytes, unsigned short type, the number of bits occupied by each number;

[0093] 53-56: 4 bytes, integer, the bit length occupied by each digit, p;

[0094] 57-58: 2 bytes, short type, minimum value mn;

[0095] 59-60: 2 bytes, short type, maximum value mx;

[0096] 61~64: 4 bytes, integer, data segment length len;

[0097] The data segment starts from byte 65, and its length and content are defined as follows:

[0098] 65~(64+len): len bytes, the encoded data content byteArray;

[0099] S18, after encoding processing, the data capacity of the entire grid meteorological data field matrix Matrix3 is reduced from {m×n×4} bytes to {m×n×p÷8+64} bytes while maintaining data accuracy. It is generally about 1 / 8 of the original data capacity. In extreme cases such as precipitation phase, it is 1 / 16 of the original data capacity.

[0100] The encoded data is decoded in sequence according to the data storage order during the decoding process. After reading p bits in sequence from the data segment, the binary bits are converted into integer data vi, and then converted into the original sparse matrix field through the following calculation:

[0101] Matrix_rev=(vi+mn)×scaleFactor

[0102] The original matrix field can be obtained by adding the entire field to the adjacent grid point values ​​in sequence. The calculation method is as follows:

[0103]

[0104] The matrix Matrix_ori is the same as the original matrix Matrix0. After this decoding process, the encoded matrix field is restored to the original matrix.

[0105] The above-mentioned embodiments of the present application can be implemented in various hardware, software coding or a combination of the two. For example, the embodiments of the present application can also represent the program code for executing the above-mentioned method in a data signal processor. The present application can also relate to the various functions performed by a computer processor, a digital signal processor, a microprocessor or a field programmable gate array. The above-mentioned processor can be configured according to the present application to perform specific tasks, which are completed by executing machine-readable software code or firmware code that defines the specific method disclosed by the present application. The software code or firmware code development can represent different programming languages ​​and different formats or forms. Different target platform compiled software codes can also be represented. However, the different code styles, types and languages ​​of the software code for performing tasks according to the present application and other types of configuration codes do not depart from the spirit and scope of the present application.

[0106] The foregoing description of specific exemplary embodiments of the present invention has been presented for the purpose of illustration and description. It is not intended to exclude or limit the present invention to the precise form disclosed, and it is obvious that many modifications and changes are feasible in view of the above teachings. Exemplary embodiments are selected and described to explain certain principles of the present invention and their practical application, so that other technicians in the field can make or use various exemplary embodiments of the present invention, and various alternatives and modifications thereof. It is intended that the scope of the present invention will be defined by the claims attached hereto and their equivalents.

[0107] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present invention, but the present invention is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A new grid meteorological data encoding and compression method based on sparse matrix, It is characterized in that The method comprises: (1) Read the grid meteorological data matrix Matrix0 with a resolution of {m×n} from the grid meteorological data field numerical model; (2) Multiply the matrix Matrix0 by the scaling factor and convert it into a new matrix Matrix1; (3) Extract the first value of the grid meteorological data field [0, 0] index as the reference value; (4) Create a new matrix Matrix2 and assign values ​​point by point, with the value being the difference between the grid point and the previous adjacent point; (5) Convert each vertex value of matrix Matrix2 into grid meteorological data field matrix Matrix3 and encode it using p bits; (6) A 64-byte file describing the resolution and value of the entire grid meteorological data field; (7) Save the matrix Matrix3 as a binary file of {m×n×p / 8+64} bytes, complete the data encoding, calculate and convert it into the original sparse matrix field.

2. A novel grid meteorological data encoding and compression method based on sparse matrix according to claim 1, It is characterized in that The step (1) includes analyzing and processing the grid meteorological data field, reading the grid meteorological data matrix Matrix0 with a resolution of {m×n} from the numerical model of the grid meteorological data field, and defining the scale factor scaleFactor of the data.

3. A novel grid meteorological data encoding and compression method based on sparse matrix according to claim 1, It is characterized in that The step (2) comprises: Multiply the grid meteorological data matrix Matrix0 by the scale factor scaleFactor and convert it into a new matrix Matrix1; Matrix1≈|Matrix0×scaleFactor+0.5|; Among them, the value of the scale factor scaleFactor is 10.

4. The novel grid meteorological data encoding and compression method based on sparse matrix according to claim 1, It is characterized in that The step (3) includes extracting the first row and first column data of the grid meteorological data field [0, 0] as the data encoding reference value in the new matrix Matrix1: Value0=Matrix1[0,0].

5. The novel grid meteorological data encoding and compression method based on sparse matrix according to claim 1, It is characterized in that The step (4) includes constructing a new matrix Matrix2 and assigning values ​​point by point. In the matrix Matrix2, the value of each grid point position is as follows: Set the first value of the vertex [0,0] as the reference value of the full-field encoding, which defaults to 0; Starting from the second row, the value of the first column of each row is the difference between the current position and the first column position of the previous row; Starting from the second column of each row, the value of each column is the difference between the current value and the previous value; The calculation formulas for each value of matrix Matrix2 are as follows: The value of each Matrix2 is recorded as the difference between adjacent grid points.

6. A novel grid meteorological data encoding and compression method based on sparse matrix according to claim 1, It is characterized in that The step (5) includes traversing the matrix Matrix2, searching for the maximum value mx and the minimum value mn in the array, and obtaining the overall range of the data upper and lower limits as follows: range = mx-mn; Calculate the number of bits required for the range data range: The value p indicates that p bits are used to express a value between mx and mn. Subtract mn from the matrix Matrix2 as a whole and convert it into a new {m×n} resolution grid meteorological data field matrix Matrix3: Matrix3 = Matrix2-mn; The new grid meteorological data field matrix Matrix3 has a minimum value of 0 and a maximum value of range, which is a positive integer grid meteorological data field. All data are represented by p-bit binary data. Convert each data in the matrix Matrix3 into binary data marked by p bits; After converting each data in the matrix Matrix3 into binary, it is stored in byteArray bytes sequentially according to p bits.

7. The novel grid meteorological data encoding and compression method based on sparse matrix according to claim 1, It is characterized in that The step (6) includes defining a file header with a length of 64 bytes to describe the resolution and data segments of the entire grid meteorological data field.

8. The novel grid meteorological data encoding and compression method based on sparse matrix according to claim 1, It is characterized in that The step (7) saves the matrix Matrix3 as a binary file of {m×n×p / 8+64} bytes, completes data encoding, reads p bits in sequence in the data segment, converts the binary bits into integer data vi, and calculates the original sparse matrix field: Matrix_rev=(vi+mn)×scaleFactor The entire field is added to the adjacent grid point values ​​in turn to obtain the original matrix field. The calculation method is as follows: The matrix Matrix_ori is the same as the original matrix Matrix0, and restores the encoded matrix field to the original matrix.

Images