Weather forecast data dimension reduction compression method

Abstract

The invention discloses a weather forecast data dimension reduction compression method. The method comprises the following steps: decoding and segmenting original data; the data volume dimension reduction step comprises the steps of obtaining data in the data volume layer by layer and line by line, and arranging and splicing the data into a data list L; a duplicate value processing step: merging continuously appearing duplicate values to obtain a merged data list L ', and connecting data in the merged data list L' by adopting connection characters to form a temporary character string T; in the coding step, part of character combinations in the temporary character string T correspond to coded characters in a coding table, and a coded character string T'is constructed and formed; a compression transmission step: splicing all the coded character strings to form a one-dimensional data set S3; and carrying out compression transmission on the data set S3. According to the meteorological data dimension reduction compression method, complete one-dimensional processing is carried out on each data body, and it is ensured that data cannot be missing. And meanwhile, the length of the data is reduced, and the compression ratio is greatly improved compared with original data. And 100% restoration can be realized in the data restoration process, and complete and error-free original data can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of meteorological data processing, and in particular to a dimensionality reduction and compression method for meteorological forecast data. Background Art

[0002] The meteorological numerical model is a large-scale simulation calculation program running on a supercomputer. It uses real data from meteorological stations and satellite detection around the world as the initial field, and generates global weather forecast data through a preset calculation process. The data contains dozens of meteorological variables such as temperature, air pressure, humidity, wind direction, wind speed, precipitation, etc. Each element is arranged in chronological order, so the meteorological forecast data is a multi-dimensional nested structured data. The first layer contains two dimensions: variables and time, and the second layer contains three dimensions: longitude, latitude, and altitude.

[0003] The current mainstream international meteorological forecasting agencies include the China Meteorological Administration, the European Center for Medium-Range Weather Forecasts, and the National Center for Environmental Prediction of the United States. The meteorological numerical models developed by these agencies have reached a horizontal resolution of 0.125 degrees, nearly 50 vertical layers, a time interval of 1-3 hours, and a forecast of 10 days. They are still developing in the direction of higher resolution, longer forecast days, and higher accuracy. High-resolution meteorological forecast data is large in size (grib2 format, about 14 GB to 15GB per time). For meteorological users using wired transmission networks, this data volume is acceptable. However, for meteorological users using satellite wireless transmission networks, such as ships at sea, high mountains and islands, the network bandwidth is low and expensive, and large-volume data transmission is difficult to achieve. Therefore, it is necessary to compress the forecast data generated by the meteorological numerical model.

[0004] At present, data compression algorithms for meteorological data can be divided into two types: lossless compression and lossy compression. Lossless compression can retain all the information in the source data, but the compression ratio is not high, generally 2:1 to 5:1. The main lossless compression codes include predictive coding, transform coding, vector coding, arithmetic coding, etc. Lossful compression refers to the use of compressed data for reconstruction, and the data obtained is different from the source data, but it will not cause people to misunderstand the information expressed by the source data. Lossy compression can obtain a higher compression ratio. Chinese patent CN116683915A proposes a meteorological data compression method, system and medium, which discards a large amount of detailed data within a certain scale and all data smaller than the scale. Although for users, the general situation of the weather conditions can be restored from these data, all details cannot be fully restored. It is essentially a lossy compression algorithm.

[0005] Based on this, how to achieve lossless compression of meteorological data is the main technical problem to be solved by the present invention. Summary of the invention

[0006] In order to solve the technical problem that the existing data compression algorithm of meteorological data causes data loss and makes it impossible to completely restore the detailed information, the present invention proposes a dimensionality reduction and compression method for meteorological forecast data, which can solve the above problem.

[0007] In order to solve the above technical problems, the present invention adopts the following technical solutions to achieve the above problems: A method for reducing dimension and compressing weather forecast data, comprising: The step of decoding and segmenting the original data is to decode and segment the original data according to the two dimensions of time and meteorological variables, and obtain a plurality of data bodies arranged according to time and meteorological variables, wherein the data in the data bodies are arranged according to the three dimensions of longitude, latitude and altitude, and the meteorological variables include at least temperature, air pressure, humidity, wind direction, wind speed and precipitation; The step of reducing the dimension of the data body includes: obtaining the data in the data body layer by layer and row by row and arranging and splicing them into a data list L, wherein the data list L is arranged in a row or a column, and each data body corresponds to a data list; The repeated value processing step includes: respectively detecting repeated data in each data list, and if there are k consecutive identical data Float, representing the k consecutive identical data as Float*k and replacing them to form a merged data list L'; Connect the data in the merged data list L' using a connection character to form a temporary string T, wherein the temporary string T includes a plurality of characters; An encoding step, wherein some character combinations in the temporary string T have corresponding encoding characters in the encoding table, the corresponding encoding characters are found in the encoding table, and the encoding characters are used to replace the character combinations to construct an encoding string T', wherein the encoding characters in the encoding table are different from any characters in the temporary string; In the compression transmission step, all the encoded strings are connected end to end in sequence to form a one-dimensional data set S3; The data set S3 is compressed and transmitted.

[0008] In some embodiments, the data body dimensionality reduction step also includes obtaining the total number of rows l, the total number of columns m, the total number of layers n in any data body, and the time t for obtaining all data bodies to form the basic information code Info.

[0009] In some embodiments, in the encoding step, the method of encoding a partial character combination in the temporary string T includes: Count the occurrence frequencies of the integer parts of all the data in the combined data list L' to obtain the first M high-frequency integers; The character combinations of the M high-frequency integers in the merged data list are respectively encoded, wherein the character combinations include a combination of a connecting character, a high-frequency integer and a decimal point, and the character combinations correspond to any coded characters in the coding table, and different character combinations correspond to different coded characters.

[0010] In some embodiments, the encoding step also includes recording the correspondence between the character combination and the encoded character, and storing it in the basic information code Info.

[0011] In some embodiments, the numeric characters in the temporary string T that are after the decimal point character and before the next connecting character are not encoded.

[0012] In some embodiments, the position order of all coded characters in the coding table is fixed, and the number of coded characters is M. The encoding step also includes sorting the first M high-frequency integers from large to small according to the frequency of occurrence, and any character combination of high-frequency integers corresponds to the coded character at the same position in the coding table.

[0013] In some embodiments, for the first character in the encoded string T', if it is an encoded character, the connecting character in the corresponding character combination is removed during decompression and restoration.

[0014] In some embodiments, in the compression transmission step, the data set S3 is compressed into any format of zip, rar, 7z, or tgz.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: the dimensionality reduction and compression method of the weather forecast data of the present invention decodes and segments the complete original data by variables and time in the original data decoding and segmentation step, and the data body dimensionality reduction step performs complete one-dimensional processing on each data body to ensure that the data will not be missing. In the repeated value processing step, the same value merging can be completely restored to k continuous data Float according to "Float*k"; taking advantage of the particularly high repetition rate of the integer part of the same meteorological variable, in the encoding step, the character combination (including multiple characters) corresponding to the integer part in the meteorological variable data is represented by a coded character, the redundant part is removed, and the length of the data is further reduced. The compression ratio relative to the original data is greatly improved, and it can achieve 100% restoration in the data restoration process to obtain complete and error-free original data.

[0016] Other features and advantages of the present invention will become more apparent after reading the detailed description of the embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1It is a schematic diagram of arrangement of original data in two dimensions of time and meteorological variables in one embodiment of the weather forecast data dimensionality reduction and compression method proposed by the present invention; Figure 2 It is a schematic diagram of arranging a single data body in three dimensions of longitude, latitude and altitude in one embodiment of the weather forecast data dimensionality reduction and compression method proposed by the present invention; Figure 3 It is a schematic diagram of dimensionality reduction of a single data body in one embodiment of the weather forecast data dimensionality reduction and compression method proposed by the present invention; Figure 4 It is a schematic diagram of data states corresponding to different processing steps in an embodiment of the weather forecast data dimensionality reduction and compression method proposed by the present invention. DETAILED DESCRIPTION

[0018] The specific implementation modes of the present invention will be further described in detail below in conjunction with the accompanying drawings.

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The weather forecast data dimensionality reduction and compression method of this embodiment includes: The raw data decoding and segmentation step decodes and segments the raw data according to the two dimensions of time and meteorological variables, and obtains multiple data bodies arranged according to time and meteorological variables. The data in the data body are arranged according to the three dimensions of longitude, latitude and altitude. The meteorological variables include at least temperature, air pressure, humidity, wind direction, wind speed and precipitation.

[0021] like Figure 1 As shown in the figure, the original grib2 data is decoded and segmented according to the two dimensions of variables and time, and a data set S1 consisting of multiple data volumes Cube is obtained: S1=[ [C 11 , C 12 , C 13 ……C 1t ], [C 21 , C 22 , C 23 ……C 2t ], [C 31 , C 32 , C33 ……C 3t ], … [C v1 , C v2 , C v3 ……C vt ], ].

[0022] like Figure 2 As shown, for any data body Cube in the data set S1, its internal data is arranged in order according to the three dimensions of longitude x-latitude y-height z. In addition to the core data required by this embodiment, each point also includes auxiliary information such as longitude, latitude, height, time, data unit, data type, etc., such as Figure 3 shown.

[0023] This embodiment also includes a data volume dimensionality reduction step, which specifically includes: acquiring the data in the data volume layer by layer and row by row and arranging and splicing them into a data list L, the data list L is arranged in a row or a column, and each data volume corresponds to a data list.

[0024] It is understandable that the data list L is one-dimensional data. By reducing the dimension of the data body, the amount of data calculation can be greatly saved. By performing complete one-dimensional processing on each data body Cube, it is ensured that data will not be missing.

[0025] The three-dimensional data body Cube is transformed into a one-dimensional data List. Since any row of the data list [L v1 , L v2 , L v3 ...L vt ] corresponds to only one meteorological variable, and its variance is small, that is, the deviation from the average value of the row is small, while the deviation between rows (representing different meteorological variables) is large. For example, a row of pressure data may fluctuate between 1000-1100hPa, and a row of temperature data may fluctuate between 0-20. Based on this, the data lists of each row are connected in sequence to form a single variable data list covering all time. At this time, the data set S21 is transformed into the following data set S22: S22=[L1, L2, L3, ... L v , Info ].

[0026] For any data list L in the data set S22, since it only corresponds to a certain meteorological variable, the values ​​are most likely to be the same in adjacent time or space, that is, there are a large number of duplicate data in any data list L, and the duplicate data will occupy a large amount of space. In order to reduce the amount of transmitted data, this embodiment also includes a duplicate value processing step for merging and transmitting the duplicate data to reduce the amount of transmitted data. The merged data is marked so that the receiving end can perform 100% restoration after receiving it to achieve lossless transmission.

[0027] The repeated value processing step in this embodiment includes: respectively detecting repeated data in each data list, and if there are k consecutive identical data Float, then the k consecutive identical data are represented as Float*k and replaced to form a merged data list L'. Figure 4 As shown in the figure, in a row of air pressure data, there are 4 consecutive 1018.73 and 3 consecutive 1018.45, which are replaced by 1018.73*4 and 1018.45*3 respectively. When restoring the data, the receiving end can completely restore it to k consecutive Floats according to "Float*k".

[0028] The data in the merged data list L' are connected using a connection character to form a temporary string T, wherein the temporary string T includes a plurality of characters.

[0029] For example, in some embodiments, the English comma "," is used as a connecting character, and all data in L' are sequentially connected with the English comma "," to form a temporary string T.

[0030] Since the merged data list L' contains multiple numbers, the computer will pre-allocate a certain amount of memory space when storing each number to cope with its possible precision changes, resulting in a large overall memory usage. However, the memory occupied by the string T is very small. In this embodiment, by using a connection character to sequentially connect all the data in L' to form a temporary string T, storage space can be greatly saved.

[0031] For example, when the computer treats 3.1415 as a decimal, it may allocate 8 or 16 bits or even more data space in advance. However, if it is treated as a string, it will generally only allocate 6 bits.

[0032] In this embodiment, the temporary character string T includes five characters: numbers, English comma “,”, decimal point “.”, asterisk “*”, and minus sign “-”.

[0033] For the data of the same meteorological variable worldwide, there are still a large number of integers that appear repeatedly. In order to further shorten the length of the temporary character string T, this embodiment also includes an encoding step, in which a single character is used to replace multiple characters with a high repetition rate, thereby shortening the length of T.

[0034] Considering that the integer part of the meteorological variable value is more repeated, while the decimal part is more random, it is preferred in this scheme to encode only some character combinations, that is, to replace the character combinations with higher repetition rate with single characters.

[0035] In some embodiments, the encoding step includes: some character combinations in the temporary string T have corresponding coded characters in the coding table, searching the corresponding coded characters from the coding table, and using the coded characters to replace the character combinations to construct a coded string T'.

[0036] The coded characters in the coding table are different from any characters in the temporary string to avoid confusion during restoration.

[0037] By replacing the part of the character combination with a higher repetition rate with a single character, the length of T can be further shortened.

[0038] In the compression transmission step, all the encoded character strings are sequentially connected end to end to form a one-dimensional data set S3. The data set S3 is compressed and transmitted.

[0039] For the data set S3, it contains v+1 strings, which are compressed into a compressed package using a common compression algorithm and can be used for transmission. After the user receives the compressed data, he can reversely restore the data according to the above steps.

[0040] In some embodiments, the data body dimensionality reduction step also includes obtaining the total number of rows l, the total number of columns m, the total number of layers n in any data body, and the time t for obtaining all data bodies to form the basic information code Info.

[0041] Since all Cubes are split from regular original weather, l, m, n of each Cube are the same, and t is synchronously increased. Based on this, l, m, n, and t form the basic information code Info, where l, m, and n only retain one group, and t retains all groups, that is: Info={ l: total number of rows, m: total number of columns, n: total number of layers, t:[t1, t2, t3...... t t ] }.

[0042] At this point, the three-dimensional data body Cube is transformed into a combination of List and Info, and the data set S1 is transformed into the following data set S21: S21=[ [L 11 , L 12 , L 13 ...L 1t ], [L 21 , L 22 , L 23 ...L 2t ], [L 31 , L 32 , L 33 ...L 3t ], … [L v1 , L v2 , L v3 ...L vt ], Info ].

[0043] The code table can be constructed in advance, and the coded characters are stored inside it. For example, construct a code table code as follows: code="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz`~!@#$%^&()_+=[]{}\|;':" / <>?".

[0044] In the above example, the code contains a total of 80 characters, including 26 uppercase English letters, 26 lowercase English characters, and 28 English punctuation marks (excluding English commas, decimal points, asterisks, and minus signs).

[0045] In some embodiments, in the encoding step, the method for encoding a partial character combination in the temporary string T includes: The occurrence frequencies of the integer parts of all the data in the merged data list L' are counted to obtain the first M high-frequency integers.

[0046] The character combinations of the M high-frequency integers in the merged data list are respectively encoded, the character combination includes a combination of a connecting character, a high-frequency integer and a decimal point, the character combination corresponds to any coded character in the coding table, and different character combinations correspond to different coded characters.

[0047] In some embodiments, the encoding step also includes recording the correspondence between the character combination and the encoded character, and storing it in the basic information code Info.

[0048] Since the decimal part of the meteorological variable value is relatively random and has a low repetition rate, in some embodiments, in the encoding step, the numeric characters (i.e., the decimal part of the meteorological variable value) in the temporary string T that are located after the decimal point character and before the next connecting character are not encoded.

[0049] In some embodiments, the position order of all coded characters in the coding table is fixed, and the number of coded characters is M. The encoding step also includes sorting the first M high-frequency integers from large to small according to the frequency of occurrence, and any character combination of high-frequency integers corresponds to the coded character at the same position in the coding table.

[0050] Taking M=80 as an example, count the frequencies of the integer parts of all data in L', arrange them from large to small according to the frequencies, and get the first 80 high-frequency integers.

[0051] Replace the above high-frequency integers one by one: For the i-th high-frequency integer D, whose character combination in T is ",D.", then replace all ",D." in T with the i-th character in code to obtain T', and record the corresponding relationship between the integer and the code as R.

[0052] like Figure 4 As shown, count the number of times all integers appear in L' and arrange them by frequency, which are: 1015: 2 times; 1018: 2 times; 1013: 1 time.

[0053] They correspond to the first three characters [A, B, C] in the code respectively, and the corresponding relationship R is: A corresponds to ",1015."; B corresponds to ",1018."; C corresponds to ",1013." The replaced L' is: C42A46B73*4B45*3A47.

[0054] A record is added to Info to indicate the correspondence R between integers and codes, which is used to reverse restore T'. At this time, Info becomes Info add ,Right now: Info add ={ l: total number of rows, m: total number of columns, n: total number of layers, t:[t1, t2, t3...... t t ], R:[",D1.", ",D2.", ",D3.", ... ",D 80 .”] }.

[0055] The data set S22 is transformed into the following data set S3: S3=[T'1, T'2, T'3, ... T' v , Info add ].

[0056] Using Info add The R in S3 and the predefined code can restore the T' in S3 to the original data one by one. Therefore, the restored data is the complete and error-free original data.

[0057] In some embodiments, for the first character in the encoded string T', if it is an encoded character, the connecting character in the corresponding character combination is removed during decompression and restoration.

[0058] In some embodiments, in the compression transmission step, the data set S3 is compressed into any format of zip, rar, 7z, or tgz.

[0059] Using the compression method of this embodiment, for meteorological data in grib2 format, in addition to the core data value, other auxiliary information such as longitude, latitude, altitude, time, data unit, data type, etc. are redundant parts, accounting for about 33%. This solution omits the redundant parts and replaces them with the integrated basic information code Info. Therefore, after completing the data body dimensionality reduction step, the compression ratio relative to the original data can reach 1.5:1.

[0060] For the data list L formed by dimensionality reduction v If the weather system is relatively stable, the meteorological data has a small change range and a small variance, then L v Convert to T v The more repeated values ​​are processed in the process, the greater the compression ratio is, and vice versa. v Compressed to T v The compression ratio of the original data has a certain fluctuation, generally between 2:1 and 3:1. At this point, after completing the repeated value processing step, the compression ratio relative to the original data can reach 3:1 to 4.5:1.

[0061] For the data set S3, a common compression algorithm is used to compress it into zip format. Since the data set S3 contains irregular characters, the compression ratio is generally 2:1. So far, the compression ratio relative to the original data can reach 6:1 to 9:1.

[0062] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by ordinary technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for dimensionality reduction and compression of weather forecast data, characterized in that: include: The step of decoding and segmenting the original data is to decode and segment the original data according to the two dimensions of time and meteorological variables, and obtain a plurality of data bodies arranged according to time and meteorological variables, wherein the data in the data bodies are arranged according to the three dimensions of longitude, latitude and altitude, and the meteorological variables include at least temperature, air pressure, humidity, wind direction, wind speed and precipitation; The step of reducing the dimension of the data body includes: obtaining the data in the data body layer by layer and row by row and arranging and splicing them into a data list L, wherein the data list L is arranged in a row or a column, and each data body corresponds to a data list; The repeated value processing step includes: respectively detecting repeated data in each data list, and if there are k consecutive identical data Float, representing the k consecutive identical data as Float*k and replacing them to form a merged data list L'; Connect the data in the merged data list L' using a connection character to form a temporary string T, wherein the temporary string T includes a plurality of characters; An encoding step, wherein some character combinations in the temporary string T have corresponding encoding characters in the encoding table, the corresponding encoding characters are found in the encoding table, and the encoding characters are used to replace the character combinations to construct an encoding string T', wherein the encoding characters in the encoding table are different from any characters in the temporary string; In the compression transmission step, all the encoded strings are connected end to end in sequence to form a one-dimensional data set S3; The data set S3 is compressed and transmitted.

2. The method for dimensionality reduction and compression of weather forecast data according to claim 1, characterized in that: The data volume dimensionality reduction step also includes obtaining the total number of rows l, the total number of columns m, the total number of layers n in any data volume, and the time t for obtaining all data volumes to form the basic information code Info.

3. The method for dimensionality reduction and compression of weather forecast data according to claim 2, characterized in that: In the encoding step, the method for encoding a partial character combination in the temporary character string T includes: Count the occurrence frequencies of the integer parts of all the data in the combined data list L' to obtain the first M high-frequency integers; The character combinations of the M high-frequency integers in the merged data list are respectively encoded, wherein the character combinations include a combination of a connecting character, a high-frequency integer and a decimal point, and the character combinations correspond to any coded characters in the coding table, and different character combinations correspond to different coded characters.

4. The method for dimensionality reduction and compression of weather forecast data according to claim 3, characterized in that: The encoding step also includes recording the corresponding relationship between the character combination and the encoded character, and saving it in the basic information code Info.

5. The method for dimensionality reduction and compression of weather forecast data according to claim 3, characterized in that: In the encoding step, the numeric characters in the merged data list in the temporary string T that are located after the decimal point character and before the next connection character are not encoded.

6. The method for dimensionality reduction and compression of weather forecast data according to claim 3, characterized in that: The position order of all the coded characters in the coding table is fixed, and the number of coded characters is M. The encoding step also includes sorting the first M high-frequency integers from large to small according to the frequency of occurrence, and any character combination of high-frequency integers corresponds to the coded character at the same position in the coding table.

7. The method for dimensionality reduction and compression of weather forecast data according to claim 3, characterized in that: For the first character in the encoded string T', if it is an encoded character, the connecting character in the corresponding character combination is removed during decompression and restoration.

8. The method for dimensionality reduction and compression of weather forecast data according to any one of claims 1 to 7, characterized in that: In the compression transmission step, the data set S3 is compressed into any format of zip, rar, 7z, or tgz.

Images