Data processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of performing data type conversion on a first type of meteorological data of which the accuracy is first accuracy to obtain converted first type of meteorological data, and compressing the converted first type of meteorological data according to a first compression ratio to obtain first compressed meteorological data. And for the second type of meteorological data of which the accuracy is second accuracy and the meteorological data is non-negative, directly compressing the second type of meteorological data according to a second compression ratio to obtain second compressed meteorological data. According to the meteorological data compression method and device, different types of meteorological data are compressed in different modes, the compression mode is simple, subsequent decompression is facilitated, and waste of computing resources and time is reduced.

Description

Technical Field

[0001] The present application relates to the field of meteorological data processing, and in particular to a data processing method, device, electronic equipment, storage medium and program product. Background Art

[0002] For large-scale data, such as meteorological data used to generate background fields, the meteorological data needs to be compressed during storage and transmission, where the background field refers to the initial state or initial conditions used to initialize the numerical weather forecast model.

[0003] At present, meteorological data can be compressed according to the gridded binary (grib) encoding method (such as grib2) to obtain a compressed file. However, since the encoding method is relatively complex, the method of reading the compressed file in subsequent processing is also relatively complex, resulting in a waste of computing resources and time. Summary of the invention

[0004] The present application provides a data processing method, device, electronic device, storage medium and program product to solve the problem that the grid binary encoding method is relatively complex, resulting in a relatively complex method of reading compressed files in subsequent processing, resulting in a waste of computing resources and time.

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] Acquire meteorological data to be compressed, the meteorological data to be compressed includes first-category meteorological data and second-category meteorological data; wherein the precision of the first-category meteorological data is a first precision; the precision of the second-category meteorological data is a second precision, and the second-category meteorological data is a non-negative number; wherein the second precision is greater than the first precision;

[0007] Converting the data type of the first-category meteorological data to obtain converted first-category meteorological data; and compressing the converted first-category meteorological data according to a first compression ratio to obtain first compressed meteorological data;

[0008] compressing the second type of meteorological data according to a second compression ratio to obtain second compressed meteorological data;

[0009] The first compressed meteorological data and the second compressed meteorological data are stored.

[0010] In this embodiment, different types of meteorological data can be compressed in different ways. For example, for the first type of meteorological data with a first precision, the first type of meteorological data can be converted into a data type to obtain the converted first type of meteorological data, and then the converted first type of meteorological data can be compressed according to a first compression ratio to obtain first compressed meteorological data. For the second type of meteorological data with a second precision and the meteorological data is a non-negative number, the second type of meteorological data can be directly compressed according to a second compression ratio to obtain second compressed meteorological data. This compression method is relatively simple, convenient for subsequent decompression, and reduces the waste of computing resources and time.

[0011] Optionally, in the method as described above, the first category of meteorological data includes meteorological data corresponding to a plurality of first category variables;

[0012] The first-category meteorological data is subjected to data type conversion to obtain the converted first-category meteorological data, including:

[0013] For any meteorological data corresponding to the first-category variable, obtain a reference offset and a scaling factor corresponding to the first-category variable; perform data type conversion on each meteorological data corresponding to the first-category variable according to the reference offset and the scaling factor, and obtain converted meteorological data corresponding to a plurality of first-category variables;

[0014] The converted first-category meteorological data is determined according to the converted meteorological data corresponding to the plurality of first-category variables.

[0015] In this embodiment, for the first type of meteorological data, data type conversion is performed using the reference offset and scale factor corresponding to each first type variable, thereby reducing the storage requirement of the data without affecting the accuracy of the data.

[0016] Optionally, in the method described above, the first type of meteorological data is floating point data; obtaining the reference offset and the scale factor corresponding to the first type of variable includes:

[0017] Determine the maximum and minimum values ​​in the meteorological data corresponding to the first type of variables;

[0018] Determine an average value of the maximum value and the minimum value as the reference offset;

[0019] Get the number of values ​​contained in the consecutive bits occupied by the short integer data;

[0020] determining a first difference between the minimum and maximum values;

[0021] The proportional factor is determined according to the number of values ​​and the first difference.

[0022] In this embodiment, the scale factor is determined by the maximum value, the minimum value, and the number of values ​​accommodated by the continuous number of bits occupied by the short integer data, and the reference offset is determined according to the maximum value and the minimum value, so as to achieve a balance between data accuracy and storage efficiency.

[0023] Optionally, according to the method described above, data type conversion is performed on each meteorological data in the meteorological data corresponding to the first category variables according to the reference offset and the scale factor to obtain converted meteorological data corresponding to a plurality of first category variables, including:

[0024] For each meteorological data in the meteorological data corresponding to the first category variable, a second difference between the meteorological data and the reference offset is determined; and a ratio of the second difference to the proportional factor is determined as the converted meteorological data.

[0025] In this embodiment, the storage requirement of data is reduced by converting floating point numbers into integers.

[0026] Optionally, in the method as described above, storing the first compressed meteorological data and the second compressed meteorological data comprises:

[0027] respectively taking the reference offsets corresponding to the plurality of first-category variables as first attribute values, and taking the scale factor as second attribute value;

[0028] storing the first compressed meteorological data and the first attribute values ​​and the second attribute values ​​corresponding to the plurality of first-category variables; the first attribute values ​​and the second attribute values ​​corresponding to the plurality of first-category variables are used to decompress the first compressed meteorological data;

[0029] The second compressed meteorological data is stored.

[0030] In this embodiment, when storing the first compressed meteorological data, the reference offset and the scale factor of each first-category variable may be stored as attribute values, and the second compressed meteorological data may be stored to facilitate subsequent decompression.

[0031] Optionally, the method as described above further comprises:

[0032] decompressing the first compressed meteorological data to obtain first decompressed meteorological data;

[0033] Obtaining third attribute values ​​of a plurality of first-category variables, wherein the third attribute values ​​are used to indicate whether the meteorological data corresponding to the first-category variables include negative values;

[0034] Converting a negative value in the meteorological data corresponding to the target variable of the first decompressed meteorological data into a positive value to obtain the converted first decompressed meteorological data; the target variable being a first attribute indicates that the meteorological data corresponding to the target variable includes a negative value;

[0035] According to the first attribute values ​​and the second attribute values ​​corresponding to the multiple first-category variables, data type conversion is performed on the converted first decompressed meteorological data to obtain the first-category meteorological data.

[0036] In this embodiment, the first type of meteorological data is compressed after data type conversion, which may compress some positive numbers into negative numbers. Therefore, after decompressing the meteorological data of this type of variables, it is necessary to positiveen the negative numbers and then perform data type conversion based on the two attribute values ​​determined during the data type conversion before compression to improve the accuracy of the data.

[0037] In a second aspect, the present application provides a data processing device, including:

[0038] An acquisition module, used for acquiring meteorological data to be compressed, wherein the meteorological data to be compressed includes first-category meteorological data and second-category meteorological data; wherein the accuracy of the first-category meteorological data is a first accuracy; the accuracy of the second-category meteorological data is a second accuracy, and the second-category meteorological data is a non-negative number; wherein the second accuracy is greater than the first accuracy;

[0039] A first compression module is used to convert the data type of the first-category meteorological data to obtain converted first-category meteorological data; and compress the converted first-category meteorological data according to a first compression ratio to obtain first compressed meteorological data;

[0040] A second compression module, used for compressing the second type of meteorological data according to a second compression ratio to obtain second compressed meteorological data;

[0041] A storage module is used to store the first compressed meteorological data and the second compressed meteorological data.

[0042] Optionally, in the device as described above, the first category of meteorological data includes meteorological data corresponding to a plurality of first category variables;

[0043] The first compression module is specifically used for:

[0044] For any meteorological data corresponding to the first-category variable, obtain a reference offset and a scaling factor corresponding to the first-category variable; perform data type conversion on each meteorological data corresponding to the first-category variable according to the reference offset and the scaling factor, and obtain converted meteorological data corresponding to a plurality of first-category variables;

[0045] The converted first-category meteorological data is determined according to the converted meteorological data corresponding to the plurality of first-category variables.

[0046] Optionally, in the above device, the first type of meteorological data is floating point data; and the first compression module is specifically used to:

[0047] Determine the maximum and minimum values ​​in the meteorological data corresponding to the first type of variables;

[0048] Determine an average value of the maximum value and the minimum value as the reference offset;

[0049] Get the number of values ​​contained in the consecutive bits occupied by the short integer data;

[0050] determining a first difference between the minimum and maximum values;

[0051] The proportional factor is determined according to the number of values ​​and the first difference.

[0052] Optionally, in the above-mentioned device, the first compression module is specifically used for:

[0053] For each meteorological data in the meteorological data corresponding to the first category variable, a second difference between the meteorological data and the reference offset is determined; and a ratio of the second difference to the proportional factor is determined as the converted meteorological data.

[0054] Optionally, the device and storage module as described above are specifically used for:

[0055] respectively taking the reference offsets corresponding to the plurality of first-category variables as first attribute values, and taking the scale factor as second attribute value;

[0056] storing the first compressed meteorological data and the first attribute values ​​and the second attribute values ​​corresponding to the plurality of first-category variables; the first attribute values ​​and the second attribute values ​​corresponding to the plurality of first-category variables are used to decompress the first compressed meteorological data;

[0057] The second compressed meteorological data is stored.

[0058] Optionally, the device as described above further comprises a decompression module, and the decompression module is used to:

[0059] decompressing the first compressed meteorological data to obtain first decompressed meteorological data;

[0060] Obtaining third attribute values ​​of a plurality of first-category variables, wherein the third attribute values ​​are used to indicate whether the meteorological data corresponding to the first-category variables include negative values;

[0061] Converting a negative value in the meteorological data corresponding to the target variable of the first decompressed meteorological data into a positive value to obtain the converted first decompressed meteorological data; the target variable being a first attribute indicates that the meteorological data corresponding to the target variable includes a negative value;

[0062] According to the first attribute values ​​and the second attribute values ​​corresponding to the multiple first-category variables, data type conversion is performed on the converted first decompressed meteorological data to obtain the first-category meteorological data.

[0063] The data processing device provided in the embodiment of the present application can execute the technical solution in the above method embodiment, and its beneficial effects are similar, which will not be described again here.

[0064] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0065] The memory stores computer-executable instructions;

[0066] The processor executes the computer-executable instructions stored in the memory to implement the method according to the first aspect.

[0067] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a computer, they are used to implement the method described in the first aspect.

[0068] The computer-readable storage medium provided in the embodiment of the present application can execute the technical solution in the above method embodiment, and its beneficial effects are similar, which will not be repeated here.

[0069] In a fifth aspect, the present application provides a computer program product, including a computer program, which is used to implement the method described in the first aspect when executed by a computer.

[0070] The computer program product provided in the embodiment of the present application can execute the technical solution in the above method embodiment, and its beneficial effects are similar and will not be described in detail here.

[0071] The present application provides a data processing method, device, electronic device, storage medium and program product. For a first type of meteorological data with a first precision, the first type of meteorological data can be converted into a data type to obtain the converted first type of meteorological data, and then the converted first type of meteorological data can be compressed according to a first compression ratio to obtain first compressed meteorological data. For a second type of meteorological data with a second precision and the meteorological data is a non-negative number, the second type of meteorological data can be directly compressed according to a second compression ratio to obtain second compressed meteorological data. The present application compresses different types of meteorological data in different ways, and the compression method is relatively simple, which is convenient for subsequent decompression and reduces the waste of computing resources and time. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0073] Figure 1 A flowchart of a data processing method provided in an embodiment of the present application;

[0074] Figure 2 A flowchart of another data processing method provided in an embodiment of the present application;

[0075] Figure 3 A flowchart of another data processing method provided in an embodiment of the present application;

[0076] Figure 4 A flowchart of a data processing method provided in an embodiment of the present application;

[0077] Figure 5 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0078] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0079] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0080] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0081] For some meteorological models that include multiple numerical models of global-regional integration, the raw data output by the model is in the network Common Data Format (NetCDF) 3-64bit format, the data type of each meteorological variable is floating point, and the number of variables is hundreds. The amount of net meteorological data generated by the meteorological model every day is close to 20TB. Without compression processing, the amount of net meteorological data generated by the model each year is as high as 7PB. In order to support the long-term data accumulation needs in scientific research and the accuracy verification needs of historical data in business, most meteorological models need to retain two years of meteorological data, which puts huge storage pressure.

[0082] There are commonly used compression tools such as zip, gzip, and bzip2 in Linux systems, which can compress files losslessly into a specific format. These tools generally support multiple compression levels, and multiple compression levels correspond to multiple different compression ratios. The higher the compression level, the higher the compression ratio, and the longer the compression time required.

[0083] The new files generated by compression tools under Linux systems have different formats from the original files and cannot be directly read and used. The historical data of the meteorological model will be read repeatedly by different users at irregular intervals. When using historical data, it is necessary to decompress and generate a global recovery file before reading it. The decompression process requires a large amount of computing resources, and the recovery file occupies a large amount of storage space. After reading, the recovery file needs to be deleted. Therefore, these compression tools cannot meet the needs of compressing meteorological data of large-scale meteorological models and directly using the meteorological data.

[0084] Grib2 is the currently recommended table-driven Grib encoding method, which is widely used to compress and store meteorological data such as atmosphere and ocean. It has efficient compression and decompression capabilities and is suitable for storing and transmitting meteorological data output by large-scale meteorological models.

[0085] The grib2 encoding method can only compress meteorological data of equal latitude and longitude grids, and cannot be used to compress meteorological data of spherical cube grids. Moreover, the compressed grib2 file needs to be converted to other common formats before it can be read and used by existing processing tools (such as WRF for generating background fields). For example, the netCDF format file has a complex conversion step and takes a long time. The grib2 encoding method is only suitable for scenarios where the original data will not be re-read after compression.

[0086] Therefore, the present application provides a data processing method, in which different types of meteorological data are compressed according to different compression methods, wherein different types of meteorological data may have different precisions and whether there are negative numbers, etc. It can be understood that these data features are features that may affect the accuracy of meteorological data used for weather forecasting and other applications. Compressing different types of meteorological data according to different compression methods makes the compression method simpler, facilitates subsequent decompression, and reduces the waste of computing resources and time.

[0087] The application scenarios to which the embodiments of the present application are applicable may be, for example, weather forecast, climate simulation, disaster warning and other scenarios, and data used in weather forecast, climate simulation, disaster warning and other scenarios may be compressed and decompressed.

[0088] Exemplarily, multiple types of meteorological data can be output by the meteorological model. In a possible implementation, the meteorological model uses a spherical cubic coordinate system in the horizontal direction, with a total of 6 spheres. In order to achieve fast output, multiple types of variables can be edited in the output variable table, and output according to different types can be achieved, such as at least one output file corresponding to each variable type, and each output file includes meteorological data corresponding to multiple variables. Exemplarily, each sphere is divided into nx×ny small blocks according to the x and y direction layout, so there are at least 6×nx×ny output files corresponding to each variable type, and the size of each output file can reach tens of MB to hundreds of MB, for example.

[0089] Exemplarily, the first type of meteorological data may include multiple output files, and the second type of meteorological data may also include multiple output files. Each output file includes meteorological data corresponding to multiple variables. It can be understood that the multiple variables in each output file belong to the same type of variables. When compressing the first type of meteorological data and the second type of meteorological data, the multiple output files can be compressed respectively. Taking the first type of meteorological data as an example, the multiple output files in the first type of meteorological data can be compressed respectively to obtain multiple compressed output files, thereby obtaining the first compressed meteorological data.

[0090] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0091] Figure 1 This is a flow chart of a data processing method provided in an embodiment of the present application. The method can be executed by a data processing device, which can be implemented by a computer program; it can also be implemented by a medium storing relevant computer programs, such as a USB flash drive and / or an optical disk, or it can also be implemented by a physical device integrated or installed with relevant computer programs, such as a chip or an electronic device. The electronic device can be a server, a server cluster, a terminal device such as a computer, etc. The following takes a server as an example for explanation. Figure 1 As shown, the method may include the following steps.

[0092] S101. Obtain meteorological data to be compressed, where the meteorological data to be compressed includes first-category meteorological data and second-category meteorological data.

[0093] The precision of the first type of meteorological data is the first precision. The precision of the second type of meteorological data is the second precision, and the second type of meteorological data is a non-negative number. The second precision is greater than the first precision.

[0094] Exemplarily, the server may obtain the meteorological data to be compressed from a meteorological model deployed in the server. For example, the meteorological model may output the first type of meteorological data and the second type of meteorological data, respectively. That is, the embodiment of the present application may compress or decompress the meteorological data on the spherical cube grid plane.

[0095] In a possible implementation, the meteorological model can output first sub-category meteorological data, second sub-category meteorological data, and second category meteorological data, respectively, wherein the first sub-category meteorological data and the second sub-category meteorological data belong to the first category meteorological data, and the corresponding compression methods are the same, but the decompression methods are different. This method facilitates the reading and decompression of the first compressed meteorological data obtained after the first category meteorological data is compressed, and there is no need to further classify the first sub-category meteorological data and the second sub-category meteorological data, thereby improving the decompression processing efficiency.

[0096] Exemplarily, the first type of meteorological data may include meteorological data corresponding to multiple first type variables. Exemplarily, the first type of variables may include, for example, temperature, wind speed of 10-meter latitudinal wind, and humidity (e.g., relative humidity at a height of two meters from the ground), etc., and changes in the values ​​of these variables of the order of one ten-thousandth will not affect the evaluation of the accuracy of the weather forecast. The second type of meteorological data may include meteorological data corresponding to multiple second type variables. Exemplarily, the second type of variables may include, for example, total rainfall and total hourly snowfall, etc. Such variables can only be negative numbers, and values ​​of the order of one ten-thousandth will affect the evaluation of the accuracy of the weather forecast. Taking total rainfall as an example, values ​​of the order of one millionth will be classified as light rain weather. It can be seen from this that the second type of variables have higher requirements for the accuracy of meteorological data, and the first type of variables have lower requirements for the accuracy of meteorological data. The second accuracy of the second type of meteorological data can be greater than the first accuracy of the first type of meteorological data.

[0097] In a possible implementation, the first type of meteorological data may include multiple first output files, and the second type of meteorological data may include multiple second output files. Exemplarily, the number of first output files and the number of second output files may be the number of grids generating meteorological data in the meteorological model.

[0098] S102, converting the data type of the first category meteorological data to obtain converted first category meteorological data; and compressing the converted first category meteorological data according to a first compression ratio to obtain first compressed meteorological data.

[0099] The server may linearly package the first type of meteorological data and then compress it. Specifically, the server may convert the data type of the first type of meteorological data to obtain converted first type of meteorological data, and then compress the converted first type of meteorological data according to a first compression ratio to obtain first compressed meteorological data.

[0100] Exemplarily, the server may convert the data type of the first type of meteorological data using the linear packing algorithm of netCDF, and the server may compress the converted first type of meteorological data using the LZ77 algorithm. When running the LZ77 algorithm, the server may use the first compression ratio and the compression level corresponding to the compression time required to compress the data.

[0101] S103: compress the second type of meteorological data according to a second compression ratio to obtain second compressed meteorological data.

[0102] The server may compress the second type of meteorological data according to the second compression ratio to obtain second compressed meteorological data. Exemplarily, the server may compress the second type of meteorological data according to the second compression ratio using a netCDF data compression algorithm. When running the netCDF data compression algorithm, the server may use the second compression ratio and the compression level corresponding to the compression time required for compression to compress the data.

[0103] It should be noted that, for multiple second-category variables, the second compression ratio may include compression ratios corresponding to multiple second-category variables, and the compression ratios corresponding to different variables may be different. For variables whose values ​​are mostly zero, such as snowfall, the compression ratio is higher.

[0104] It can be understood that the compression ratio refers to the ratio of the size of the uncompressed data to the size of the compressed data.

[0105] This is equivalent to a compression algorithm with a fixed compression ratio. The embodiment of the present application can use different compression ratios to compress different types of meteorological data, which greatly saves storage resources.

[0106] S104. Store the first compressed meteorological data and the second compressed meteorological data.

[0107] After compressing the first type of meteorological data and the second type of meteorological data, the first compressed meteorological data and the second compressed meteorological data may be stored for subsequent retrieval.

[0108] In this embodiment, for the first type of meteorological data with a first precision, the data type conversion can be performed on the first type of meteorological data to obtain the converted first type of meteorological data, and then the converted first type of meteorological data is compressed according to the first compression ratio to obtain the first compressed meteorological data. For the second type of meteorological data with a second precision and the meteorological data is a non-negative number, the second type of meteorological data can be directly compressed according to the second compression ratio to obtain the second compressed meteorological data. The present application compresses different types of meteorological data in different ways, so that the compression method is simpler, which is convenient for subsequent decompression and reduces the waste of computing resources and time.

[0109] Next, how to perform data type conversion on the first category meteorological data to obtain the converted first category meteorological data is described.

[0110] Figure 2A flowchart of another data processing method provided in an embodiment of the present application, the method can be executed by a data processing device, and the data processing device can be implemented by a computer program; it can also be implemented by a medium storing relevant computer programs, such as a USB flash drive and / or a CD, or it can also be implemented by a physical device integrated or installed with relevant computer programs, such as a chip or electronic device. The electronic device can be a server, a server cluster, a computer or other intelligent terminal, etc., and the following takes a server as an example for explanation. Figure 2 As shown, the method may include the following steps.

[0111] S201. For any meteorological data corresponding to a first-category variable, obtain a reference offset and a scaling factor corresponding to the first-category variable, and perform data type conversion on each meteorological data in the meteorological data corresponding to the first-category variable according to the reference offset and the scaling factor to obtain converted meteorological data corresponding to multiple first-category variables.

[0112] For any first-category variable, the server can obtain the reference offset and scale factor corresponding to the first-category variable.

[0113] In a possible implementation, the first type of meteorological data is floating point data. For any first type of variable, the server can determine the maximum and minimum values ​​in the meteorological data corresponding to the first type of variable, and determine the average of the maximum and minimum values ​​as the reference offset. Obtain the number of values ​​accommodated by the number of consecutive bits occupied by the short integer data. Determine the first difference between the minimum and maximum values. Then determine the scale factor according to the number of values ​​and the first difference.

[0114] Exemplarily, the number of bits occupied by short integer data is 16 bits, and the value range of short integer data is [-32768, 32767], where -32767 is used to accommodate missing values. After removing this bit, the remaining continuous value interval is [-32766, 32767], that is, the remaining continuous number of bits is 65534, so that the number of values ​​accommodated by the continuous number of bits is 65534. It can be understood that 65534 values ​​are continuous discrete values. Among them, the missing value is the value of the missing value attribute of the first category variable, for example, it can be defaulted to -32767s, where s indicates that the value is a short integer. The missing value attribute can be used by other user devices or programs to read the meteorological data corresponding to the first category variable. Data consistency and interoperability can be ensured.

[0115] Exemplarily, the server may determine the reference offset and the scale factor according to the following formula:

[0116] (1)

[0117] (2)

[0118] Among them, sf is the reference offset, ao is the scale factor, min is the minimum value, max is the maximum value, and ndrv is the number of values ​​​​contained by the number of consecutive bits occupied by the short integer data.

[0119] It can be understood that data type conversion can be understood as converting 4-byte floating-point data into 2-byte short integer data with reduced precision, and the data volume is reduced by half. The scale factor is determined by the maximum value, minimum value, and the number of values ​​contained in the continuous number of bits occupied by the short integer data, and the reference offset is determined according to the maximum value and minimum value, so as to achieve a balance between data accuracy and storage efficiency.

[0120] In a possible implementation, after the server determines the converted first-category meteorological data, for any first-category variable, for each meteorological data in the meteorological data corresponding to the first-category variable, a second difference between the meteorological data and the reference offset is determined, and a ratio of the second difference to the scale factor is determined as the converted meteorological data.

[0121] Specifically, the converted meteorological data can be obtained according to formula (3):

[0122] (3)

[0123] Among them, pck is the converted meteorological data, and upk is the original meteorological data. The storage requirements of data are reduced by converting floating point numbers to integers.

[0124] S202. Determine converted first-category meteorological data according to the converted meteorological data corresponding to the plurality of first-category variables.

[0125] After the data type conversion is performed on the meteorological data corresponding to each first-category variable, the converted first-category meteorological data can be obtained according to the converted meteorological data corresponding to the plurality of first-category variables.

[0126] In this embodiment, for the first type of meteorological data, data type conversion is performed using the reference offset and scale factor corresponding to each type of variable, thereby reducing the storage requirement of the data without affecting the accuracy of the data.

[0127] Next, the decompression of the first compressed meteorological data and the second compressed meteorological data will be described.

[0128] Figure 3A flowchart of another data processing method provided in an embodiment of the present application, the method can be executed by a data processing device, and the data processing device can be implemented by a computer program; it can also be implemented by a medium storing relevant computer programs, such as a USB flash drive and / or a CD, or it can also be implemented by a physical device integrated or installed with relevant computer programs, such as a chip or electronic device. The electronic device can be a server, a server cluster, a terminal device such as a computer, etc., and the following description takes a server as an example. Figure 3 As shown, the method may include the following steps.

[0129] S301. Use the reference offsets corresponding to the plurality of first-category variables as first attribute values, and use the scale factors as second attribute values.

[0130] After obtaining the first compressed meteorological data, for any first-category variable, the server can use the reference offset corresponding to the first-category variable as the first attribute value of the first-category variable, and use the proportional factor as the second attribute value of the first-category variable.

[0131] S302. Store the first compressed meteorological data and the first attribute values ​​and the second attribute values ​​corresponding to the multiple first-category variables, wherein the first attribute values ​​and the second attribute values ​​corresponding to the multiple first-category variables are used to decompress the first compressed meteorological data.

[0132] The server can store the first compressed meteorological data, and store the first attribute value and the second value corresponding to each first-category variable, so that when the meteorological data of any first-category variable in the first compressed meteorological data is subsequently read, it can be decompressed based on the first attribute value and the second attribute value.

[0133] S303: Store the second compressed meteorological data.

[0134] In this embodiment, when storing the first compressed meteorological data, the reference offset and the scale factor of each first-category variable may be stored as attribute values, and the second compressed meteorological data may be stored to facilitate subsequent decompression.

[0135] In a possible implementation, the server may decompress the first compressed meteorological data in the following manner:

[0136] The server can decompress the first compressed meteorological data to obtain the first decompressed meteorological data, and then obtain the third attribute values ​​of the plurality of first-category variables, the third attribute values ​​being used to indicate whether the meteorological data corresponding to the first-category variables include negative values. The negative values ​​in the meteorological data corresponding to the target variable of the first decompressed meteorological data are converted to positive values ​​to obtain the converted first decompressed meteorological data, wherein the target variable is the first attribute indicating that the meteorological data corresponding to the target variable includes negative values. Then, according to the first attribute values ​​and the second attribute values ​​corresponding to the plurality of first-category variables, the converted first decompressed meteorological data is decompressed to obtain the first-category meteorological data.

[0137] Exemplarily, converting a negative value in the meteorological data corresponding to the target variable of the first decompressed meteorological data into a positive value may be performed by taking the absolute value of the negative value to obtain a positive value.

[0138] That is to say, for the first category of meteorological data, the server can identify the first sub-category variables and the second sub-category variables among the multiple first category variables based on the third attribute value, wherein the first sub-category variables are meteorological data that include negative values, and the second sub-category variables are meteorological data that are non-negative values. Exemplarily, the first sub-category variables may include, for example, air temperature and 10-meter latitudinal wind speed, and the second sub-category variables may include variables such as humidity. It can be understood that the second sub-category variables have a numerical restriction range, such as the relative humidity at a height of 2 meters from the ground, and the value can only be between 0 and 100%. The first sub-category variables have no numerical restriction range. The above-mentioned target variable is the second sub-category variable.

[0139] It can be understood that the server can decompress the first compressed meteorological data according to the LZ77 algorithm.

[0140] Exemplarily, after decompressing the first compressed meteorological data, the server may perform data type conversion for any meteorological data of the first category variable in the following manner:

[0141] (4)

[0142] Among them, pck is the decompressed meteorological data, and upk is the meteorological data after data conversion, that is, the short integer meteorological data is converted into floating point meteorological data.

[0143] It can be understood that formula (4) is for the first type of variable whose first attribute value and second attribute value are not empty, that is, the variable has the first attribute value and the second attribute value.

[0144] For any meteorological data of the first-category variable, if the second attribute value of the first-category variable is empty, it means that ao is 0; if the first attribute value of the first-category variable is empty, it means that the meteorological data of the first-category variable is a constant, that is, a value that does not change with time.

[0145] In this embodiment, for the first type of meteorological data, after data type conversion is performed and then data compression is performed, some positive numbers may be compressed into negative numbers. Therefore, after decompressing the meteorological data of this type of variables, it is necessary to positiveen the negative numbers and then perform data type conversion based on the two attribute values ​​determined during the data type conversion before compression to improve the accuracy of the data.

[0146] It can be understood that for the second compressed meteorological data, the server can decompress the second compressed meteorological data according to a compression algorithm used to compress the second type of meteorological data, such as a data compression algorithm of netCDF, to obtain the second type of meteorological data.

[0147] For ease of understanding, the following Figure 4 The compression and decompression processes of various types of meteorological data are explained. Figure 4 A flowchart of a data processing method provided in an embodiment of the present application is shown in FIG. Figure 4 As shown, the method includes the following steps.

[0148] S401, obtaining first subcategory meteorological data, second subcategory meteorological data and second category meteorological data.

[0149] S402: Convert the data type of the first subcategory meteorological data to obtain converted first subcategory meteorological data.

[0150] S403: compress the converted first sub-category meteorological data according to a first compression ratio to obtain first sub-compressed meteorological data.

[0151] S404: Perform data type conversion on the second subcategory meteorological data to obtain converted second subcategory meteorological data.

[0152] S405: compress the converted second sub-category meteorological data according to the first compression ratio to obtain second sub-compressed meteorological data.

[0153] Among them, for the first compressed meteorological data in the above embodiment, the first compressed meteorological data includes first sub-compressed meteorological data and second sub-compressed meteorological data.

[0154] S406: Decompress the first sub-compressed meteorological data to obtain second decompressed meteorological data.

[0155] S407: Perform data type conversion on the second decompressed meteorological data to obtain first sub-category meteorological data.

[0156] S408. Decompress the second sub-compressed meteorological data to obtain third decompressed meteorological data.

[0157] S409: Perform data type conversion on the third decompressed meteorological data to obtain converted third decompressed meteorological data.

[0158] S410. Take the absolute value of the negative value in the converted third decompressed meteorological data to obtain the second subcategory meteorological data.

[0159] Among them, the first decompressed meteorological data in the above embodiment includes the second decompressed meteorological data and the third decompressed meteorological data.

[0160] S411. Compress the converted second-category meteorological data according to a second compression ratio to obtain second compressed meteorological data.

[0161] S412: Decompress the second compressed meteorological data to obtain second-category meteorological data.

[0162] The specific implementation method and technical effects of this embodiment are similar to those of the above embodiment and will not be repeated here.

[0163] Figure 5 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the device 50 includes an acquisition module 501 , a first compression module 502 , a second compression module 503 and a storage module 504 .

[0164] The acquisition module 501 is used to acquire the meteorological data to be compressed, wherein the meteorological data to be compressed includes first-category meteorological data and second-category meteorological data. The accuracy of the first-category meteorological data is the first accuracy. The accuracy of the second-category meteorological data is the second accuracy, and the second-category meteorological data is a non-negative number. The second accuracy is greater than the first accuracy.

[0165] The first compression module 502 is used to convert the data type of the first type of meteorological data to obtain converted first type of meteorological data, and compress the converted first type of meteorological data according to a first compression ratio to obtain first compressed meteorological data.

[0166] The second compression module 503 is used to compress the second type of meteorological data according to a second compression ratio to obtain second compressed meteorological data.

[0167] The storage module 504 is used to store the first compressed meteorological data and the second compressed meteorological data.

[0168] In a possible implementation, the first category of meteorological data includes meteorological data corresponding to a plurality of first category variables.

[0169] The first compression module 502 is specifically configured to:

[0170] For any meteorological data corresponding to the first-class variable, a reference offset and a scale factor corresponding to the first-class variable are obtained. According to the reference offset and the scale factor, data type conversion is performed on each meteorological data corresponding to the first-class variable to obtain converted meteorological data corresponding to multiple first-class variables.

[0171] The converted first-category meteorological data is determined according to the converted meteorological data corresponding to the plurality of first-category variables.

[0172] In a possible implementation, the first type of meteorological data is floating point data. The first compression module 502 is specifically configured to:

[0173] Determine the maximum and minimum values ​​in the meteorological data corresponding to the first category variable.

[0174] The average of the maximum and minimum values ​​is determined as the reference offset.

[0175] Gets the number of values ​​​​contained by the consecutive bits occupied by short integer data.

[0176] Determine the first difference between the minimum and maximum values.

[0177] A scaling factor is determined according to the number of values ​​and the first difference.

[0178] In a possible implementation, the first compression module 502 is specifically configured to:

[0179] For each meteorological data in the meteorological data corresponding to the first type of variable, a second difference between the meteorological data and the reference offset is determined, and a ratio of the second difference to the proportional factor is determined as the converted meteorological data.

[0180] In a possible implementation, the storage module 504 is specifically configured to:

[0181] The reference offsets corresponding to the plurality of first-category variables are respectively used as the first attribute value, and the scale factor is used as the second attribute value.

[0182] The first compressed meteorological data and the first attribute values ​​and the second attribute values ​​corresponding to the plurality of first-class variables are stored. The first attribute values ​​and the second attribute values ​​corresponding to the plurality of first-class variables are used to decompress the first compressed meteorological data.

[0183] The second compressed meteorological data is stored.

[0184] In a possible implementation, the device further includes a decompression module, and the decompression module is used to:

[0185] The first compressed meteorological data is decompressed to obtain first decompressed meteorological data.

[0186] The third attribute values ​​of the plurality of first-category variables are obtained, where the third attribute values ​​are used to indicate whether the meteorological data corresponding to the first-category variables include negative values.

[0187] The negative value in the meteorological data corresponding to the target variable of the first decompressed meteorological data is converted into a positive value to obtain the converted first decompressed meteorological data. The target variable is a first attribute indicating that the meteorological data corresponding to the target variable includes a negative value.

[0188] According to the first attribute values ​​and the second attribute values ​​corresponding to the plurality of first-category variables, data type conversion is performed on the converted first decompressed meteorological data to obtain the first-category meteorological data.

[0189] The device of this embodiment can be used to execute the technical solution of the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0190] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the electronic device 60 may include: at least one processor 601 and a memory 602 .

[0191] The memory 602 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer-executable instructions.

[0192] The memory 602 may include a random access memory (RAM), and may also include a non-volatile memory (Non-volatile Memory), such as at least one disk memory.

[0193] The processor 601 is used to execute the computer-executable instructions stored in the memory 602 to implement the method described in the above method embodiment. The processor 601 may be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0194] Optionally, the electronic device 60 may further include: a communication interface 603. In a specific implementation, if the communication interface 603, the memory 602 and the processor 601 are implemented independently, the communication interface 603, the memory 602 and the processor 601 may be interconnected through a bus and communicate with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0195] Optionally, in a specific implementation, if the communication interface 603, the memory 602 and the processor 601 are integrated on a chip, the communication interface 603, the memory 602 and the processor 601 can communicate through an internal interface.

[0196] The electronic device 60 may be a server, a server cluster, a computer or other terminal device.

[0197] The electronic device of this embodiment can be used to execute the technical solution of the above method embodiment. The specific implementation method and technical effect are similar and will not be repeated here.

[0198] An embodiment of the present application provides a computer-readable storage medium, which may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a RAM, a disk or an optical disk, and other media that can store computer-executable instructions. Specifically, the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a computer, the technical solution shown in the above method embodiment is executed. The specific implementation method and technical effect are similar and will not be repeated here.

[0199] An embodiment of the present application provides a computer program product, including a computer program. When the computer program is executed by a computer, the technical solution shown in the above method embodiment is executed. The specific implementation method and technical effect are similar and will not be repeated here.

[0200] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0201] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0202] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.

[0203] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.

[0204] If the integrated unit / module is implemented in the form of hardware, the hardware may be a digital circuit, an analog circuit, etc. The physical implementation of the hardware structure includes but is not limited to transistors, memristors, etc. Unless otherwise specified, the processor may be any appropriate hardware processor, such as a CPU, a GPU, an FPGA, a DSP, an ASIC, etc. Unless otherwise specified, the storage unit may be any appropriate magnetic storage medium or magneto-optical storage medium, such as a resistive random access memory (RRAM), a dynamic random access memory (DRAM), a static random access memory (SRAM), an enhanced dynamic random access memory (EDRAM), a high-bandwidth memory (HBM), a hybrid memory cube (HMC), etc.

[0205] If the integrated unit / module is implemented in the form of a software program module and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a memory and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned memory includes: various media that can store program codes, such as USB flash drives, ROM, RAM, mobile hard disks, magnetic disks or optical disks.

[0206] In the above embodiments, the description of each embodiment has its own emphasis. For parts not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0207] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0208] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data processing method, characterized in that: include: Acquire meteorological data to be compressed, the meteorological data to be compressed includes first-category meteorological data and second-category meteorological data; wherein the precision of the first-category meteorological data is a first precision; the precision of the second-category meteorological data is a second precision, and the second-category meteorological data is a non-negative number; wherein the second precision is greater than the first precision; Converting the data type of the first-category meteorological data to obtain converted first-category meteorological data; and compressing the converted first-category meteorological data according to a first compression ratio to obtain first compressed meteorological data; compressing the second type of meteorological data according to a second compression ratio to obtain second compressed meteorological data; The first compressed meteorological data and the second compressed meteorological data are stored.

2. The method according to claim 1, characterized in that The first type of meteorological data includes meteorological data corresponding to a plurality of first type variables; The first-category meteorological data is subjected to data type conversion to obtain the converted first-category meteorological data, including: For any meteorological data corresponding to the first-category variable, obtain a reference offset and a scaling factor corresponding to the first-category variable; perform data type conversion on each meteorological data corresponding to the first-category variable according to the reference offset and the scaling factor, and obtain converted meteorological data corresponding to a plurality of first-category variables; The converted first-category meteorological data is determined according to the converted meteorological data corresponding to the plurality of first-category variables.

3. The method according to claim 2, characterized in that The first type of meteorological data is floating point data; Obtaining the reference offset and scale factor corresponding to the first type of variables includes: Determine the maximum and minimum values ​​in the meteorological data corresponding to the first type of variables; Determine an average value of the maximum value and the minimum value as the reference offset; Get the number of values ​​contained in the consecutive bits occupied by the short integer data; determining a first difference between the minimum and maximum values; The proportional factor is determined according to the number of values ​​and the first difference.

4. The method according to claim 2 or 3, characterized in that: According to the reference offset and the scale factor, data type conversion is performed on each meteorological data in the meteorological data corresponding to the first type of variables to obtain converted meteorological data corresponding to a plurality of first type of variables, including: For each meteorological data in the meteorological data corresponding to the first category variable, a second difference between the meteorological data and the reference offset is determined; and a ratio of the second difference to the proportional factor is determined as the converted meteorological data.

5. The method according to claim 2 or 3, characterized in that: Storing the first compressed meteorological data and the second compressed meteorological data includes: respectively taking the reference offsets corresponding to the plurality of first-category variables as first attribute values, and taking the scale factor as second attribute value; storing the first compressed meteorological data and the first attribute values ​​and the second attribute values ​​corresponding to the plurality of first-category variables; the first attribute values ​​and the second attribute values ​​corresponding to the plurality of first-category variables are used to decompress the first compressed meteorological data; The second compressed meteorological data is stored.

6. The method according to claim 5, characterized in that The method further comprises: decompressing the first compressed meteorological data to obtain first decompressed meteorological data; Obtaining third attribute values ​​of a plurality of first-category variables, wherein the third attribute values ​​are used to indicate whether the meteorological data corresponding to the first-category variables include negative values; Converting a negative value in the meteorological data corresponding to the target variable of the first decompressed meteorological data into a positive value to obtain the converted first decompressed meteorological data; the target variable being a first attribute indicates that the meteorological data corresponding to the target variable includes a negative value; According to the first attribute values ​​and the second attribute values ​​corresponding to the multiple first-category variables, data type conversion is performed on the converted first decompressed meteorological data to obtain the first-category meteorological data.

7. A data processing device, characterized in that: include: An acquisition module, used for acquiring meteorological data to be compressed, wherein the meteorological data to be compressed includes first-category meteorological data and second-category meteorological data; wherein the accuracy of the first-category meteorological data is a first accuracy; the accuracy of the second-category meteorological data is a second accuracy, and the second-category meteorological data is a non-negative number; wherein the second accuracy is greater than the first accuracy; A first compression module is used to convert the data type of the first-category meteorological data to obtain converted first-category meteorological data; and compress the converted first-category meteorological data according to a first compression ratio to obtain first compressed meteorological data; A second compression module, used for compressing the second type of meteorological data according to a second compression ratio to obtain second compressed meteorological data; A storage module is used to store the first compressed meteorological data and the second compressed meteorological data.

8. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 6 when being executed by a processor.

Images