A GRIB data extraction method based on precise positioning and fast customizability

Abstract

The present invention discloses a GRIB data extraction method based on precise positioning and rapid customization. The present invention can realize precise positioning of GRIB data blocks, save the time consumed by data block search in the existing method, thereby improving the data extraction speed; realize rapid customization of GRIB data extraction, filter the dimensional data required by the business on demand, reduce the writing and transmission of data not required by the extraction business, improve the efficiency of data extraction, and reduce the waste of computing and storage resources.

Description

Technical Field

[0001] The present invention relates to the technical field of GRIB data extraction, and in particular to a GRIB data extraction method based on precise positioning and rapid customization. Background Art

[0002] GRIB code is a compressed binary code recommended by the World Meteorological Organization (WMO) that is independent of the computer operating system. It is a binary file format used to exchange and store regularly distributed data. The current GRIB code versions are GRIB1 and GRIB2. It has the characteristics of convenient multi-dimensional data storage, high compression ratio, support for multiple compression methods, strong scalability, etc. It is particularly good at exchanging large-capacity multi-dimensional unstructured data and is widely used in the storage and exchange of numerical weather forecast products, ocean hydrology and other data. At the same time, data stored and exchanged with GRIB code are usually used in data application fields dominated by fusion analysis applications, such as forecast, prediction and early warning applications of meteorology, oceanography and hydrology, which require decoding of GRIB-encoded data, including decompression, data format conversion and other data analysis. Therefore, the rapid processing technology of large-capacity unstructured data in these fields is one of the core capabilities.

[0003] At present, there are two decoding tools for GRIB codes in China. The first one is the wgrib command line tool, which can extract GRIB data through the command line through different parameter combinations. The second method is the GRIB API for programming languages, which can extract data through programming using get and set key value operations. The first method is widely used in China. The use of the first method can basically meet the business application needs of data extraction, but it is the extraction of the entire data block, which may cause the data that is not needed for the extraction business to flow in the extraction process, resulting in low data extraction efficiency and high consumption of space resources. Summary of the invention

[0004] 1. Technical issues to be resolved

[0005] In view of the deficiencies of the prior art, the present invention provides a GRIB data extraction method based on precise positioning and rapid customization, which can solve the above technical problems.

[0006] (II) Technical solution

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions: a GRIB data extraction method based on precise positioning and rapid customization, comprising the following steps:

[0008] Step S1: Obtain metadata information of each GRIB data block in the GRIB file to obtain a metadata body, wherein the metadata information includes a starting position of the GRIB data block;

[0009] Step S2: Decode each GRIB data block into binary data to obtain a physical data body;

[0010] Step S3: According to the correspondence between the metadata body and the entity data body, the metadata body is sorted to determine the starting position of the GRIB data block;

[0011] Step S4: After the starting position is determined, the data offset is calculated according to the GRIB attribute information required for data extraction, and then the position of the GRIB data block in the entity data body is located. Further, according to the spatial range required for data extraction, the located GRIB data block is intercepted and the data corresponding to the required GRIB dimension is extracted.

[0012] Preferably, the GRIB data extraction method based on precise positioning and rapid customization adopts a file-driven multi-threaded processing method.

[0013] Preferably, before decoding each GRIB data block into binary data in step S2, the step further includes: filtering metadata information of the GRIB data block.

[0014] Preferably, the decoding process in step S2 specifically includes decompression and decoding.

[0015] Preferably, the metadata information also includes the end position of the GRIB data block.

[0016] Preferably, the order in which the entity data bodies are stored corresponds one-to-one to the order in which the metadata bodies are stored.

[0017] Preferably, the data interception of the located GRIB data block in step S4 specifically includes data interception from two directions of longitude and latitude.

[0018] Preferably, the GRIB attribute information includes latitude and longitude range and spatial resolution.

[0019] Preferably, the GRIB dimensions include the element types of numerical forecast products, the generation time of numerical forecast products, and the forecast timeliness.

[0020] Preferably, the GRIB dimension also includes hierarchical height and member variables.

[0021] (III) Beneficial effects

[0022] Compared with the prior art, the present invention provides a GRIB data extraction method based on precise positioning and rapid customization, which has the following beneficial effects: by obtaining a metadata body and an entity data body, and then locating the position of a GRIB data block in the entity data body, precise positioning of the GRIB data block is achieved, saving the time consumed by data block search in the prior method, thereby improving the data extraction speed; in addition, data is intercepted on the GRIB data block according to the spatial range requirements required for data extraction, and data corresponding to the required GRIB dimensions are extracted, thereby achieving rapid customization of GRIB data extraction, filtering dimensional data required for the business on demand, reducing the writing and transmission of data not required for the extraction business, improving the efficiency of data extraction, and reducing the waste of computing and storage resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 The present invention is a flow chart of a GRIB data extraction method based on precise positioning and rapid customization. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] The present invention provides a GRIB data extraction method based on precise positioning and rapid customization, comprising the following steps:

[0026] Step S1: Obtain metadata information of each GRIB data block in the GRIB file to obtain a metadata body, wherein the metadata information includes a starting position of the GRIB data block.

[0027] The object processed by the present invention can be a numerical forecast product such as meteorology stored in GRIB data. A GRIB data block corresponds to a plane data of the numerical forecast product. The plane data is an unstructured grid data, and the number of grid points is determined by the spatial resolution of the numerical forecast product. In addition, the metadata information may also include information data such as the end position of the GRIB data block. The step S1 specifically prints out a short directory of data files through wgrib, and then obtains the metadata information of each GRIB data block, wherein the metadata information obtained is lightweight metadata information, and the corresponding time consumption is at the millimeter level; wgrib is a command line tool developed by NCEP (National Center for Environmental Prediction) for reading, creating, and modifying GRIB1 and GRIB2 files.

[0028] Step S2: Decode each GRIB data block into binary data to obtain a physical data body.

[0029] The decoding process of step S2 specifically includes decompression and decoding. Preferably, before decoding each GRIB data block into binary data in step S2, the process also includes: filtering metadata information of the GRIB data block; specifically, the metadata information of the GRIB data block can be filtered out by the above-mentioned wgrib tool command through a combination of command parameters.

[0030] The order in which the entity data bodies are stored corresponds one-to-one to the order in which the metadata bodies are stored, that is, the order in which the binary data obtained by the decoding process are stored in the entity data body corresponds one-to-one to the order in which the metadata information is stored in the metadata body.

[0031] Step S3: According to the correspondence between the metadata body and the entity data body, the metadata body is sorted to determine the starting position of the GRIB data block.

[0032] This step S3 determines the starting position of the GRIB data block by sorting the metadata body, that is, classifying the metadata information.

[0033] Step S4: After the starting position is determined, the data offset is calculated according to the GRIB attribute information required for data extraction, and then the position of the GRIB data block in the entity data body is located. Further, according to the spatial range required for data extraction, the located GRIB data block is intercepted and the data corresponding to the required GRIB dimension is extracted.

[0034] The above GRIB attribute information includes the latitude and longitude range and spatial resolution. In addition, the GRIB attribute information may also include data from south to north (indicated by application 1) and data from north to south (indicated by application 0). The GRIB dimensions include the element type (VarName) of the numerical forecast product, the generation time (DataTime) of the numerical forecast product, and the forecast timeliness (ForcastName); the GRIB dimensions also include the level height (LevelName) and the member variable (MumberName), etc., that is, the above plane data Data[v] can be represented by a mathematical function: Data[v] = F(VarName, DataTime, LevelName, ForcastName, ...). The data corresponding to the above required GRIB dimensions is the final extracted GRIB data, which is binary data.

[0035] In step S4, data interception of the located GRIB data block specifically includes data interception from two directions of longitude and latitude, that is, the above-mentioned spatial range requirement corresponds to the longitude and latitude range requirement.

[0036] The GRIB data extraction method based on precise positioning and rapid customization of the present invention has the following beneficial effects: by obtaining a metadata body and an entity data body, and then locating the position of a GRIB data block in the entity data body, precise positioning of the GRIB data block is achieved, and the time consumed by the data block search in the existing method is saved, thereby improving the data extraction speed; in addition, data is intercepted from the GRIB data block according to the spatial range requirements required for data extraction, and data corresponding to the required GRIB dimension is extracted, thereby achieving rapid customization of GRIB data extraction, screening dimensional data required for the business on demand, reducing the writing and transmission of data not required for the extraction business, improving the efficiency of data extraction, and reducing the waste of computing and storage resources.

[0037] In addition, the storage specification of numerical forecast product data is classified and stored according to the forecast time, for example, all forecast product data for the next 3 hours are stored in a GRIB file. Preferably, the GRIB data extraction method based on precise positioning and rapid customization of the present invention adopts a file-driven multi-threaded processing method, such as 4 threads, 8 threads, 16 threads, etc. The user can independently set the number of threads according to the resource situation of the data processing server, so as to realize multi-threaded processing of GRIB files under the condition of matching resources and processing performance. The efficiency of GRIB data extraction can be further improved through multi-threaded processing. The actual test results show that the speed of 16-thread processing is increased from 257ms to 37ms compared with the single process processing.

[0038] Preferably, the customized information data required for data extraction such as GRIB attribute information, GRIB dimensions, and number of threads can be saved in a master control configuration file. The data extraction processing of the present invention is unified controlled based on the master control configuration file. The master control configuration file can be in xml format, which supports dynamic attribute modification.

[0039] It should be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0040] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A GRIB data extraction method based on precise positioning and rapid customizability, characterized in that: The following steps are involved: Step S1: acquiring metadata information of each GRIB data block in a GRIB file to obtain a metadata body, wherein the metadata information includes a starting position of the GRIB data block; Step S2: decoding each of the GRIB data blocks into binary data to obtain a physical data body; The decoding process of step S2 specifically includes decompression and decoding; the order in which the entity data bodies are stored corresponds to the order in which the metadata bodies are stored; Step S3: According to the correspondence between the metadata body and the entity data body, the metadata body is sorted to determine the starting position of the GRIB data block; Step S4: after the starting position is determined, the data offset is calculated according to the GRIB attribute information required for data extraction, thereby locating the position of the GRIB data block in the entity data body, and further data interception is performed on the located GRIB data block according to the spatial range required for data extraction, and data corresponding to the required GRIB dimension is extracted; The method adopts a file-driven multi-threaded processing method; The metadata information also includes the end position of the GRIB data block.

2. The method according to claim 1, characterized in that Before decoding each of the GRIB data blocks into binary data in step S2, the step further includes: filtering metadata information of the GRIB data blocks.

3. The method according to claim 1, characterized in that: The data interception of the located GRIB data block in step S4 specifically includes data interception from two directions of longitude and latitude.

4. The method according to claim 1, characterized in that: The GRIB attribute information includes latitude and longitude range and spatial resolution.

5. The method according to claim 1, characterized in that: The GRIB dimensions include the element types of numerical forecast products, the generation time of numerical forecast products, and the forecast timeliness.

6. The method according to claim 5, characterized in that: The GRIB dimension also includes hierarchical height and member variables.

Images