A large-scale grib grid data distributed storage method

Abstract

This application relates to a distributed storage method for large-scale GRIB grid data. The method includes: a GRIB objectification technique based on encoding features, which treats the entire GRIB message as an object data body and self-describing information as metadata, achieving native support for grid fields of arbitrary sizes; a core metadata dataset definition technique based on virtual keys, which maps each GRIB message in the GRIB file to a single storage object and maps the self-describing information in the GRIB message to the metadata of the object storage; and fully utilizing the independent and distributed storage characteristics of GRIB objects, employing distributed parallel pipeline processing to leverage the advantages of distributed computing, thereby significantly improving the distributed parallel processing efficiency of large-scale grid data. This application solves the problems of coarse granularity, poor consistency, and low access efficiency in traditional retrieval methods, and can be widely applied to Earth observation, numerical weather prediction, and big data analysis scenarios.

Description

Technical Field

[0001] This application relates to the field of distributed storage technology for GRIB grid data, and more particularly to a method for large-scale distributed storage of GRIB grid data. Background Technology

[0002] In Earth system science research, with the continuous advancement of observation technology and high-performance computing capabilities, the scale of data generated by numerical simulations has experienced explosive growth. Satellite remote sensing, high-resolution meteorological models, and reanalysis products are constantly outputting massive amounts of data. For example, the European Centre for Medium-Range Weather Forecasts (ECMWF) adds approximately 400 TB of GRIB-formatted grid data daily, and a single GRIB file typically contains hundreds to thousands of independent grid fields.

[0003] Traditional storage architectures manage GRIB data holistically using a file system, leading to significant drawbacks in data access: retrieval granularity is too coarse; even when a user needs to retrieve a single grid field, the entire file must be read, resulting in numerous redundant I / O operations. Separate storage of file content and index information makes data consistency issues prone to arise due to asynchronous updates. Querying or aggregating grid fields across multiple files requires complex manual operations, which are not only cumbersome but also significantly reduce query efficiency. These problems become increasingly prominent when dealing with the exponentially growing volume of GRIB data, severely restricting real-time data processing capabilities in fields such as weather forecasting and climate analysis. Summary of the Invention

[0004] The following is an overview of the subject matter described in detail herein. This overview is not intended to limit the scope of the claims.

[0005] The main objective of this disclosure is to propose a large-scale GRIB grid data distributed storage method that can solve the problems of coarse retrieval granularity, poor consistency, and low access efficiency in existing technologies. It can be widely applied to Earth observation, numerical forecasting, and big data analysis scenarios.

[0006] A first aspect of this application provides a method for distributed storage of large-scale GRIB grid data, the method comprising:

[0007] Respond to GRIB data write requests and retrieve GRIB grid data; Read the GRIB messages and descriptive attribute information from the GRIB mesh data; The GRIB message is mapped to a GRIB object in the distributed object storage, and the descriptive attribute information in the GRIB message is mapped to the metadata of the corresponding GRIB object; Write the GRIB object into the distributed object storage, generate a set of key names based on the metadata of the GRIB object, and establish a mapping relationship between the set of key names and the corresponding GRIB object; Respond to GRIB data read requests and retrieve key-value pairs for retrieval; Based on the key-value pairs and the mapping relationship, a list of GRIB objects is retrieved from the distributed object storage; the list of GRIB objects contains at least one GRIB object that meets the retrieval requirements of the key-value pairs. For at least one GRIB object in the GRIB object list, perform GRIB object data processing, and output at least one corresponding GRIB file for the at least one GRIB object after data processing.

[0008] The large-scale GRIB grid data distributed storage method provided in this embodiment has at least the following beneficial effects: This method effectively solves the problem of coarse retrieval granularity in large-scale GRIB grid data storage by decomposing GRIB messages into independent storage units and combining them with a metadata-driven indexing mechanism, enabling fine-grained data access from the file level to the message level. Integrated data and index storage ensures data consistency, fundamentally eliminating the inconsistency risks caused by file and index divide-and-conquer. Metadata-based key name sets enable search conditions to accurately match object attributes, significantly improving access efficiency and simplifying the complexity of cross-file grid field retrieval and aggregation operations, thereby enhancing access efficiency and simplifying cross-file retrieval operations.

[0009] In some embodiments of this application, the GRIB object is written to the distributed object storage using a main process and an asynchronous child process. The main process includes: The codec tool interface is invoked to read each GRIB message from the GRIB grid data and to start an asynchronous processing subprocess. Each time the asynchronous processing subprocess is started, and there are still unread GRIB packets in the GRIB grid data, the GRIB codec tool interface is called to read the next GRIB packet from the GRIB grid data, and the asynchronous processing subprocess is started. The main process terminates each time the asynchronous processing subprocess is started and all GRIB messages in the GRIB grid data are read. The asynchronous processing subprocess includes: The interface of the distributed object storage is invoked to map the GRIB message to a GRIB object in the distributed object storage, and the GRIB object is written into the distributed object storage. Call the encoding / decoding tool interface to extract the descriptive attribute information from the GRIB message, and map the descriptive attribute information from the GRIB message to the metadata of the corresponding GRIB object; Generate a key set based on the metadata of the GRIB object, and establish a mapping relationship between the key set and the GRIB object.

[0010] In some embodiments of this application, generating a set of key names based on the metadata of the GRIB object includes: Several core pieces of information are extracted from the descriptive attribute information in the GRIB message; Based on multiple core pieces of information, generate multiple corresponding core key names; Generate auxiliary key names; the auxiliary key names include at least one of the following: the path to the inbound file, the size of the GRIB message, the feature name, and the feature unit; A set of key names is generated based on the multiple core key names and the auxiliary key names.

[0011] In some embodiments of this application, the data processing includes spatial clipping and spatial trimming; Data processing of at least one GRIB object in the GRIB object list includes: At least one GRIB object in the GRIB object list is spatially clipped to obtain at least one GRIB object after spatial clipping. Spatial interpolation is performed on at least one of the GRIB objects after spatial clipping to obtain at least one GRIB object after spatial interpolation.

[0012] In some embodiments of this application, the process of performing spatial interpolation includes: Pre-calculate interpolation weights; Spatial interpolation processing is performed on the GRIB object after spatial interpolation processing according to the preset interpolation algorithm and the interpolation weight.

[0013] In some embodiments of this application, the process of outputting the processed GRIB object as a GRIB file includes: The binary byte stream of the data body of the GRIB object after data processing is output as a GRIB file.

[0014] In some embodiments of this application, the naming of the GRIB file includes any one of the following: date naming, feature naming, and date plus feature naming.

[0015] A second aspect of this application provides a large-scale GRIB grid data distributed storage device, the device comprising: The write request and response module is used to respond to GRIB data write requests and obtain GRIB grid data; The data reading module is used to read GRIB messages and descriptive attribute information from the GRIB grid data. The data mapping module is used to map the GRIB message to a GRIB object in the distributed object storage, and to map the descriptive attribute information in the GRIB message to the metadata of the corresponding GRIB object; The object writing module is used to write the GRIB object to the distributed object storage, generate a set of key names based on the metadata of the GRIB object, and establish a mapping relationship between the set of key names and the corresponding GRIB object; The read request and response module is used to respond to read requests for GRIB data and obtain key-value pairs for retrieval. An object reading module is used to read a list of GRIB objects from the distributed object storage based on the key-value pairs and the mapping relationship; the list of GRIB objects contains at least one GRIB object that meets the retrieval requirements of the key-value pairs; The object processing and output module is used to process the data of at least one GRIB object in the GRIB object list, and output at least one corresponding GRIB file for the at least one GRIB object after data processing.

[0016] A third aspect of this application provides an electronic device including at least one controller and a memory for communicatively connecting to the controller; the memory stores instructions executable by the at least one controller, the instructions being executed by the at least one controller to cause the at least one controller to perform a large-scale GRIB grid data distributed storage method as described above.

[0017] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions for causing a computer to perform a large-scale GRIB grid data distributed storage method as described above.

[0018] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of a large-scale GRIB grid data distributed storage method provided in one embodiment of this application; Figure 2 This is a flowchart illustrating an embodiment of a large-scale GRIB grid data distributed storage method provided in another embodiment of this application; Figure 3 This is a schematic diagram of a GRIB file object-oriented design method provided in one embodiment of this application; Figure 4 This is a flowchart of the write function processing provided in one embodiment of this application; Figure 5 This is a flowchart illustrating the retrieval and processing of GRIB objects according to an embodiment of this application; Figure 6 This is a flowchart of the read function processing provided in one embodiment of this application; Figure 7 This is a flowchart of the spatial clipping function processing provided in one embodiment of this application; Figure 8 This is a flowchart of a spatial interpolation function processing according to an embodiment of this application; Figure 9 This is a flowchart of an integrated mesh field reading and processing function provided in one embodiment of this application; Figure 10 This is a structural block diagram of a large-scale GRIB grid data distributed storage device provided in one embodiment of this application; Figure 11 This is a structural block diagram of an electronic device provided in one embodiment of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] With the rapid development of Earth observation, high-performance computing, and assimilation and modeling, the types of numerical model data in Earth system science are constantly increasing, and the spatiotemporal resolution is continuously improving, resulting in an exponential increase in data volume. Satellite observations, high-resolution models, and reanalysis products are continuously being released. The European Centre for Medium-Range Weather Forecasts (ECMWF) adds approximately 400 TB of GRIB-formatted grid data daily, with individual files often containing hundreds to thousands of grid fields. Traditional data access methods suffer from problems such as coarse granularity of retrieval, poor data consistency, and low access efficiency; the separation of files and indexes easily leads to inconsistencies between files and indexes; and cross-file grid field retrieval and aggregation result in complex operations and low efficiency.

[0023] For example, in meteorological data processing systems, when temperature field data for a specific time range and geographical area is extracted, multiple GRIB files are scanned sequentially, each containing hundreds to thousands of grid fields. Due to inconsistencies between file and index information, some files may lack valid index entries, making it impossible to locate the target grid field. Simultaneously, cross-file aggregation operations require multiple independent file reads and manual data integration, resulting in additional processing steps, lengthening the data access path, and increasing the risk of errors. Specifically, user-specified search requests with single attribute conditions (such as specific feature types and timestamps) are forcibly expanded to the entire file range, making it impossible to directly locate the target grid field. The file-index divide-and-conquer mechanism further desynchronizes index updates with file writes, compromising the integrity of search results.

[0024] like Figures 1 to 3 As shown in one embodiment of this application, a method for large-scale GRIB grid data distributed storage is provided, the method comprising: Step S100: Respond to the GRIB data write request and obtain the GRIB grid data; The GRIB grid data primarily consists of GRIB messages, each message including a set of latitude and longitude coordinates, an array of variables, and descriptive attribute information.

[0025] Step S200: Read the GRIB messages and descriptive attribute information in the GRIB mesh data; This can be read using GRIB encoding / decoding tools.

[0026] Step S300: Map the GRIB message to a GRIB object in the distributed object storage, and map the descriptive attribute information in the GRIB message to the metadata of the corresponding GRIB object; Mapping GRIB messages to GRIB objects in distributed object storage means treating each GRIB message as an independent storage entity. Mapping descriptive attribute information to metadata of the corresponding GRIB object means managing the attribute information in the message as additional information for the object.

[0027] Step S400: Write the GRIB object to the distributed object storage, generate a key name set based on the metadata of the GRIB object, and establish a mapping relationship between the key name set and the corresponding GRIB object. Specifically, generating a set of keys and establishing a mapping relationship based on the metadata of a GRIB object refers to constructing an index key for retrieval based on metadata attributes. Specifically, keys can be generated using a combination of a fixed time range and spatial coordinates, or unique keys can be generated by encoding attribute information using the SHA-256 hash algorithm.

[0028] Step S500: Respond to the GRIB data read request and obtain the key-value pairs used for retrieval; Step S600: Based on the key-value pairs and mapping relationships, retrieve the GRIB object list from the distributed object storage; the GRIB object list contains at least one GRIB object that meets the key-value pair retrieval requirements. Step S700: Perform data processing on at least one GRIB object in the GRIB object list, and output at least one corresponding GRIB file for the at least one GRIB object after data processing.

[0029] In a large-scale distributed storage method for GRIB grid data, GRIB grid data is retrieved upon receiving a write request. GRIB messages and their descriptive attribute information within the GRIB grid data are read; this process is achieved by parsing the detailed features of individual messages, laying the foundation for subsequent fine-grained operations. Each GRIB message is mapped to an independent GRIB object in the distributed object storage, and the descriptive attribute information in the GRIB message is mapped to the metadata of the corresponding GRIB object, thus achieving integrated storage of data and indexes. GRIB objects are written to the distributed object storage, and a set of key names is generated based on the GRIB object's metadata. The mapping relationship between this key name set and the corresponding GRIB object is established. When a read request for GRIB data is responded to, key-value pairs for retrieval are obtained. Based on these key-value pairs and the pre-established mapping relationship, a list of GRIB objects that meet the retrieval requirements is read from the distributed object storage; this list contains at least one GRIB object. Data processing is performed on the GRIB objects in the GRIB object list, and the processed GRIB objects are output as corresponding GRIB files.

[0030] For example, when processing the daily updates of GRIB data from the European Centre for Medium-Range Weather Forecasts (ECMWF), each GRIB message is parsed into an independent storage object in a distributed object store. Its descriptive attributes, such as timestamps, spatial extent, and feature type, are extracted and stored as metadata. A set of keys is generated based on this metadata, such as keys constructed using time, space, or feature information, allowing users to perform efficient searches by specifying a specific time range or geographic region. As a preferred implementation, this method separates the message from its attribute information during the write phase, making each GRIB message an independently operable unit, avoiding the limitations of traditional file-level processing.

[0031] This method effectively solves the problem of coarse retrieval granularity in large-scale GRIB grid data storage by decomposing GRIB messages into independent storage units and combining them with a metadata-driven indexing mechanism, enabling fine-grained data access from the file level to the message level. Integrated data and index storage ensures data consistency, fundamentally eliminating the inconsistency risks caused by file and index divide-and-conquer. Metadata-based key name sets enable search conditions to accurately match object attributes, significantly improving access efficiency and simplifying the complexity of cross-file grid field retrieval and aggregation operations, thereby enhancing access efficiency and simplifying cross-file retrieval operations.

[0032] Specifically, the above scheme proposes to map GRIB messages to GRIB objects and write them to distributed object storage to achieve efficient data management. However, if the write operation is performed synchronously, the data reading process will be blocked by the I / O response delay of the storage system.

[0033] In response, this application further proposes a method of using a main process and asynchronous child processes to write GRIB objects into a distributed object storage; The main process includes: Call the codec tool interface to read each GRIB packet from the GRIB grid data and start an asynchronous processing subprocess; Each time an asynchronous processing subprocess is started, and there are still unread GRIB packets in the GRIB grid data, the GRIB codec tool interface is called to read the next GRIB packet from the GRIB grid data and start the asynchronous processing subprocess. The main process terminates each time an asynchronous processing subprocess is started and all GRIB messages in the GRIB grid data have been read. Asynchronous child process processing includes: Call the distributed object storage interface to map the GRIB message to a GRIB object in the distributed object storage, and write the GRIB object to the distributed object storage; call the codec tool interface to extract the descriptive attribute information in the GRIB message, and map the descriptive attribute information in the GRIB message to the metadata of the corresponding GRIB object; Generate a key set based on the metadata of the GRIB object, and establish a mapping relationship between the key set and the GRIB object.

[0034] The main process is responsible for the scheduling and management of continuous reading and asynchronous processing subprocesses of GRIB grid data. It can be implemented using the process creation mechanism provided by the operating system. The asynchronous processing subprocess can be understood as an auxiliary process that runs independently of the main process. It is dedicated to handling the storage operations of GRIB objects. It can be implemented using a multi-threaded or multi-process model. The purpose is to separate time-consuming I / O operations from the main process and release the computing resources of the main process. The purpose of calling the encoding and decoding tools is to efficiently parse the GRIB message structure and extract key information. The purpose of calling the distributed object storage interface is to persist GRIB objects to the distributed storage system. The key set generation refers to building index identifiers based on the metadata of GRIB objects. It can be generated using hash functions or string concatenation. The purpose is to quickly establish and retrieve the mapping relationship between objects and metadata.

[0035] Specifically, this embodiment achieves pipelined processing of data read and write operations through the collaborative work of a main process and asynchronous processing subprocesses. The main process continuously calls the encoding / decoding tool interface to read GRIB messages, launching an asynchronous processing subprocess for each message read, ensuring that the data reading process is not blocked by I / O operations. When the GRIB grid data has not been completely read, the main process immediately reads the next message after launching the subprocess, ensuring the continuity of the input stream. Once the data reading is complete, the main process terminates to avoid resource waste. Simultaneously, each asynchronous processing subprocess independently performs the GRIB object write operation, including calling the distributed object storage interface to map messages to objects, extracting descriptive attribute information as metadata, and generating a key set to establish mapping relationships. This architecture design allows data reading, metadata processing, and object writing to overlap in time, effectively utilizing the parallel capabilities of multi-core processors and avoiding CPU idleness during I / O wait periods, thereby significantly improving the system's processing efficiency and throughput for large-scale GRIB data.

[0036] The method provided in this embodiment effectively avoids I / O response delay blocking caused by synchronous writing when processing large-scale GRIB grid data, makes full use of the parallel computing capabilities of multi-core processors, reduces CPU resource idleness, and significantly improves data writing throughput and overall system response efficiency.

[0037] like Figure 4 The following is an explanation of the principle of writing GRIB objects to distributed object storage (pseudocode is used to illustrate the design principle): (1) Function declaration form: int gors_write(string grib_fpath).

[0038] (2) Function description: Write a GRIB file to the distributed object storage and return the number of GRIB objects generated, which should be equal to the number of messages in the GRIB file.

[0039] (3) Input parameter: grib_fpath, which is the path to the input GRIB file, and its type is string.

[0040] (4) Function return value: The function return value represents the number of GRIB objects created, and its type is integer.

[0041] (5) Function Processing Logic: To improve processing efficiency, the asynchronous processing logic is presented in a multi-process manner. The main process task calls the interface of the encoding / decoding tool, and starts an asynchronous processing sub-process task for each message read and parsed; after the asynchronous processing sub-process task starts, it does not block the main process, which can significantly improve processing efficiency. The asynchronous processing sub-process task calls the object storage interface to create a GRIB object and set its metadata. When processing metadata, it is necessary to call the encoding / decoding tool interface to obtain the self-description information of the GRIB message.

[0042] (6) Functional expansion design; To facilitate feature configuration, this function can be further extended. Its function declaration format is as follows: int gors_write(string[] grib_fpaths, string[]keys).

[0043] In some embodiments of this application, step S400, which generates a set of key names based on the metadata of the GRIB object, includes: Step S410: Extract multiple core pieces of information from the descriptive attribute information in the GRIB message; Specifically, multiple core information refers to the key attributes in the GRIB message that describe the essence of the data. These can be implemented using information such as timestamps, latitude and longitude coordinates, and forecast timeliness. The purpose is to focus on the inherent characteristics of the data to improve retrieval accuracy. Among them, multiple core key names can be understood as structured identifiers generated based on core information. These can be generated using hash encoding or string concatenation. The purpose is to strengthen the logical relationship of the data and support multi-dimensional queries.

[0044] Step S420: Generate multiple corresponding core key names based on multiple core information; Step S430: Generate auxiliary key names; the auxiliary key names include at least one of the following: import file path, GRIB message size, feature name, and feature unit; In practical applications, auxiliary key names are key names for supplementary dimensions, such as storage path, data size, business elements, etc., with the aim of expanding the scope of search conditions to meet diverse query needs.

[0045] Step S440: Generate a set of key names based on multiple core key names and auxiliary key names.

[0046] Specifically, a key name set refers to a combination of core key names and auxiliary key names. It can be stored using a list or set data structure, with the aim of building a hierarchical key name system to optimize data location efficiency.

[0047] Specifically, this embodiment first extracts multiple core information from the descriptive attribute information of the GRIB message. This core information represents the key dimensions of the data. Then, based on this core information, multiple corresponding core key names are generated, transforming abstract attributes into queryable identifiers. At the same time, at least one auxiliary key name, including the path to the file to be stored, is generated to supplement the storage and business dimensions. Finally, the core key names and auxiliary key names are integrated to form a key name set. This set serves as the basis for mapping relationships in distributed object storage, enabling efficient matching of multi-condition queries during retrieval, thereby achieving rapid data location and reducing consistency risks.

[0048] This embodiment provides a method that enables fine-grained construction of the key name set, allowing data retrieval to support multi-condition combined queries, significantly improving retrieval granularity and query dimensions, thereby effectively improving the efficiency of cross-file data location and reducing the risk of data inconsistency caused by the separation of index and file.

[0049] In the above embodiments, data processing steps are proposed to generate GRIB files. However, the specific implementation method of data processing is not clearly defined, resulting in the absence of spatial clipping and spatial interpolation operations. This causes the output GRIB file to contain a large amount of irrelevant regional data, increasing the storage and transmission burden. At the same time, the resolution mismatch reduces the data accuracy and applicability, affecting the efficiency and accuracy of subsequent meteorological analysis.

[0050] In this regard, such as Figure 5 As shown in the embodiments of this application, data processing is further proposed, including spatial clipping and spatial interpolation; Step S700, which involves performing GRIB object data processing on at least one GRIB object in the GRIB object list, includes: Step S710: Perform spatial clipping on at least one GRIB object in the GRIB object list to obtain at least one GRIB object after spatial clipping. Spatial clipping refers to the operation of accurately extracting relevant grid data based on the geographical boundaries specified by the user. It can be achieved by using a coordinate range filtering algorithm based on geographic information systems or a vector polygon clipping tool. The purpose is to eliminate the interference of irrelevant areas in the original data and significantly reduce the data volume.

[0051] Step S720: Perform spatial interpolation on at least one GRIB object after spatial clipping to obtain at least one GRIB object after spatial interpolation.

[0052] Spatial interpolation can be understood as the process of adjusting the resolution based on local data characteristics. It can be implemented using algorithms independent of pre-calculated weights, such as bilinear interpolation, kriging interpolation, or spline interpolation. The goal is to adapt to the resolution requirements of the target application scenario and improve data accuracy and applicability.

[0053] Specifically, this embodiment first performs spatial clipping on the GRIB objects in the GRIB object list. Relevant grid data is precisely extracted based on the user-specified geographic region boundaries to generate spatially clipped GRIB objects, thereby eliminating irrelevant regional data and reducing data volume. Subsequently, spatial interpolation is performed on the spatially clipped GRIB objects. Based on the local characteristics of the clipped data, an interpolation algorithm is applied to adjust the resolution, generating spatially interpolated GRIB objects. This orderly arrangement of steps ensures that spatial clipping provides a compact dataset for subsequent spatial interpolation, allowing interpolation operations to be applied only to key regions, avoiding redundant calculations in full-region processing. This reduces storage and transmission burden while improving data accuracy and applicability.

[0054] This embodiment can effectively eliminate irrelevant area data in the output GRIB file, significantly reducing the storage and transmission burden. At the same time, it improves data accuracy and applicability through resolution adaptation, thereby improving the efficiency and accuracy of subsequent meteorological analysis.

[0055] In the above implementation, spatial interpolation is proposed to realize coordinate transformation or resolution adjustment of grid data. However, in its implementation, each interpolation operation requires real-time calculation of interpolation weights, which leads to increased repetitive calculation overhead. Especially in the scenario of large-scale GRIB grid data processing, it significantly reduces data processing efficiency and increases system resource consumption.

[0056] In this regard, this application further proposes a spatial interpolation process including: Step S721: Pre-calculate interpolation weights; Step S722: Perform spatial interpolation processing on the GRIB object after spatial interpolation processing according to the preset interpolation algorithm and the interpolation weight.

[0057] This embodiment moves the calculation of interpolation weights to the data processing preparation stage, eliminating the need to re-parse the grid topology during subsequent spatial interpolation. When the system receives a spatial interpolation instruction, it directly calls the pre-stored interpolation weights and preset algorithm logic to perform weighted calculations on the GRIB object, thus forming a highly efficient data processing pipeline: first, the weights are pre-calculated and stored, and then only simple numerical calculations are needed in the actual interpolation stage, significantly reducing computational complexity. This design fully utilizes the relatively stable grid layout in GRIB data, decoupling the originally dynamically coupled calculation process into independent preprocessing and execution stages, effectively avoiding the bottleneck of repetitive calculations.

[0058] This application further proposes a process for outputting the processed GRIB object as a GRIB file, including: Output the binary byte stream of the data body of the GRIB object after data processing as a GRIB file.

[0059] The binary byte stream output of the data body refers to the direct generation of a GRIB file using the raw binary data stored in the GRIB object. This can be achieved using a standard file system write interface or a streaming protocol. The purpose is to avoid data decoding and re-encoding, ensuring that the output process only involves the transmission of physical byte streams rather than logical conversion, thereby eliminating the risk of conversion errors and reducing system resource consumption.

[0060] This embodiment identifies that the GRIB object has fully preserved the binary structure of the original message in the distributed storage, so that the data body naturally conforms to the GRIB file format specification. After the data processing is completed, the binary byte stream of the data body is directly extracted as the output content without performing additional format parsing or reorganization operations. This creates an efficient direct path from the storage object to the target file, significantly optimizing the data flow efficiency.

[0061] like Figure 6 As shown, the principle of reading GRIB files from distributed object storage is explained based on object retrieval and processing: (1) Function declaration form: int gors_read(map<string, string> []kv_pairs, string[]required_keys, string target).

[0062] (2) Function description: Read GRIB objects that meet the metadata retrieval conditions from the distributed object storage and output them to a GRIB file.

[0063] (3) Input parameters: ① Metadata key-value pairs, kv_pairs, are dictionaries used to specify the matching conditions for GRIB objects; ②Required metadata keys, required_keys, are a list of strings used to specify the key names that must be included in the key-value pairs in ①; ③ Output filename, target, which is a string and is used to specify the location to save the file after reading.

[0064] (4) Function return value: The function return value represents the number of GRIB objects that meet the metadata conditions, and its type is integer.

[0065] (5) Function processing logic: Based on the input metadata conditions, retrieve GRIB objects that meet the conditions by calling the metadata table query interface of the distributed object storage; for the matched GRIB objects, directly output the binary byte stream of the object's data body as a GRIB file.

[0066] like Figure 7 As shown, the principle of spatial clipping is explained: Taking the eccodes encoding / decoding interface as an example, the codes_handle (or grib_handle) structure is used to represent the GRIB message loaded into memory, and the interface design is based on the processing of a single GRIB message.

[0067] (1) Function declaration form: grib_handle gors_crop(grib_handle h, string area).

[0068] (2) Function description: Perform space trimming operation on the specified GRIB message.

[0069] (3) Input parameters: ① The processing handle h, which has loaded the GRIB message into memory, is a pointer to a grib_handle structure. ② The target area for spatial clipping, area, is a string.

[0070] (4) Function return value: The function returns a new GRIB message in memory (after space trimming), which is a pointer to a grib_handle structure.

[0071] (5) Function processing logic: When implementing this, consider directly calling interfaces such as MIR and cdo toolkits.

[0072] (6) Functional expansion design: ① Directly process objects in the distributed object store. Its interface declaration can be designed as follows: (A) grib_handle gors_crop(string obj_key, string area); This function interface uses the key obj_key of the distributed object storage to retrieve a GRIB object, whose data body is a GRIB message. This message is loaded into memory using the GRIB encoding tool interface, and the rest of the operations are the same.

[0073] (B) string gors_crop(string obj_key, string area); This function interface is the same as (A), but in terms of result processing, it stores the new GRIB as an object and returns the key of its object storage.

[0074] ② Integration with mesh field reading. The spatial clipping function can be integrated with the mesh field reading function, and its output is saved as a file.

[0075] like Figure 8 The principle of spatial interpolation is explained as follows: Generally, interpolation weights are pre-calculated, and then all subsequent interpolation algorithms can use these weights to improve interpolation efficiency. Therefore, this function was designed with a focus on two parameters: the interpolation algorithm and the interpolation weights.

[0076] (1) Function declaration form: grib_handle gors_interpolation (grib_handle h, stringgrid, string method, FILE (weights_map=null).

[0077] (2) Function description: Perform spatial interpolation operation on the specified GRIB message.

[0078] (3) Input parameters: ① A handler, h, for loading the GRIB message into memory; it is a pointer to a grib_handle structure. Note that this structure is the GRIB message structure defined by eccodes encoding / decoding for loading into memory. ② The target grid for horizontal spatial interpolation, grid, is of type string and has the form grid=lon / lat, which represents the interpolation resolution in the longitudinal and latitudinal directions, respectively; ③ Interpolation algorithm name, method, of type string, used to specify the interpolation algorithm used, such as Nearestneighbor (default), Bilinear, Bicubic, etc.

[0079] ④ Interpolation weights, type FILE The pre-calculated interpolation weights (saved in a file) are initially set to NULL by default and need to be calculated.

[0080] (4) Function return value: The function returns a new GRIB message in memory (after spatial interpolation), which is a pointer to a grib_handle structure.

[0081] (5) Function processing logic: Consider calling interfaces such as MIR and cdo toolkit.

[0082] (6) Functional expansion design: ① Directly process objects in the distributed object store. Its interface declaration can be designed as follows: (A) grib_handle gors_interpolation (string obj_key, string grid, string method, FILE weights_map=null); This function interface uses the key obj_key of the distributed object storage to retrieve a GRIB object, whose data body is a GRIB message. This message is loaded into memory using the GRIB encoding tool interface, and the rest of the operations are the same.

[0083] (B) string gors_interpolation (string obj_key, string grid, stringmethod, FILE weights_map=null); This function interface is the same as (A), but in terms of result processing, it stores the new GRIB as an object and returns the key of its object storage.

[0084] ② Integration with mesh field reading. Spatial clipping and spatial interpolation functions can be integrated with the mesh field reading function, and the output is saved as a GRIB file.

[0085] like Figure 9As shown, the principles of the integrated design of spatial clipping, spatial interpolation, and reading functions are explained: (1) Function declaration form: int gors_power_read(map<string, string> []kv_pairs,string []required_keys, string target, string area=null, string grid=null, string method=null).

[0086] (2) Functional description: Reads GRIB objects that meet the metadata retrieval conditions from the distributed object storage, and can perform spatial clipping and spatial interpolation processing simultaneously, and finally outputs and saves them to a GRIB file.

[0087] (3) Input parameters: ① Metadata key-value pairs, kv_pairs, are dictionaries used to specify the matching conditions for GRIB objects; ②Required metadata keys, required_keys, are a list of strings used to specify the key names that must be included in the key-value pairs in ①; ③ Output filename, target, which is a string and is used to specify the location to save the file after reading; ④ The target rectangle for spatial clipping, area, is of type string; ⑤ The target grid for horizontal spatial interpolation, grid, is of type string; ⑥ Interpolation algorithm name, method, of type string, used to specify the interpolation algorithm used.

[0088] (4) Function return value: The function return value represents the number of GRIB messages in the output file, and its type is integer.

[0089] (5) Function processing logic: Perform spatial interpolation and spatial clipping as needed, and output as a file.

[0090] To ensure the accuracy of interpolation at the edges, when interpolation and clipping are performed simultaneously, the interpolation operation should be performed first, followed by the clipping operation.

[0091] In practical applications, some of the solutions mentioned above in this application propose outputting the processed GRIB object as a GRIB file. However, in this process, the lack of a unified naming standard leads to chaotic file management, low retrieval efficiency, and the risk of inconsistency between the file and the index, making it difficult to support rapid positioning and aggregation operations based on time or feature dimensions.

[0092] In this regard, this application further proposes that the naming of GRIB files include: date naming, feature naming, or date plus feature naming.

[0093] Among them, date naming refers to a file naming method based on time information, which can be implemented using the year-month-day format (such as YYYYMMDD). The purpose is to organize files by time dimension, making it easier to quickly locate data in a specific time period during retrieval. Element naming refers to a file naming method based on meteorological element types, which can be implemented using element names (such as temperature, wind speed). The purpose is to organize files by element category to meet the specific needs of different analysis scenarios. Date plus element naming refers to a file naming method that combines time information and element type. It can be implemented using a combination of date and element name (such as YYYYMMDD_element name). The purpose is to provide more precise retrieval granularity and reduce interference from irrelevant data.

[0094] This embodiment effectively solves the management and retrieval problems caused by inconsistent file naming, enabling files to be efficiently organized and retrieved based on time or element dimensions, avoiding the inefficiency of full-screen scanning, reducing the risk of inconsistency between files and indexes, and improving the efficiency and reliability of data processing.

[0095] One embodiment of this application provides a large-scale GRIB grid data distributed storage method, and the hardware system used in the embodiment is as follows: The prototype system consists of 3 nodes, each equipped with 2×Intel Xeon Silver 4314 CPUs and 256 GB of memory; Storage is tiered by role: 2 x 3.2 TB NVMe SSDs as metadata disks, 4 x 3.2 TB SATA SSDs as cache disks, 20 x 10.91 TB SATA HDDs for data storage, and 2 x 960 GB SATA SSDs as system disks; The network side provides a 1×1 GE management network, 2×10 GE service networks and 2×10 GE trunking networks to achieve high throughput, low latency and redundancy protection.

[0096] The method includes the following steps: Step S9110: Distributed system selection and software implementation.

[0097] Step S9120: Obtain the test dataset.

[0098] In step S910, the "Massive Object Storage MOS" developed by Sugon Data Technology Co., Ltd. is selected as the underlying platform. Its design specifications can support EB-level capacity and hundreds of billions of objects. Following the Amazon S3 standard interface protocol, it can theoretically be seamlessly replaced by open-source object storage solutions such as Ceph and MinIO. Due to implementation time constraints, the prototype only focuses on key technology verification. The core functions already implemented cover key modules such as GRIB object-oriented storage, metadata management, retrieval, and post-processing. Other design aspects will be further improved in subsequent iterations.

[0099] In step S920, the dataset source is hourly data from the ERA5 multi-barosphere system, containing 37 barospheres. The selected data conditions are four time intervals (00:00, 06:00, 12:00, and 18:00) from January 1, 2004 to November 1, 2004, primarily including vertical velocity, zonal wind speed, meridional wind speed, and three-dimensional elements such as temperature, relative humidity, specific humidity, and geopotential. Each file represents an hourly element, approximately 512MB in size, with a total storage space of 612GB. After objectifying and storing this dataset, 315,800 objects are generated, with an object storage space of 612GB.

[0100] In some embodiments of this application, the experimental verification of key technologies includes the following steps: Step S9210, objectification of the GRIB file; Step S9220: Metadata definition of the GRIB object; Step S9230: Retrieval and processing of GRIB objects; Step S9230: Command line interaction.

[0101] In step S9210, the specific steps are as follows: (1) Use the grib_ls tool to view the file, which is composed of 259 GRIB messages, and the self-description information of each message is as follows; (2) Select the file to upload in the storage bucket; ③ The file upload is successful, the GRIB object storage is completed, and 259 objects are generated, which is consistent with the number of messages in the file. After the GRIB object storage is completed.

[0102] In step S9220, the metadata of an object in the bucket includes 12 core metadata and 8 custom auxiliary metadata.

[0103] In step S9230, the specific steps are as follows: (1) Start the search interface; (2) Set the metadata search conditions; (3) Set the data post-processing options and send a download request; (4) The gosr system has completed the search and found 296 matching GRIB objects, and started data post-processing and downloading; (5) The file download is complete; (6) The results can be verified using grib tools, such as grib_dump or visual tools like metview.

[0104] In step S9240, the specific steps are as follows: (1) When writing data, use the gors write command to store a GRIB file as an object; (2) When retrieving data, use the gors read command to retrieve data based on metadata and save it as a GRIB file. At the same time, the command can set the --area and --grid options to realize spatial pruning and spatial interpolation post-processing of the retrieval results; (3) Verify whether the number of objects and features are consistent; (4) Verify whether the post-processing operation is completed. This method has at least the following beneficial effects: Compared to the traditional "file + index" approach, this application directly objectifies each GRIB message, enabling fine-grained management at the grid field level. Based on virtual keys using 12 core metadata, it can locate objects in milliseconds, significantly improving retrieval efficiency. Leveraging a parallel pipeline framework for distributed object storage, spatial pruning and interpolation can be horizontally scaled to EB-level, with throughput growing linearly with the number of nodes. The standard S3 interface ensures seamless replacement with backends such as Ceph and MinIO, while metadata uniqueness constraints and timestamp mechanisms prevent data inconsistencies. Overall, it provides a highly scalable, highly reliable, and high-performance unified storage and analysis platform for massive GRIB data.

[0105] like Figure 10 One embodiment of this application provides a large-scale GRIB grid data distributed storage device, the device comprising: The write request response module 1001 is used to respond to GRIB data write requests and obtain GRIB grid data; The data reading module 1002 is used to read GRIB messages and descriptive attribute information from GRIB grid data. The data mapping module 1003 is used to map GRIB messages to GRIB objects in the distributed object storage, and to map the descriptive attribute information in the GRIB messages to the metadata of the corresponding GRIB objects; The object writing module 1004 is used to write GRIB objects to distributed object storage and generate a key name set based on the metadata of the GRIB objects to establish a mapping relationship between the key name set and the corresponding GRIB objects; The read request response module 1005 is used to respond to GRIB data read requests and obtain key-value pairs for retrieval; The object reading module 1006 is used to read a list of GRIB objects from the distributed object storage based on key-value pairs and mapping relationships; the list of GRIB objects contains at least one GRIB object that meets the retrieval requirements of key-value pairs; The object processing and output module 1007 is used to process the data of at least one GRIB object in the GRIB object list, and output the processed data of at least one GRIB object into at least one corresponding GRIB file.

[0106] It should be noted that the large-scale GRIB grid data distributed storage device provided in this embodiment and the large-scale GRIB grid data distributed storage method described above are based on the same inventive concept. Therefore, the content of the large-scale GRIB grid data distributed storage device described in this embodiment is also applicable to the content of the large-scale GRIB grid data distributed storage method described above, and will not be repeated here.

[0107] like Figure 11 An embodiment of this application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned large-scale GRIB grid data distributed storage method. The electronic device includes: At least one battery; At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the large-scale GRIB grid data distributed storage method described above in this disclosure.

[0108] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0109] The electronic devices according to embodiments of this application will now be described in detail.

[0110] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to implement a large-scale GRIB grid data distributed storage method according to the embodiments of this disclosure.

[0111] The input / output interface 1800 is used to implement information input and output. The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900); The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0112] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described large-scale GRIB grid data distributed storage method.

[0113] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0114] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0115] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0118] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0119] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0120] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0121] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0122] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0123] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0124] The above is a detailed description of the preferred embodiments of this application. However, the embodiments of this application are not limited to the above-described implementation methods. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the embodiments of this application. All such equivalent modifications or substitutions are included within the scope defined by the claims of the embodiments of this application.

Claims

1. A large-scale GRIB grid data distributed storage method, characterized in that, The method includes: Respond to GRIB data write requests and retrieve GRIB grid data; Read the GRIB messages and descriptive attribute information from the GRIB mesh data; The GRIB message is mapped to a GRIB object in the distributed object storage, and the descriptive attribute information in the GRIB message is mapped to the metadata of the corresponding GRIB object; Write the GRIB object into the distributed object storage, generate a set of key names based on the metadata of the GRIB object, and establish a mapping relationship between the set of key names and the corresponding GRIB object; Respond to GRIB data read requests and retrieve key-value pairs for retrieval; Based on the key-value pairs and the mapping relationship, a list of GRIB objects is retrieved from the distributed object storage; the list of GRIB objects contains at least one GRIB object that meets the retrieval requirements of the key-value pairs. For at least one GRIB object in the GRIB object list, perform GRIB object data processing, and output at least one corresponding GRIB file for the at least one GRIB object after data processing.

2. The large-scale GRIB grid data distributed storage method according to claim 1, characterized in that, The GRIB object is written to the distributed object storage using a main process and asynchronous child processes. The main process includes: The codec tool interface is invoked to read each GRIB message from the GRIB grid data and to start an asynchronous processing subprocess. Each time the asynchronous processing subprocess is started, and there are still unread GRIB packets in the GRIB grid data, the GRIB codec tool interface is called to read the next GRIB packet from the GRIB grid data, and the asynchronous processing subprocess is started. The main process terminates each time the asynchronous processing subprocess is started and all GRIB messages in the GRIB grid data are read. The asynchronous processing subprocess includes: The interface of the distributed object storage is invoked to map the GRIB message to a GRIB object in the distributed object storage, and the GRIB object is written into the distributed object storage. Call the encoding / decoding tool interface to extract the descriptive attribute information from the GRIB message, and map the descriptive attribute information from the GRIB message to the metadata of the corresponding GRIB object; Generate a key set based on the metadata of the GRIB object, and establish a mapping relationship between the key set and the GRIB object.

3. The large-scale GRIB grid data distributed storage method according to claim 2, characterized in that, The step of generating a set of key names based on the metadata of the GRIB object includes: Several core pieces of information are extracted from the descriptive attribute information in the GRIB message; Based on multiple core pieces of information, generate multiple corresponding core key names; Generate auxiliary key names; the auxiliary key names include at least one of the following: the path to the inbound file, the size of the GRIB message, the feature name, and the feature unit; A set of key names is generated based on the multiple core key names and the auxiliary key names.

4. The large-scale GRIB grid data distributed storage method according to claim 1, characterized in that, The data processing includes spatial clipping and spatial clipping; Data processing of at least one GRIB object in the GRIB object list includes: At least one GRIB object in the GRIB object list is spatially clipped to obtain at least one GRIB object after spatial clipping. Spatial interpolation is performed on at least one of the GRIB objects after spatial clipping to obtain at least one GRIB object after spatial interpolation.

5. A large-scale GRIB grid data distributed storage method according to claim 4, characterized in that, The process of performing spatial interpolation includes: Pre-calculate interpolation weights; Spatial interpolation processing is performed on the GRIB object after spatial interpolation processing according to the preset interpolation algorithm and the interpolation weight.

6. The large-scale GRIB grid data distributed storage method according to claim 4, characterized in that, The process of outputting the GRIB file from the processed data GRIB object includes: The binary byte stream of the data body of the GRIB object after data processing is output as a GRIB file.

7. A large-scale GRIB grid data distributed storage method according to claim 6, characterized in that, The naming of GRIB files can include any one of the following: date naming, feature naming, or date plus feature naming.

8. A large-scale GRIB grid data distributed storage device, characterized in that, The device includes: The write request and response module is used to respond to GRIB data write requests and obtain GRIB grid data; The data reading module is used to read GRIB messages and descriptive attribute information from the GRIB grid data. The data mapping module is used to map the GRIB message to a GRIB object in the distributed object storage, and to map the descriptive attribute information in the GRIB message to the metadata of the corresponding GRIB object; The object writing module is used to write the GRIB object to the distributed object storage, generate a set of key names based on the metadata of the GRIB object, and establish a mapping relationship between the set of key names and the corresponding GRIB object; The read request and response module is used to respond to read requests for GRIB data and obtain key-value pairs for retrieval. An object reading module is used to read a list of GRIB objects from the distributed object storage based on the key-value pairs and the mapping relationship; the list of GRIB objects contains at least one GRIB object that meets the retrieval requirements of the key-value pairs; The object processing and output module is used to process the data of at least one GRIB object in the GRIB object list, and output at least one corresponding GRIB file for the at least one GRIB object after data processing.

9. An electronic device, characterized in that, It includes at least one controller and a memory for communicatively connecting with the controller; the memory stores instructions executable by the at least one controller, which, when executed by the at least one controller, causes the at least one controller to perform a large-scale GRIB grid data distributed storage method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions for causing a computer to perform a large-scale GRIB grid data distributed storage method as described in any one of claims 1 to 7.

Images