Method and apparatus for processing meteorological science data files based on time series database

Abstract

The application provides a meteorological scientific data file processing method and device based on a time series database, and relates to the technical field of computers.The method comprises the following steps: establishing a storage model in the time series database, wherein the storage model is used for storing meteorological scientific data files; receiving a calling request of an upper application for a file corresponding to a target directory; in the case where a file access path comprises a target field, loading a message index in a meteorological scientific data file corresponding to the target field from the time series database, and processing the meteorological scientific data file based on the message index and an access operation. The corresponding meteorological scientific data file is obtained through the file access path, and the message index is obtained, so that the processing of the file content can be realized without downloading the complete file, the file processing efficiency is improved, and the file processing cost is reduced.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method and apparatus for processing meteorological scientific data files based on a time-series database. Background Technology

[0002] A large number of grid-based scientific data files are continuously generated in fields such as weather forecasting, remote sensing observation, ocean simulation, geospatial analysis, and Earth system science. Taking the General Regularly-distributed Information in Binary form (GRIB) / GRIB2 file as an example, a single file typically contains multiple messages or grid fields. Each message corresponds to self-descriptive information such as variables, levels, forecast lead times, members, spatial extent, grid size, resolution, and units. In practice, operational analysts often only need to read a specific variable, a certain height level, a certain forecast lead time, a certain spatial extent, or a specific internal message, rather than reading the entire file.

[0003] Existing conventional file systems typically only recognize directory, filename, size, permissions, and time attributes, and cannot identify the variables, hierarchy, grid extent, and message layout within a GRIB file. If a user needs to access a target grid field, they often need to download the entire file first, then call a dedicated format parsing library to scan the file's internal messages and perform decoding, trimming, or interpolation. For massive amounts of meteorological files and high-frequency access scenarios, this approach results in repetitive network transmissions, disk input / output (I / O), CPU decoding, and memory overhead, leading to low file processing efficiency and high overhead. Summary of the Invention

[0004] This invention provides a method and apparatus for processing meteorological scientific data files based on a time-series database, in order to solve the problems of low file processing efficiency and high overhead.

[0005] This invention provides a method for processing meteorological scientific data files based on a time-series database, comprising: A storage model is established in the time-series database, which is used to store meteorological science data files; Receive a request from an upper-layer application to access a file in a target directory, the request including the file access path and access operation; If the file access path includes a target field, the message index of the meteorological scientific data file corresponding to the target field is loaded from the time series database, and the meteorological scientific data file is processed based on the message index and the access operation.

[0006] According to the meteorological scientific data file processing method based on time-series database provided by the present invention, the file access path further includes the target directory and file name; The step of loading the message index from the meteorological scientific data file corresponding to the target field from the time-series database includes: The file access path is parsed to obtain the target directory, the file name, and the target field; Based on the target directory, the file name, and the target field, candidate meteorological science data files are determined from the time-series database; Load the message index from the candidate meteorological science data file.

[0007] The processing of the meteorological scientific data file based on the message index and the access operation includes: Based on the message index, determine the target message corresponding to the message index; Based on the access operation, the content of the meteorological scientific data file corresponding to the target message is processed.

[0008] According to a meteorological scientific data file processing method based on a time-series database provided by the present invention, the step of processing the content of the meteorological scientific data file corresponding to the target message based on the access operation includes: In cases where the access operation includes writing or reading a file, determine whether the format of the file to be written or read belongs to the meteorological scientific data format; If the format of the file to be written or the file to be read is a meteorological scientific data format, the format adapter is called to parse the file to be written or the file to be read, and the parsing result is obtained; Based on the analysis results, the content of the meteorological scientific data file corresponding to the target message is processed.

[0009] According to a meteorological scientific data file processing method based on a time-series database provided by the present invention, the step of processing the content of the meteorological scientific data file corresponding to the target message based on the parsing result includes: If the access operation includes writing to a file, for each message in the file to be written that is included in the parsing result, a message index corresponding to the message is generated, and the metadata corresponding to the message is extracted; Based on the write path included in the write request, the file offset in the file to be written, and the length of the metadata, the metadata is written into the meteorological scientific data file content corresponding to the target message in the time series database.

[0010] According to a meteorological scientific data file processing method based on a time-series database provided by the present invention, the step of processing the content of the meteorological scientific data file corresponding to the target message based on the parsing result includes: When the access operation includes reading a file and the read request is to return raw binary data, the write file is used to read the meteorological scientific data file content corresponding to the target message from the file to be read, which is included in the parsing result and matches the target message, according to the file offset and read length included in the read request.

[0011] According to a meteorological scientific data file processing method based on a time-series database provided by the present invention, the step of processing the content of the meteorological scientific data file corresponding to the target message based on the parsing result includes: When the access operation includes reading a file and the read request returns a numerical array, spatial clipping result, or interpolation result, for the message in the file to be read that matches the target message, which is included in the parsing result, the format decoding and spatial processing module is invoked, or the processing related to the read request is converted into a user-defined function in the time series database; The user-defined function is used to read, decode, spatially prune, or interpolate the meteorological scientific data file content corresponding to the target message in the time series database, so as to obtain the decoded numerical array, spatial pruning result, or interpolation result.

[0012] According to the meteorological scientific data file processing method based on time-series database provided by the present invention, the storage model is also used to store the metadata and message index of the meteorological scientific data file.

[0013] According to the present invention, a method for processing meteorological scientific data files based on a time-series database, the method further includes: The metadata and the message index are determined as extended attributes of the meteorological scientific data file; the extended attributes are used to obtain the semantic information of the meteorological scientific data file without fully downloading it.

[0014] According to the present invention, a method for processing meteorological scientific data files based on a time-series database is provided. The storage model includes a metadata directory table and an object content table. The metadata directory table is used to store the attribute information and metadata of the meteorological scientific data file, and the object content table is used to store the binary objects, object pointers, message indexes, and file size of the meteorological scientific data file.

[0015] According to the present invention, a method for processing meteorological scientific data files based on a time-series database, the method further includes: Write the binary object, the object pointer, the message index, and the file size of the meteorological science data file into the same file version; When the binary object, the object pointer, and the file size of the meteorological science data file are updated to form a new version, the message index is synchronized to form a new version.

[0016] The present invention also provides a meteorological scientific data file processing device based on a time-series database, comprising: A module is established to create a storage model in the time-series database, the storage model being used to store meteorological science data files; The receiving module is used to receive a call request from the upper-layer application for a file corresponding to the target directory. The call request includes the file access path and access operation. The first processing module is configured to, when the file access path includes a target field, load the message index of the meteorological scientific data file corresponding to the target field from the time series database, and process the meteorological scientific data file based on the message index and the access operation.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the meteorological scientific data file processing method based on the time-series database as described above.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the meteorological scientific data file processing method based on a time-series database as described above.

[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the meteorological scientific data file processing method based on a time-series database as described above.

[0020] This invention provides a method and apparatus for processing meteorological scientific data files based on a time-series database. The method establishes a storage model in the time-series database to store meteorological scientific data files. It receives a request from an upper-layer application to access a file corresponding to a target directory; the request includes a file access path and an access operation. If the file access path includes a target field, the method loads a message index from the time-series database corresponding to the target field in the meteorological scientific data file. Based on the message index and the access operation, the method processes the meteorological scientific data file. By using the file access path, the corresponding meteorological scientific data file can be retrieved from the time-series database. Furthermore, by obtaining the message index, file content processing can be achieved without downloading the complete meteorological scientific data file, thus improving file processing efficiency and reducing processing overhead. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the meteorological scientific data file processing method based on a time-series database provided by the present invention.

[0023] Figure 2 This is a schematic diagram of the system architecture of the meteorological scientific data file processing method based on time-series database provided by the present invention.

[0024] Figure 3 This is a schematic diagram of meteorological metadata fusion and tree-table dual model provided by the present invention.

[0025] Figure 4 This is a flowchart of the meteorological data format recognition, streaming metadata extraction, and message-level index construction provided by the present invention.

[0026] Figure 5 This is a flowchart of the extended filename virtual file query, object offset reading, and calculation pushdown provided by the present invention.

[0027] Figure 6 This is a flowchart of the extended attribute semantic retrieval, semantic caching, and cross-file export provided by the present invention.

[0028] Figure 7 This is a schematic diagram of the meteorological scientific data file processing device based on a time-series database provided by the present invention.

[0029] Figure 8This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0031] To facilitate a clear understanding of the various embodiments of this application, the relevant background knowledge will first be introduced.

[0032] The existing technologies generally have the following shortcomings: (1) Standard file systems can only provide byte-level read and write, and cannot perceive the internal structure of meteorological scientific data files. Business programs need to parse the complete files themselves; (2) Meteorological metadata usually exists only in the internal or external index of the file, and lacks consistency with file system metadata, object content and file version; (3) There is no unified index for messages inside the file. When reading a message or a variable, it is often necessary to scan or decode the complete file; (4) Spatial pruning, array decoding and grid resampling are often repeatedly executed in business services or clients, and it is difficult to reuse the database-side computing, caching and distributed capabilities; (5) Query results are often returned through dedicated RPC, database interface or temporary file, and it is difficult to be consumed uniformly by ordinary file tools, script programs and graphical file managers; (6) Cross-file spatiotemporal queries require traversing directories level by level and parsing file by file, and it is difficult to quickly locate candidate files that meet the conditions of variables, levels, regions and time.

[0033] Therefore, a solution is needed to unify format recognition, file-level metadata, message-level indexing, extended attributes, virtual file querying, space processing, and cross-file export to the file system layer without disrupting the standard file system read / write model, and to improve the processing efficiency of meteorological scientific data by leveraging the object storage, querying, versioning, and distributed capabilities of time-series databases.

[0034] The following is combined with Figures 1 to 6 This invention describes a meteorological scientific data file processing method based on a time-series database.

[0035] Figure 1 This is a flowchart illustrating the meteorological scientific data file processing method based on a time-series database provided by the present invention, as shown below. Figure 1 As shown, the method includes the following steps 101-103.

[0036] Step 101: Establish a storage model in the time series database. The storage model is used to store meteorological scientific data files.

[0037] Specifically, a storage model can be established within the time-series database to store meteorological scientific data files. The storage model also stores metadata and message indexes for these meteorological scientific data files. Furthermore, the storage model can store standard files (i.e., ordinary files).

[0038] The storage model includes a metadata directory table and an object content table. The metadata directory table stores attribute information and metadata for meteorological scientific data files, while the object content table stores the binary objects, object pointers, message indexes, and file sizes of the meteorological scientific data files. The metadata directory table also stores attribute information for standard files, and the object content table further stores the binary objects, object pointers, and file sizes of standard files.

[0039] The attribute information of a standard file includes at least one of the following: logical path, parent directory, file name, file size, permissions, user, group, creation time, modification time, and access time. The attribute information of a meteorological scientific data file includes at least one of the following: logical path, parent directory, file name, file size, permissions, user, group, creation time, modification time, and access time. The metadata of a meteorological scientific data file includes at least one of the following: data type, data type, member name, region, element, element unit, hierarchy type, hierarchy value, starting longitude, ending longitude, starting latitude, ending latitude, latitude / longitude step, number of longitude grid points, number of latitude grid points, forecast lead time, forecast lead time unit, grid precision, altitude / pressure unit, resolution, and start time.

[0040] It should be noted that both the metadata directory table and the object content table use logical path and time version as key dimensions to ensure that the metadata, binary objects, object pointers, message indexes, and file sizes of meteorological scientific data files remain consistent.

[0041] Step 102: Receive a request from the upper-layer application to access the file corresponding to the target directory. The request includes the file access path and access operation.

[0042] Specifically, the target directory can be a mount point containing files. File access paths can include regular file access paths and access paths that include target fields. Target fields can be query strings, which can contain function names, variables, levels, regions, output types, interpolation methods, target resolutions, and data types. Access operations include file writing, file reading, and file closing.

[0043] The system receives requests from upper-layer applications for accessing files in the target directory via a filesystem in userspace (FUSE) or an equivalent userspace filesystem. The requests include the file access path and access operation.

[0044] Step 103: If the file access path includes the target field, load the message index of the meteorological scientific data file corresponding to the target field from the time series database, and process the meteorological scientific data file based on the message index and the access operation.

[0045] Specifically, when the file access path includes the target field, since the time series database stores the message index of the meteorological scientific data file, the message index of the meteorological scientific data file corresponding to the target field can be loaded from the time series database without downloading the entire meteorological scientific data file, and the meteorological scientific data file can be processed based on the message index and access operation.

[0046] This invention provides a meteorological science data file processing method based on a time-series database. The method establishes a storage model in the time-series database to store meteorological science data files. It receives a request from an upper-layer application to access a file corresponding to a target directory; the request includes a file access path and an access operation. If the file access path includes a target field, the method loads a message index from the time-series database corresponding to the target field in the meteorological science data file. Based on the message index and the access operation, the method processes the meteorological science data file. By using the file access path, the corresponding meteorological science data file can be retrieved from the time-series database. Furthermore, by obtaining the message index, file content processing can be achieved without downloading the complete meteorological science data file, thus improving file processing efficiency and reducing file processing overhead.

[0047] In some embodiments, when the file access path represents a normal file access path, the standard file corresponding to the normal file access path can be returned from the time series database, and the normal file can be processed based on the access operation.

[0048] Specifically, when the file access path represents a regular file access path, the standard file corresponding to the regular file access path can be directly returned from the time series database, and the standard file can be processed based on the access operation.

[0049] In some embodiments, the method further includes: The metadata and the message index are determined as extended attributes of the meteorological scientific data file; the extended attributes are used to obtain the semantic information of the meteorological scientific data file without fully downloading it.

[0050] Specifically, metadata and message index aggregation information are projected into extended attributes of meteorological science data files; these extended attributes are used to obtain semantic information of meteorological science data files without fully downloading them.

[0051] External programs can use listxattr, getxattr, or equivalent interfaces to view the variables, levels, forecast lead time, spatial extent, grid size, units, number of messages, and format versions supported by the meteorological science data file through extended attributes, without downloading the entire file.

[0052] For files that are not supported or fail to be parsed, the system still retains normal file read and write capabilities, and can mark the file as lacking meteorological format extension capabilities or record the reason for parsing failure in the extended attributes.

[0053] In some embodiments, a metadata catalog table can also be used to provide file-level semantic retrieval by variable, level, region, reporting start time, forecast duration, or data type.

[0054] In some embodiments, the method further includes: Write the binary object, object pointer, message index, and file size of the meteorological scientific data file into the same file version; when the binary object, object pointer, and file size of the meteorological scientific data file are updated to form a new version, the message index is synchronously updated to form a new version.

[0055] Specifically, this application can also construct message indexes for messages, grid fields, or array blocks in meteorological scientific data files. A message index includes at least the variable or feature identifier, hierarchy type, hierarchy value, forecast lead time, member name, spatial extent, grid size, data unit, message byte offset, message byte length, and format version.

[0056] Write the binary object, object pointer, message index, and file size of the meteorological scientific data file into the same file version to ensure consistency in semantic conditions, byte positions, and object content; when the binary object, object pointer, and file size of the meteorological scientific data file are updated to form a new version, the message index can be synchronously updated to form a new version.

[0057] In some embodiments, the file access path further includes the target directory and filename; loading the message index from the meteorological scientific data file corresponding to the target field from the time-series database includes: The file access path is parsed to obtain the target directory, the file name, and the target field; based on the target directory, the file name, and the target field, candidate meteorological scientific data files are determined from the time series database; the message index in the candidate meteorological scientific data files is loaded.

[0058] Specifically, the file access path can also include the target directory and filename. Parsing the file access path yields the target directory, filename, and target fields. Based on the target directory, filename, and target fields, the candidate meteorological scientific data file corresponding to the filename in the target directory can be determined, and the content related to the target fields in the candidate meteorological scientific data file can be accessed. Therefore, candidate meteorological scientific data files can be determined from the time series database; the message index in the candidate meteorological scientific data file is then loaded.

[0059] It is also possible to determine the message content corresponding to the message index based on the message index, and construct a virtual file context by combining the message index, the message content corresponding to the message index, and the file version of the candidate meteorological scientific data file for subsequent reading calls, so as to return the binary data, decoded numerical array, spatial pruning result, or spatial interpolation result corresponding to the target message.

[0060] In some embodiments, processing the meteorological scientific data file based on the message index and the access operation includes: Based on the message index, the target message corresponding to the message index is determined; based on the access operation, the content of the meteorological scientific data file corresponding to the target message is processed.

[0061] Specifically, based on the message index, the target message corresponding to the message index can be determined; based on the access operation, the content of the meteorological scientific data file corresponding to the target message can be processed.

[0062] In some embodiments, processing the meteorological scientific data file content corresponding to the target message based on the access operation includes: When the access operation includes writing or reading a file, it is determined whether the format of the file to be written or read belongs to the meteorological scientific data format; if the format of the file to be written or read belongs to the meteorological scientific data format, the format adapter is called to parse the file to be written or read to obtain the parsing result; based on the parsing result, the content of the meteorological scientific data file corresponding to the target message is processed.

[0063] Specifically, when the access operation includes writing or reading a file, it can be determined whether the format of the file to be written or read belongs to the meteorological scientific data format based on the file extension, file header magic number, GRIB file version identifier, message header structure, or format adapter detection results.

[0064] If the format of the file to be written or read is a meteorological scientific data format, the format adapter can be called to parse the file to be written or read and obtain the parsing result; then, based on the parsing result, the content of the meteorological scientific data file corresponding to the target message can be processed.

[0065] In some embodiments, processing the meteorological scientific data file content corresponding to the target message based on the parsing result includes: When the access operation includes writing to a file, for each message in the file to be written that is included in the parsing result, a message index corresponding to the message is generated, and the metadata corresponding to the message is extracted; according to the write path included in the write request, the file offset in the file to be written, and the length of the metadata, the metadata is written into the meteorological scientific data file content corresponding to the target message in the time series database.

[0066] Specifically, when the access operation includes writing to a file, for each message in the file to be written that is included in the parsing result, a message index corresponding to the message can be generated and the metadata corresponding to the message can be extracted. Based on the write path included in the write request, the file offset in the file to be written, and the length of the metadata, if it is determined that the write path exists and the write path has write permissions, the metadata is written to the meteorological scientific data file content corresponding to the target message in the time series database, that is, written to the binary object or the binary storage area pointed to by the object pointer.

[0067] Metadata can also be placed in a local cache. If a complete message is formed in the local cache, the semantic fields, byte offset, and byte length of the complete message can be extracted and appended to the message index in the file. If the message corresponding to the metadata is the first message or meets the conditions for extracting file-level metadata, the meteorological file-level metadata fields in the metadata catalog table are updated synchronously based on the metadata. When the sealing flag is true, it can be checked whether there are still incomplete messages in the local cache. If there are no incomplete messages, the message index in the file is written back to the object content table and the file version convergence is completed.

[0068] In some embodiments, processing the meteorological scientific data file content corresponding to the target message based on the parsing result includes: When the access operation includes reading a file and the read request is to return raw binary data, for the message in the file to be read that matches the target message, the content of the meteorological scientific data file corresponding to the target message is read according to the file offset and read length included in the read request.

[0069] Specifically, when the access operation includes reading a file and the read request returns raw binary data, for the message in the file to be read that matches the target message, based on the file offset and read length included in the read request, the object field local read interface is called to read from the meteorological scientific data file content corresponding to the target message, that is, only the target message bytes are read.

[0070] In some embodiments, processing the meteorological scientific data file content corresponding to the target message based on the parsing result includes: When the access operation includes reading a file and the read request returns a numerical array, spatial pruning result, or interpolation result, for the message in the file to be read that matches the target message, which is included in the parsing result, the format decoding and spatial processing module is invoked, or the processing related to the read request is converted into a user-defined function in the time series database; the user-defined function is used to complete the reading, decoding, spatial pruning, or interpolation of the meteorological scientific data file content corresponding to the target message in the time series database, to obtain the decoded numerical array, spatial pruning result, or interpolation result.

[0071] Specifically, when the access operation includes reading a file and the read request returns a numerical array, spatial pruning result, or interpolation result, for the message in the file to be read that matches the target message, which is included in the parsing result, the format decoding and spatial processing module can be called, or the processing related to the read request can be converted into a user-defined function in the time series database. The user-defined function then completes the reading, decoding, spatial pruning, or interpolation of the meteorological scientific data file content corresponding to the target message in the time series database, obtaining the decoded numerical array, spatial pruning result, or interpolation result. The decoded numerical array, spatial pruning result, or interpolation result is returned to the file system layer, thereby reducing cross-system data transmission and repeated client-side parsing.

[0072] It should be noted that user-defined functions support runtime registration, display, invocation, and uninstallation. Grid field reading, spatial clipping, spatial interpolation, or other meteorological processing functions can be implemented based on predefined interfaces of the time-series database. During the query execution phase, the corresponding function is selected based on the virtual file parameters, and the target file path, version time, message offset, message length, and processing parameters are passed to the user-defined function. This approach eliminates the need to modify the general file system read / write process when adding meteorological data processing capabilities; it only requires extending the format adapter, message index field, or user-defined function.

[0073] In some embodiments, a file-oriented message index cache, message cache, decoding result cache, and virtual query result cache may also be maintained to reduce redundant parsing, redundant reading, and redundant calculation.

[0074] It can also provide cross-file export tools or software development kits (SDKs) to filter multiple files based on path, variables, hierarchy, time, region or other semantic conditions, batch read target messages and export them as local data files, packaged streams or result sets that can be consumed by upper-layer programs.

[0075] Figure 2 This is a schematic diagram of the system architecture of the meteorological scientific data file processing method based on time-series database provided by the present invention, as shown below. Figure 2 As shown, the system includes client applications, FUSE mount points or file system access points, and a time-series database file system. The client applications include command-line tools, scripts, a graphical file manager, and a Web / SDK client. The time-series database file system includes a file system interface module, a version management module, an object read / write module, an open file context management module, and a metadata management module. The time-series database file system also includes a meteorological format recognition module, a format adapter module, a metadata extraction module, a message-level indexing module, a virtual file query module, a format decoding and space processing module, a user-defined function module, a semantic caching module, a cross-file export module, and a Web or SDK access module.

[0076] A storage model is established in the time-series database within the time-series database file system. The storage model is used to store standard files and meteorological scientific data files. The standard file system layer is responsible for the directory, reading and writing, attributes, and persistence of object content of the original standard files. The meteorological format extension layer is responsible for perceiving the message structure inside meteorological scientific data files such as GRIB / GRIB2, and provides the application layer with semantic viewing, partial reading, online processing, and cross-file retrieval capabilities.

[0077] The time-series database file system receives call requests from command-line tools, scripts, graphical file managers, and Web / SDK clients based on the FUSE mount point or file system access entry point. These requests include file access paths and access operations. When the file access path represents a regular file access path, the system returns the corresponding regular file from the time-series database and processes it based on the access operation. When the file access path includes a target field (i.e., a query string), meaning the access path contains a query string, the system does not require a physical result file. Instead, it loads the message index from the meteorological scientific data file corresponding to the target field from the time-series database and processes the meteorological scientific data file based on the message index and the access operation. Alternatively, based on the message index, the system can determine the message content corresponding to the message index, construct a virtual file context using the message index, the corresponding message content, and the file version of the candidate meteorological scientific data file. This virtual context is used for subsequent read calls to return the binary data corresponding to the target message, the decoded numerical array, the spatial pruning result, or the spatial interpolation result.

[0078] Figure 3 This is a schematic diagram of the meteorological metadata fusion and tree-table dual model provided by the present invention, as shown below. Figure 3 As shown, the metadata of meteorological scientific data files in the metadata directory table can be represented using a directory tree structure. The system maintains the user-visible file paths and directory hierarchy at the directory tree level. At the time-series database table structure level, the metadata directory table (inode) and object content table (blob) are stored. The metadata directory table includes standard file attributes and meteorological file-level metadata. The metadata directory table (inode) uses the file's first write timestamp and absolute logical path as a composite positioning dimension. The metadata directory table stores the attribute information of standard files and the attribute information and metadata of meteorological scientific data files. The attribute information of standard files includes at least one of the following: logical path, parent directory, file name, file size, permissions, user, group, creation time, modification time, and access time. The attribute information of meteorological scientific data files includes at least one of the following: logical path, parent directory, file name, file size, permissions, user, group, creation time, modification time, and access time. Metadata for meteorological scientific data files includes at least one of the following: data type, data type, member name, region, element, element unit, hierarchy type, hierarchy value, starting longitude, ending longitude, starting latitude, ending latitude, latitude / longitude step, number of longitude grid points, number of latitude grid points, forecast lead time, forecast lead time unit, grid precision, altitude / pressure unit, resolution, and start time. The object content table stores the binary objects, object pointers, and file size of standard files, as well as the binary objects, object pointers, message index, and file size of meteorological scientific data files.

[0079] The directory tree layer and the time-series database table structure layer together constitute a dual access point for meteorological documents. On the one hand, for access point one: users can access standard documents through the file system path (i.e., the file access path) as ordinary file paths; on the other hand, for access point two: the system performs retrieval based on metadata conditions, with the combined fields (i.e., target fields) being fields in the metadata directory table, such as data, data type, region, feature, hierarchical type, and start time. Candidate documents are quickly located from the time-series database, and the corresponding message index within the document is loaded through the object content table blob. Based on the message index, the target message corresponding to the message index is determined, realizing cross-file spatiotemporal penetration query.

[0080] Figure 4 This is a flowchart of the meteorological data format recognition, streaming metadata extraction, and message-level index construction process provided by the present invention, such as... Figure 4 As shown, when access operations include writing to a file, reading from a file, or closing a file, file format recognition is triggered. The system determines whether the files belong to a supported meteorological scientific data format based on the file extension, file header magic number, GRIB version field, or adapter detection results. If the file does not belong to a supported meteorological scientific data format, the system processes it as a normal file. If the file belongs to a supported meteorological scientific data format, the system calls the corresponding format adapter to parse the files, identifying each message based on message boundary identifiers to obtain the parsing results. For each message in these files included in the parsing results, a message index is generated, and the corresponding metadata is extracted, including file-level metadata such as data type, region, feature, unit, hierarchical type, grid range, resolution, and reporting start time. An in-file message index is generated synchronously during the writing, reading, or closing process. The message index serves as a message semantic field and can include byte offset and byte length. It should be noted that the metadata corresponding to a message can be extracted from the first or representative message.

[0081] In one implementation, the file write function accepts parameters such as the target write path, binary data (the file to be written), the offset of the binary data, the length of the binary data, and whether it is sealed. First, it checks if the target write path exists and if write permissions are available. If the target write path exists and write permissions are available, the binary data is written to the binary storage area pointed to by the object field or the object pointer.

[0082] Binary data can also be placed in a local cache. If a complete message is formed in the local cache, the semantic fields, byte offset, and byte length of the complete message can be extracted and appended to the message index in the file. If the message corresponding to the metadata is the first message or meets the conditions for extracting file-level metadata, the meteorological file-level metadata fields in the metadata catalog table are updated synchronously based on the metadata. When the sealing flag is true, it can be checked whether there are still incomplete messages in the local cache. If there are no incomplete messages, the message index in the file is written back to the object content table and the file version convergence is completed.

[0083] Figure 5 This is a flowchart of the extended filename virtual file query, object offset reading, and calculation pushdown process provided by the present invention, such as... Figure 5 As shown, the system is compatible with traditional file system interfaces by extending filenames. Users can append query strings (i.e., target fields) to the actual file path. For example, `show.grib2?level=500` can read a grid field with a height value of 500, `show.grib2?level=500&type=data&dtype=f64` can obtain the decoded value, `show.grib2?level=500&lat0=30&lon0=100&lat1=40&lon1=120&dtype=f64` can obtain the clipping result for a specified latitude and longitude range, or `show.grib2?level=500&interp=bilinear&dlon=0.25&dlat=0.25&dtype=f64` can obtain the target resolution result after bilinear interpolation.

[0084] When the system opens an access path that includes a query string in a call request, it first parses the file path, function type, and parameter key-value pairs based on the access path including the query string. Then, it loads the message index within the file based on the file path and version time, and determines the target message that matches the message index based on parameters such as level, variable name, hierarchy type, forecast time, and member name.

[0085] If the call request includes reading a file and the read request returns raw binary data, then for the message that matches the target message, the system calls the object field local read interface according to the file offset and read length to read from the meteorological science data file content corresponding to the target message, that is, only reads the target message bytes.

[0086] If the call request includes reading a file and the request returns a numerical array, spatial pruning result, or interpolation result, the system calls the format decoding and spatial processing module, or converts the processing related to the read request into a user-defined function in the time series database. After the user-defined function completes the reading, decoding, pruning, or interpolation of the meteorological scientific data file content corresponding to the target message in the time series database, only the decoded numerical array, spatial pruning result, or interpolation result is returned to the file system layer. This enables extended filename virtual file lookup, object offset reading and calculation pushdown, reduces cross-system data transmission and repeated client parsing, and supports on-demand expansion of processing capabilities.

[0087] In one implementation, user-defined functions support runtime registration, display, invocation, and unloading. Grid field reading, spatial clipping, spatial interpolation, or other meteorological processing functions can be implemented based on predefined interfaces of the time-series database. During the query execution phase, the corresponding function is selected based on the virtual file parameters, and the target file path, version time, message offset, message length, and processing parameters are passed to the user-defined function. This approach eliminates the need to modify the general file system read / write process when adding meteorological data processing capabilities; it only requires extending the format adapter, message index field, or user-defined function.

[0088] Figure 6 This is a flowchart of the extended attribute semantic retrieval, semantic caching, and cross-file export process provided by the present invention, such as... Figure 6 As shown, the system projects aggregated information from meteorological file-level metadata and message indexes into extended attributes. External programs can view the variables, levels, forecast lead times, spatial extent, grid size, units, number of messages, and format versions supported by the file through these extended attributes. For files that are not supported or fail to be parsed, the system still retains normal file read and write capabilities and can mark the file as lacking meteorological format extension capabilities or record the reason for parsing failure in the extended attributes.

[0089] The system maintains multiple semantic caches oriented towards file versions. These caches include message index cache, message cache, decoding result cache, and virtual query result cache to reduce redundant parsing, reading, and computation. Specifically, the message index cache caches the message index for a specified file version, avoiding repeated parsing of index fields; the message cache caches raw byte fragments of frequently accessed messages, reducing repeated offset readings of object fields; the decoding result cache caches numeric arrays for specified messages, output types, and data types; and the virtual query result cache caches the final result containing the clipped region, interpolation parameters, and output type.

[0090] It should be noted that the above caching uses logical path, version time, message semantic conditions, and query parameters as key dimensions to ensure that there is no confusion between different file versions and different query results.

[0091] When accessing across files, the system first filters candidate meteorological scientific data files from the time-series database based on the metadata catalog table, according to query fields such as data type, region, element, hierarchical type, start time, forecast lead time, or spatial range. For each candidate meteorological scientific data file, it loads the corresponding version of the file's message index and determines the target message corresponding to the message index. Finally, it performs batch numerical array, spatial pruning, or spatial interpolation on the files corresponding to the target messages, calling the format decoding and spatial processing module, or converting the related processing into user-defined functions in the time-series database. The user-defined functions then complete the reading, decoding, spatial pruning, or interpolation of the meteorological scientific data file content corresponding to the target message in the time-series database, obtaining the decoded numerical array, spatial pruning result, or interpolation result. The decoded numerical array, spatial pruning result, or interpolation result is then exported as a local data file, a packaged stream, or an SDK result set. Through this process, users can achieve cross-file spatiotemporal penetration access without traversing each directory and parsing each file individually.

[0092] This invention unifies file paths, file-level meteorological metadata, file-internal message indexes, and object content into a time-series database file system through a directory tree layer and a time-series database table structure layer. It reduces the cost of local reads after file writing through streaming parsing and message index construction. Extended attributes and filenames enable ordinary file system interfaces to perform meteorological semantic viewing and virtual file querying. Object offset reading and user-defined function calculation pushdown reduce the need for full file downloads, repeated scanning, and repeated decoding. Semantic caching and cross-file export improve the efficiency of continuous access to massive amounts of meteorological grid data. Compared to file systems that only store GRIB files as ordinary binary objects, this invention significantly improves the semantic visibility, local read efficiency, and multi-file analysis availability of meteorological scientific data files.

[0093] The meteorological scientific data file processing device based on a time-series database provided by the present invention will be described below. The meteorological scientific data file processing device based on a time-series database described below and the meteorological scientific data file processing method based on a time-series database described above can be referred to and correspond to each other.

[0094] Figure 7 This is a schematic diagram of the meteorological scientific data file processing device based on a time-series database provided by the present invention, as shown below. Figure 7 As shown, the meteorological science data file processing device 700 based on a time-series database includes: a creation module 701, a receiving module 702, and a first processing module 703; wherein, A module 701 is established to create a storage model in a time-series database, the storage model being used to store meteorological science data files; The receiving module 702 is used to receive a call request from an upper-layer application for a file corresponding to a target directory, wherein the call request includes a file access path and an access operation; The first processing module 703 is configured to, when the file access path includes a target field, load the message index of the meteorological scientific data file corresponding to the target field from the time series database, and process the meteorological scientific data file based on the message index and the access operation.

[0095] The meteorological science data file processing device based on a time-series database provided by this invention establishes a storage model in the time-series database to store meteorological science data files. It receives a request from an upper-layer application to access a file corresponding to a target directory; the request includes a file access path and an access operation. If the file access path includes a target field, it loads a message index from the meteorological science data file corresponding to the target field from the time-series database, and processes the meteorological science data file based on the message index and the access operation. By using the file access path, the corresponding meteorological science data file can be obtained from the time-series database. Furthermore, by obtaining the message index, file content processing can be achieved without downloading the complete meteorological science data file, thus improving file processing efficiency and reducing file processing overhead.

[0096] In some embodiments, the file access path further includes the target directory and filename; the first processing module 703 is specifically used for: The file access path is parsed to obtain the target directory, the file name, and the target field; Based on the target directory, the file name, and the target field, candidate meteorological science data files are determined from the time-series database; Load the message index from the candidate meteorological science data file.

[0097] In some embodiments, the first processing module 703 is further configured to: Based on the message index, determine the target message corresponding to the message index; Based on the access operation, the content of the meteorological scientific data file corresponding to the target message is processed.

[0098] In some embodiments, the first processing module 703 is further configured to: In cases where the access operation includes writing or reading a file, determine whether the format of the file to be written or read belongs to the meteorological scientific data format; If the format of the file to be written or the file to be read is a meteorological scientific data format, the format adapter is called to parse the file to be written or the file to be read, and the parsing result is obtained; Based on the analysis results, the content of the meteorological scientific data file corresponding to the target message is processed.

[0099] In some embodiments, the first processing module 703 is further configured to: If the access operation includes writing to a file, for each message in the file to be written that is included in the parsing result, a message index corresponding to the message is generated, and the metadata corresponding to the message is extracted; Based on the write path included in the write request, the file offset in the file to be written, and the length of the metadata, the metadata is written into the meteorological scientific data file content corresponding to the target message in the time series database.

[0100] In some embodiments, the first processing module 703 is further configured to: When the access operation includes reading a file and the read request is to return raw binary data, for the message in the file to be read that matches the target message, the content of the meteorological scientific data file corresponding to the target message is read according to the file offset and read length included in the read request.

[0101] In some embodiments, the first processing module 703 is further configured to: When the access operation includes reading a file and the read request returns a numerical array, spatial clipping result, or interpolation result, for the message in the file to be read that matches the target message, which is included in the parsing result, the format decoding and spatial processing module is invoked, or the processing related to the read request is converted into a user-defined function in the time series database; The user-defined function is used to read, decode, spatially prune, or interpolate the meteorological scientific data file content corresponding to the target message in the time series database, so as to obtain the decoded numerical array, spatial pruning result, or interpolation result.

[0102] In some embodiments, the storage model is also used to store metadata and message indexes of the meteorological science data files.

[0103] In some embodiments, the meteorological science data file processing device 700 based on a time-series database further includes: A determination module is used to determine the metadata and the message index as extended attributes of the meteorological scientific data file; the extended attributes are used to obtain the semantic information of the meteorological scientific data file without fully downloading the meteorological scientific data file.

[0104] In some embodiments, the storage model includes a metadata directory table and an object content table; wherein the metadata directory table is used to store the attribute information and metadata of the meteorological scientific data file, and the object content table is used to store the binary objects, object pointers, message indexes, and file size of the meteorological scientific data file.

[0105] In some embodiments, the meteorological science data file processing device 700 based on a time-series database further includes: The second processing module is used to write the binary object, the object pointer, the message index, and the file size of the meteorological scientific data file into the same file version; The third processing module is used to synchronize the message index to form a new version when the binary object, the object pointer, and the file size of the meteorological scientific data file are updated to form a new version.

[0106] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a meteorological science data file processing method based on a time-series database. This method includes: establishing a storage model in the time-series database for storing meteorological science data files; receiving a request from an upper-layer application to access a file corresponding to a target directory, the request including a file access path and access operation; if the file access path includes a target field, loading a message index from the time-series database corresponding to the target field in the meteorological science data file, and processing the meteorological science data file based on the message index and the access operation.

[0107] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0108] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the meteorological science data file processing method based on a time-series database provided by the above methods. The method includes: establishing a storage model in the time-series database for storing meteorological science data files; receiving a call request from an upper-layer application for a file corresponding to a target directory, the call request including a file access path and an access operation; and, if the file access path includes a target field, loading a message index from the meteorological science data file corresponding to the target field from the time-series database, and processing the meteorological science data file based on the message index and the access operation.

[0109] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the meteorological science data file processing method based on a time-series database provided by the above methods. The method includes: establishing a storage model in the time-series database for storing meteorological science data files; receiving a request from an upper-layer application to access a file corresponding to a target directory, the request including a file access path and an access operation; and, if the file access path includes a target field, loading a message index from the meteorological science data file corresponding to the target field from the time-series database, and processing the meteorological science data file based on the message index and the access operation.

[0110] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0112] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for processing meteorological scientific data files based on a time-series database, characterized in that, include: A storage model is established in the time-series database, which is used to store meteorological science data files; Receive a request from an upper-layer application to access a file in a target directory, the request including the file access path and access operation; If the file access path includes a target field, the message index of the meteorological scientific data file corresponding to the target field is loaded from the time series database, and the meteorological scientific data file is processed based on the message index and the access operation.

2. The meteorological scientific data file processing method based on time-series database according to claim 1, characterized in that, The file access path also includes the target directory and file name; The step of loading the message index from the meteorological scientific data file corresponding to the target field from the time-series database includes: The file access path is parsed to obtain the target directory, the file name, and the target field; Based on the target directory, the file name, and the target field, candidate meteorological science data files are determined from the time-series database; Load the message index from the candidate meteorological science data file.

3. The meteorological scientific data file processing method based on time-series database according to claim 1, characterized in that, The processing of the meteorological scientific data file based on the message index and the access operation includes: Based on the message index, determine the target message corresponding to the message index; Based on the access operation, the content of the meteorological scientific data file corresponding to the target message is processed.

4. The meteorological scientific data file processing method based on time-series database according to claim 3, characterized in that, The processing of the meteorological scientific data file content corresponding to the target message based on the access operation includes: In cases where the access operation includes writing or reading a file, determine whether the format of the file to be written or read belongs to the meteorological scientific data format; If the format of the file to be written or the file to be read is a meteorological scientific data format, the format adapter is called to parse the file to be written or the file to be read, and the parsing result is obtained; Based on the analysis results, the content of the meteorological scientific data file corresponding to the target message is processed.

5. The meteorological scientific data file processing method based on time-series database according to claim 4, characterized in that, The process of processing the meteorological scientific data file content corresponding to the target message based on the parsing results includes: If the access operation includes writing to a file, for each message in the file to be written that is included in the parsing result, a message index corresponding to the message is generated, and the metadata corresponding to the message is extracted; Based on the write path included in the write request, the file offset in the file to be written, and the length of the metadata, the metadata is written into the meteorological scientific data file content corresponding to the target message in the time series database.

6. The meteorological scientific data file processing method based on time-series database according to claim 4, characterized in that, The process of processing the meteorological scientific data file content corresponding to the target message based on the parsing results includes: When the access operation includes reading a file and the read request is to return raw binary data, for the message in the file to be read that matches the target message, the content of the meteorological scientific data file corresponding to the target message is read according to the file offset and read length included in the read request.

7. The meteorological scientific data file processing method based on a time-series database according to claim 4, characterized in that, The process of processing the meteorological scientific data file content corresponding to the target message based on the parsing results includes: When the access operation includes reading a file and the read request returns a numerical array, spatial clipping result, or interpolation result, for the message in the file to be read that matches the target message, which is included in the parsing result, the format decoding and spatial processing module is invoked, or the processing related to the read request is converted into a user-defined function in the time series database; The user-defined function is used to read, decode, spatially prune, or interpolate the meteorological scientific data file content corresponding to the target message in the time series database, so as to obtain the decoded numerical array, spatial pruning result, or interpolation result.

8. The meteorological scientific data file processing method based on time-series database according to claim 1, characterized in that, The storage model is also used to store the metadata and message index of the meteorological scientific data files.

9. The meteorological scientific data file processing method based on a time-series database according to claim 8, characterized in that, The method further includes: The metadata and the message index are determined as extended attributes of the meteorological scientific data file; the extended attributes are used to obtain the semantic information of the meteorological scientific data file without fully downloading it.

10. The meteorological scientific data file processing method based on a time-series database according to claim 1, characterized in that, The storage model includes a metadata directory table and an object content table; wherein, the metadata directory table is used to store the attribute information and metadata of the meteorological scientific data file, and the object content table is used to store the binary objects, object pointers, message indexes and file size of the meteorological scientific data file.

11. The meteorological scientific data file processing method based on a time-series database according to claim 10, characterized in that, The method further includes: Write the binary object, the object pointer, the message index, and the file size of the meteorological science data file into the same file version; When the binary object, the object pointer, and the file size of the meteorological science data file are updated to form a new version, the message index is synchronized to form a new version.

12. A meteorological scientific data file processing device based on a time-series database, characterized in that, include: A module is established to create a storage model in the time-series database, the storage model being used to store meteorological science data files; The receiving module is used to receive a call request from the upper-layer application for a file corresponding to the target directory. The call request includes the file access path and access operation. The first processing module is configured to, when the file access path includes a target field, load the message index of the meteorological scientific data file corresponding to the target field from the time series database, and process the meteorological scientific data file based on the message index and the access operation.

Images