A variable data caching method and system for NetCDF files

Abstract

The present application belongs to the field of computer optimization, and specifically discloses a variable data caching method and system for NetCDF files. It includes: starting from the root group of the NetCDF file, recursively obtaining data and metadata information therein step by step, automatically generating data indexes, data blocks, and metadata to build data boxes. If there are multiple files, each file corresponds to a data box, and these multiple data boxes are organized into a data warehouse. The present application pre-loads data blocks into data boxes in memory to build a fast index structure to achieve fast access to large-scale data sets, meet the requirements of artificial intelligence algorithms for data real-time and responsiveness; by reducing the number of repeated file reads, improve data utilization and computing resource utilization efficiency; by optimizing data access and management processes, reduce the demand for computing resources. Compared with reading data directly from files, data caching and indexing technology can reduce data access time and computing load.

Description

Technical Field

[0001] The present application belongs to the field of computer optimization, and more specifically, to a variable data caching method and system for NetCDF files. Background Art

[0002] NetCDF (Network Common Data Form) is a self-describing, machine-independent data format for storing scientific data. It supports the creation, access, and sharing of array-oriented scientific data. NetCDF is widely used in fields such as meteorology, earth science, and environmental science to store various types of multidimensional array data. As the scale of scientific data grows, the efficiency of reading data directly from NetCDF files has gradually become a bottleneck, especially in application scenarios that require frequent concurrent access to large amounts of data.

[0003] NetCDF stores and retrieves data in the format of arrays. The classic NetCDF data model contains variables, dimensions, and attributes, and is the core of the NetCDF file. After NetCDF-4, the classic data model was enhanced and some new features were introduced, such as groups, unlimited multi-dimensionality, and new data types (including user-defined types). Groups are a way to organize data, similar to directories in Unix file systems. NetCDF files have the following characteristics: (1) Users can define hierarchical groups and add variables, dimensions, and attributes to these groups; (2) Each file starts with at least one root group, and users can add more groups and obtain a new ncid for each group created; (3) Since each group can be a complete NetCDF classic dataset, it is possible to have variables with the same name in two or more different groups in the same NetCDF-4 / HDF5 file; (4) Dimensions have special scopes: they can be seen by all variables in the group and all subgroups.

[0004] The existing NetCDF file data reading mainly has the following problems: (1) Low data access efficiency: The efficiency of reading data directly from NetCDF files is low, especially for large-scale data sets or frequent access situations; (2) Difficult data retrieval: NetCDF files usually contain a large number of variables and dimensions, making it complicated and time-consuming for users to find and retrieve data in the file; (3) Inconvenient data management: For data that needs to be accessed and processed multiple times, there is a lack of effective caching and management mechanisms, resulting in frequent repeated reading and processing of data. In addition, with the development of the Internet and the popularization of various sensor technologies, the amount of data is growing exponentially, and the complexity of the data is also increasing. In the era of artificial intelligence, tasks such as pattern recognition, prediction, and classification for these large-scale, diverse, and complex data place high demands on the data access efficiency, resource utilization, and real-time performance of computing. Summary of the invention

[0005] In view of the defects of the prior art, the purpose of the present application is to provide a variable data caching method and system for NetCDF files, aiming to solve the problems of low data access efficiency, difficult data retrieval and inconvenient data management in the existing NetCDF file reading.

[0006] To achieve the above objectives, in a first aspect, the present application provides a variable data caching method for a NetCDF file, comprising:

[0007] S0. Get all NetCDF files that need to be cached in this round of calculation;

[0008] S1. Build a data box for all data from the same NetCDF file, the data box includes metadata, data blocks, data index values ​​and hash tables, as follows:

[0009] S11. Starting from the root group of the NetCDF file, recursively traverse each group step by step, read data and corresponding metadata from each group and cache them into memory;

[0010] S12. The data read from all groups are uniformly divided into data blocks according to the variable names, and the data with the same variable name are divided into the same data block;

[0011] S13. For each data block, construct a corresponding data index value according to its data type name and variable name;

[0012] S14. The starting address of the data block in the memory is used as the Value, and the corresponding data index value is used as the Key, and stored in the hash table in the form of a key-value pair;

[0013] S2. Organize all data boxes into a data warehouse according to the NetCDF file name / data box ID.

[0014] Preferably, step S11 is specifically as follows:

[0015] S111. Convert the multidimensional data in the NetCDF file into one-dimensional data. The specific steps are as follows:

[0016] (1) Initialize variables: Read the row and column values ​​of the multidimensional array in the NetCDF file and define two variables, index and multiplier. Index is used to store the final one-dimensional index and is initialized to 0. Multiplier is used for auxiliary calculation and is initialized to 1.

[0017] (2) Loop iteration: A for loop is used to traverse each dimension of a multidimensional array, starting from the highest dimension and gradually decreasing. Each loop iteration visits each dimension of the multidimensional array.

[0018] (3) Index calculation: In each loop iteration, the index value of the current dimension is obtained, multiplied by the multiplier and added to the index to convert the index of the multidimensional array into the index of the one-dimensional array, and the accumulation is carried out in the order from low dimension to high dimension;

[0019] (4) Update the multiplication factor: At each iteration, update the multiplication factor multiplier according to the size of the current dimension;

[0020] (5) Obtaining the result: After the loop ends, what is stored in index is the index value of the corresponding one-dimensional array calculated based on the index of the multidimensional array;

[0021] S112. Create a ChunkData structure, the member variables of which include: the number of data dimensions rank, the array dimensions storing the length of each dimension, the length of the one-dimensional array length, and the pointer data pointing to the starting address of the data memory. The specific process is as follows:

[0022] First, the byte size of the memory space to be allocated is calculated as length multiplied by the byte size of the data type. Then, based on the malloc function, the memory space is allocated to obtain the starting address data. Finally, the variable data in the NetCDF file is read into data.

[0023] Preferably, in step S13, the data index value consists of a first part and a second part;

[0024] The first part represents the data type name;

[0025] The second part is determined as follows:

[0026] For a float variable, the second part is its subscript index in the variable name array;

[0027] For a variable of type int, the second part is its subscript index in the variable name array multiplied by 10;

[0028] For a variable of type double, the second part is its subscript index in the variable name array multiplied by 100;

[0029] For variables of type short, the second part is its subscript index in the variable name array multiplied by 1000.

[0030] Preferably, in step S14, a hash table is constructed separately for each data type of data block, and the hash functions used by the four hash tables are not required to be the same.

[0031] Preferably, when a certain data block needs to be searched, the data type of the required data block and the corresponding variable name are specified, and the two are concatenated to obtain a data index value.

[0032] Preferably, when it is necessary to find a starting address corresponding to a data index value, the hash code of the data index value is first calculated, and then the starting address of the corresponding data block is found in the hash table.

[0033] Preferably, when you need to find a data box:

[0034] 1) Obtain the filtering conditions related to metadata, traverse each data box, and the data box whose metadata matches the filtering conditions is the target data box;

[0035] 2) Obtain the designation of the file name or the ID value of the data box, and the data box that matches the file name or the ID value of the data box is the target data box.

[0036] Preferably, each time a new computing task is executed, a new round of data caching is dynamically performed.

[0037] Preferably, the variable data caching method is applied to application scenarios of frequent concurrent access to variable data in NetCDF files or access to large-scale NetCDF file data sets.

[0038] To achieve the above objectives, in a second aspect, the present application provides a variable data caching system for NetCDF files, comprising at least one processor and at least one memory; the at least one memory is used to store computer instructions; the at least one processor is used to execute at least part of the computer instructions to implement the variable data caching method described in the first aspect.

[0039] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0040] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art:

[0041] This application uses the characteristics of NetCDF files to propose a variable data caching method and system for NetCDF files. Starting from the root group of the NetCDF file, the data and metadata information therein are recursively obtained step by step, and then the data index, data block, and metadata information structure are automatically generated to build a data box. If there are multiple files, multiple data boxes will be organized into a data warehouse. After completing the pre-fetching cache of the data, the user can enter the variable name and other screening conditions (such as temperature, longitude and latitude, etc.) according to the needs, and the method will return the data block of the specified data. Through the above method, this application brings the following changes: (1) Improvement of data access efficiency: By pre-loading data blocks into the data box in the memory and building a fast index structure, it is possible to achieve fast access to large-scale data sets and meet the requirements of artificial intelligence algorithms for data real-time and responsiveness; (2) Improvement of resource utilization: By reducing the number of repeated file readings, the utilization rate of data and the efficiency of computing resources are improved; (3) Reduction of computing power requirements: By optimizing data access and management processes, the demand for computing resources can be reduced. Compared with reading data directly from files, data caching and indexing technologies can reduce the time and computing load of data access, thereby reducing the computing power cost required by artificial intelligence algorithms when processing large-scale data; (4) Promotion of data science research: It improves the access efficiency and management capabilities of scientific data, which helps promote the progress of data science research. Scientists and researchers can more easily access and analyze large-scale scientific data, accelerating data-driven scientific discovery and innovation. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart of a variable data caching method for NetCDF files provided in an embodiment of the present application.

[0043] Figure 2 It is a schematic diagram of the data warehouse and data box structure provided in the embodiment of the present application.

[0044] Figure 3 This is the test result of the data warehouse and data box provided in the embodiment of the present application.

[0045] Figure 4 It is the data index test result provided by the embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] The term "and / or" in this article is a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The symbol " / " in this article indicates that the associated objects are in an or relationship, for example, A / B means A or B.

[0048] The terms "first" and "second" in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects. For example, a first response message and a second response message are used to distinguish different response messages rather than to describe a specific order of the response messages.

[0049] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0050] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more than two. For example, multiple processing units refer to two or more processing units, etc.; multiple elements refer to two or more elements, etc.

[0051] Next, the technical solutions provided in the embodiments of the present application are introduced.

[0052] like Figure 1 As shown, the present application provides a variable data caching method for a NetCDF file, comprising:

[0053] S0. Get all NetCDF files that need to be cached for this round of calculation.

[0054] S1. Build a data box for all data from the same NetCDF file, the data box includes metadata, data blocks, data index values ​​and hash tables, as follows:

[0055] The metadata may be data such as dimensions of a multidimensional array, variable names, timestamps, etc.

[0056] The data box also includes a file name and the number of the data box in the data warehouse.

[0057] S11. Starting from the root group of the NetCDF file, recursively traverse each group step by step, read data and corresponding metadata from each group and cache them into memory.

[0058] Get the row and column values ​​of the multidimensional array in the NetCDF file, then calculate the number of data elements in it, and finally load the data in the NetCDF file stored on the disk into the data box in memory in advance in blocks. In this way, when users need to access specific data, they can directly obtain it from the data box without having to read the data from the file every time, which greatly improves the efficiency of data access.

[0059] Preferably, step S11 is specifically as follows:

[0060] S111. Convert the multidimensional data in the NetCDF file into one-dimensional data. The specific steps are as follows:

[0061] (1) Initialize variables: Read the row and column values ​​of the multidimensional array in the NetCDF file and define two variables, index and multiplier. Index is used to store the final one-dimensional index and is initialized to 0. Multiplier is used for auxiliary calculation and is initialized to 1.

[0062] (2) Loop iteration: A for loop is used to traverse each dimension of the multidimensional array, starting from the highest dimension and gradually decreasing. Each loop iteration visits each dimension of the multidimensional array.

[0063] (3) Index calculation: In each loop iteration, the index value of the current dimension is obtained, multiplied by the multiplier and added to the index to convert the index of the multidimensional array into the index of the one-dimensional array, and the accumulation is carried out in the order from low dimension to high dimension;

[0064] (4) Update the multiplication factor: At each iteration, update the multiplication factor multiplier according to the size of the current dimension;

[0065] (5) Obtaining the result: After the loop ends, what is stored in index is the index value of the corresponding one-dimensional array calculated based on the index of the multidimensional array;

[0066] S112. Create a ChunkData structure, the member variables of which include: the number of data dimensions rank, the array dimensions storing the length of each dimension, the length of the one-dimensional array length, and the pointer data pointing to the starting address of the data memory. The specific process is as follows:

[0067] First, the byte size of the memory space to be allocated is calculated as length multiplied by the byte size of the data type. Then, based on the malloc function, the memory space is allocated to obtain the starting address data. Finally, the variable data in the NetCDF file is read into data.

[0068] S12. The data read from all groups are uniformly divided into data blocks according to the variable names, and the data with the same variable name are divided into the same data block.

[0069] This application is based on the idea of ​​data segmentation, pre-fetching data and storing it in a data box. When the user needs data, the data block is directly obtained from the data box instead of reading it from the NetCDF file.

[0070] S13. For each data block, construct a corresponding data index value according to its data type name and variable name.

[0071] Preferably, in step S13, the data index value consists of a first part and a second part;

[0072] The first part represents the data type name;

[0073] The second part is determined as follows:

[0074] For a float variable, the second part is its subscript index in the variable name array;

[0075] For a variable of type int, the second part is its subscript index in the variable name array multiplied by 10;

[0076] For a variable of type double, the second part is its subscript index in the variable name array multiplied by 100;

[0077] For variables of type short, the second part is its subscript index in the variable name array multiplied by 1000.

[0078] Users can obtain the data index value through the variable name, and then quickly locate the required data block based on the data index value, thereby achieving fast retrieval.

[0079] In this embodiment, the first letter indicates the data type of the variable data, followed by the second part determined by the variable name. The data index is also stored in a unified array to facilitate retrieval and query.

[0080] S14. The starting address of the data block in the memory is used as the Value, and the corresponding data index value is used as the Key, and stored in the hash table in the form of a key-value pair.

[0081] The mapping relationship between data index and data block is constructed by hash table.

[0082] Preferably, in step S14, a hash table is constructed separately for each data type of data block, and the hash functions used by the four hash tables are not required to be the same.

[0083] S2. Organize all data boxes into a data warehouse according to the NetCDF file name / data box ID.

[0084] like Figure 2 As shown, the constructed data warehouse consists of multiple data boxes. Each data box mainly consists of four parts: metadata Metadata, data index value DataIndex, data block DataChunk and hash table. The metadata is used to filter data boxes according to specified conditions; the data index value is in the form of a string, and each data index will be mapped to the starting address of a data block in the memory; the data index value and the data block starting address are used as key-value pairs, and then a hash table is created, specifically: a hash function is created, the hash code of each data index value is calculated, and then a hash table is created to map the data index value to the data block start, and finally the key-value pair consisting of the data index value and the data block starting address is inserted into the hash table.

[0085] Preferably, when a certain data block needs to be searched, the data type of the required data block and the corresponding variable name are specified, and the two are concatenated to obtain a data index value.

[0086] Preferably, when it is necessary to find a starting address corresponding to a data index value, the hash code of the data index value is first calculated, and then the starting address of the corresponding data block is found in the hash table.

[0087] Preferably, when you need to find a data box:

[0088] 1) Obtain metadata-related filtering conditions, traverse each data box, and the data box whose metadata matches the filtering conditions is the target data box; in meteorological scenarios, the filtering conditions can be temperature, longitude and latitude, etc.

[0089] 2) Obtain the designation of the file name or the ID value of the data box, and the data box that matches the file name or the ID value of the data box is the target data box.

[0090] Preferably, each time a new computing task is executed, a new round of data caching is dynamically performed.

[0091] Preferably, the variable data caching method is applied to application scenarios of frequent concurrent access to variable data in NetCDF files or access to large-scale NetCDF file data sets, such as meteorology, earth science, etc.

[0092] Example

[0093] The feasibility of this method was verified by automatically building data indexes, metadata information, and data indexes for multiple NetCDF files required for the WRF-Hydro model. Figure 3 and Figure 4 .

[0094] It can be understood that the detailed functional implementation of each of the above-mentioned units / modules can be found in the introduction of the aforementioned method embodiment, and will not be repeated here.

[0095] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0096] Based on the method in the above embodiment, an embodiment of the present application provides an electronic device, which may include: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The processor may call the logic instructions in the memory to execute the method in the above embodiment.

[0097] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0098] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0099] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0100] It is understandable that the processor in the embodiment of the present application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor can be a microprocessor or any conventional processor.

[0101] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0102] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0103] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0104] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A variable data caching method for NetCDF files, characterized in that: include: S0. Get all NetCDF files that need to be cached in this round of calculation; S1. Build a data box for all data from the same NetCDF file, the data box includes metadata, data blocks, data index values ​​and hash tables, as follows: S11. Starting from the root group of the NetCDF file, recursively traverse each group step by step, read data and corresponding metadata from each group and cache them into memory; S12. The data read from all groups are uniformly divided into data blocks according to the variable names, and the data with the same variable name are divided into the same data block; S13. For each data block, construct a corresponding data index value according to its data type name and variable name; S14. The starting address of the data block in the memory is used as the Value, and the corresponding data index value is used as the Key, and stored in the hash table in the form of a key-value pair; S2. Organize all data boxes into a data warehouse according to NetCDF file names or data box IDs; Step S11 is specifically as follows: S111. Convert the multidimensional data in the NetCDF file into one-dimensional data. The specific steps are as follows: (1) Initialize variables: Read the row and column values ​​of the multidimensional array in the NetCDF file and define two variables, index and multiplier. Index is used to store the final one-dimensional index and is initialized to 0. Multiplier is used for auxiliary calculations and is initialized to 1. (2) Loop iteration: Use a for loop to traverse each dimension of the multidimensional array, starting from the highest dimension and gradually decreasing. Each loop iteration visits each dimension of the multidimensional array; (3) Index calculation: In each loop iteration, the index value of the current dimension is obtained, multiplied by the multiplier and added to the index to convert the index of the multidimensional array into the index of the one-dimensional array, and the accumulation is carried out in the order from low dimension to high dimension. (4) Update the multiplication factor: At each iteration, update the multiplication factor multiplier according to the size of the current dimension; (5) Get the result: After the loop ends, the index stored in index is the index value of the corresponding one-dimensional array calculated based on the index of the multidimensional array; S112. Create a ChunkData structure, the member variables of which include: the number of data dimensions rank, the array dimensions storing the length of each dimension, the length of the one-dimensional array length, and the pointer data pointing to the starting address of the data memory. The specific process is as follows: First, the byte size of the memory space to be allocated is calculated as length multiplied by the byte size of the data type. Then, based on the malloc function, the memory space is allocated to obtain the starting address data. Finally, the variable data in the NetCDF file is read into data.

2. The variable data caching method according to claim 1, characterized in that: In step S13, the data index value consists of a first part and a second part; The first part represents the data type name; The second part is determined as follows: For a float variable, the second part is its subscript index in the variable name array; For a variable of type int, the second part is its subscript index in the variable name array multiplied by 10; For a variable of type double, the second part is its subscript index in the variable name array multiplied by 100; For variables of type short, the second part is its subscript index in the variable name array multiplied by 1000.

3. The variable data caching method according to claim 1, characterized in that: In step S14, a hash table is constructed separately for each data type of data block, and the hash functions used by the four hash tables are not required to be the same.

4. The variable data caching method according to claim 1, characterized in that: When you need to find a data block, specify the data type of the required data block and the corresponding variable name, concatenate the two to get the data index value, and then quickly locate the required data block based on the data index value, thereby achieving fast retrieval.

5. The variable data caching method according to claim 4, characterized in that: When it is necessary to find the starting address corresponding to a data index value, first calculate the hash code of the data index value, and then find the starting address of the corresponding data block in the hash table.

6. The variable data caching method according to claim 1, characterized in that: When you need to find a data box: 1) Get the metadata-related filtering conditions, traverse each data box, and the data box whose metadata matches the filtering conditions is the target data box; 2) Get the file name or data box ID value, and the data box that matches the file name or data box ID value is the target data box.

7. The variable data caching method according to any one of claims 1 to 6, characterized in that: Each time a new computing task is executed, a new round of data caching is performed dynamically.

8. The variable data caching method according to any one of claims 1 to 6, characterized in that: The variable data caching method is applied to application scenarios of frequent concurrent access to variable data in NetCDF files or access to large-scale NetCDF file data sets.

9. A variable data caching system for NetCDF files, characterized in that: comprising at least one processor and at least one memory; The at least one memory is used to store computer instructions; The at least one processor is used to execute at least part of the computer instructions to implement the variable data caching method according to any one of claims 1 to 8.

Images