Data processing method and apparatus for hdf5 data file

Abstract

This application provides a data processing method and apparatus for HDF5 data files. The method involves: obtaining a YML-based configuration file; locating the HDF5 data file to be processed based on the configuration file; processing the source metadata and source table data in the HDF5 data file according to the reuse information, data extraction information, inheritance information, and data calculation process in the configuration file; generating destination table data based on the processed data results; then generating a destination HDF5 data file using the destination table data and empty destination metadata, and storing it; this destination HDF5 data file is used for experimental data plotting and experimental analysis. This method enables low-cost data processing of HDF5 data files through configuration file settings.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, specifically to a data processing method and apparatus for HDF5 data files. Background Technology

[0002] HDF5 (Hierarchical Data Format 5) is a file format used for storing and organizing large amounts of data. HDF5 files employ a hierarchical data structure, can store rich metadata, support parallel I / O and various compression techniques, can handle large-scale datasets, and provide high-performance data read and write capabilities.

[0003] In materials science experiments, raw experimental data is often directly saved in HDF5 format to measure relevant material parameters. Table data stores parameter values, metadata records different experimental conditions, and each experiment's results are stored separately as an HDF5 file. This direct data storage leads to fragmented data storage, and since only raw data is stored, it hinders data visualization and experimental result analysis.

[0004] The data processing of the relevant HDF5 data files is all done through code, which requires professional personnel to execute, resulting in high labor costs. Summary of the Invention

[0005] In view of this, this application provides a data processing method and apparatus for HDF5 data files, which can process data in HDF5 data files at low cost by configuring configuration files.

[0006] To solve the above-mentioned technical problems, the technical solution of this application is implemented as follows:

[0007] In one embodiment, a data processing method for HDF5 data files is provided, based on a YML configuration configuration file; wherein, the configuration file defines a first type of top-level object and a second type of top-level object, as well as a method for generating the target HDF5 data file; for the first type of top-level object, reuse information and entry variables for the HDF5 data file to be processed are defined; the attribute information of the second type of top-level object is configured, the attribute information including: data extraction information, inheritance information, and data calculation process; the method includes:

[0008] Obtain the configuration file;

[0009] The HDF5 data file to be processed is located based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein, the HDF5 data file to be processed is the experimental data directly stored in HDF5 format during material experiments;

[0010] According to the reuse information in the configuration file, as well as the data extraction information, the inheritance information and the data calculation process, the source metadata and source table data in the HDF5 data file to be processed are processed accordingly, and the target table data is generated based on the processed data results.

[0011] The target HDF5 data file is generated using the target table data and empty target metadata according to the method for generating the target HDF5 data file described in the configuration file, and then stored. The target HDF5 data file is used for experimental data plotting and experimental analysis.

[0012] The reused information includes: global variables and global alias variables;

[0013] The global variable is used to define the variable values ​​used in objects other than the global variable in YML;

[0014] The global alias variable is used to define aliases for long strings in reusable variables.

[0015] The data extraction information includes:

[0016] Whether the second type of top-level object can be extracted, and whether the extraction conditions are met;

[0017] The second type of top-level object is defined by an array to determine whether it meets the extraction conditions: cc in the array indicates whether the source table data contains the specified column, and cm is used to check the source metadata. cm has two parameters: the first parameter is the metadata name, and the second parameter is the value to be matched. cc and cm are built-in functions of the configuration file handler.

[0018] The inherited information includes: the base object and local alias variables;

[0019] The base object is used to specify all second-class top-level objects that will inherit from the base object;

[0020] The local alias variable is used to define aliases for long strings in inherited variables;

[0021] If a global alias variable and a local alias variable have the same name, the local alias variable will be used.

[0022] The data calculation process includes:

[0023] The value of the variable to be calculated is obtained through the extracted information;

[0024] Perform the calculations according to the defined calculation method;

[0025] The calculation result is used as the calculation result of the second type of top-level object.

[0026] The method further includes:

[0027] If a defined calculation method contains multiple calculation steps, the result of the last calculation step will be used as the calculation result of the second type of top-level object.

[0028] The step of generating the target table data based on the processed data results includes:

[0029] The calculation results obtained for each of the second type of top-level objects are compared with the data extracted through the extraction information to obtain a set of names and values;

[0030] Combine all group names and values ​​corresponding to all top-level objects of the second type to generate the target table data.

[0031] In another embodiment, a data processing apparatus for HDF5 data files is provided, the apparatus comprising:

[0032] A configuration unit is used to configure a configuration file based on YML; wherein the configuration file defines a first type of top-level object and a second type of top-level object, as well as a method for generating the target HDF5 data file; for the first type of top-level object, it defines reuse information and entry variables for the HDF5 data file to be processed; and it configures the attribute information of the second type of top-level object, the attribute information including: data extraction information, inheritance information, and data calculation process.

[0033] The acquisition unit is used to acquire the configuration file;

[0034] The search unit is used to find the HDF5 data file to be processed based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein, the HDF5 data file to be processed is experimental data directly stored in HDF5 format during material experiments;

[0035] The processing unit is configured to process the source metadata and source table data in the HDF5 data file to be processed according to the reuse information, data extraction information, inheritance information and data calculation process in the configuration file, and generate destination table data based on the processed data results; generate a destination HDF5 data file using the destination table data and empty destination metadata according to the method for generating the destination HDF5 data file in the configuration file, and store it; the destination HDF5 data file is used for experimental data plotting and experimental analysis.

[0036] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a data processing method for HDF5 data files.

[0037] In another embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements a data processing method for HDF5 data files.

[0038] As can be seen from the above technical solution, the above embodiments provide a method to first configure a configuration file using YML, which defines a first type of top-level object and a second type of top-level object, as well as a method for generating the target HDF5 data file; based on the configuration file, the HDF5 data file to be processed is located, and the source metadata and source table data in the HDF5 data file are processed to obtain the target table data; then, according to the method for generating the target HDF5 data file, the target table data and empty target metadata are used to generate the target HDF5 data file, and the file is stored; this target HDF5 data file can be used for plotting experimental results and for experimental analysis. This method can achieve low-cost processing of data in HDF5 data files by configuring a configuration file. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a schematic diagram of the internal hierarchical structure of an HDF5 data file in the prior art;

[0041] Figure 2 This is a schematic diagram showing the contents of the entry variables of the HDF5 data file to be processed as defined in the embodiments of this application;

[0042] Figure 3 This is a schematic diagram illustrating the data extraction information of the second type of top-level object defined in this application embodiment;

[0043] Figure 4 This is a schematic diagram of the data processing method for HDF5 data files in an embodiment of this application;

[0044] Figure 5 This is a schematic diagram of the target table data generated in the embodiments of this application;

[0045] Figure 6This is a schematic diagram illustrating the configuration information for calculating id_vgg in the configuration file in this embodiment of the application.

[0046] Figure 7 This is a schematic diagram of the data processing device structure for HDF5 data files in an embodiment of this application;

[0047] Figure 8 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0049] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe the order or sequence of objects. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0050] The technical solution of the present invention will be described in detail below with reference to specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0051] HDF5 (Hierarchical Data Format 5) is a file format used for storing and organizing large amounts of data. HDF5 files employ a hierarchical data structure, can store rich metadata, support parallel I / O and various compression techniques, can handle large-scale datasets, and provide high-performance data read and write capabilities. Although HDF5 format data files are becoming increasingly widely used, data processing is not as intuitive and convenient after storage in HDF5 format. For example, extracting maximum, minimum, and average values ​​lacks universal and simple tools and requires special methods such as programming.

[0052] In materials science experiments, the raw experimental data, including measurements of relevant material parameters, is saved directly in HDF5 format. Table data stores parameter values, while metadata records different experimental conditions. Each experimental result is stored as a separate HDF5 data file. For example, when measuring the resistance of a material (the relationship between current and voltage), the raw experimental data is saved in HDF5 format. Table data stores current and voltage values, while metadata records different experimental conditions, such as temperature. Finally, multiple experimental results are stored in separate HDF5 data files.

[0053] Directly storing such data results in fragmented data storage, and since it only stores raw data, it is not conducive to data plotting and experimental result analysis.

[0054] The data processing of the relevant HDF5 data files is all done through code, which requires professional personnel to execute, resulting in high labor costs.

[0055] To address the aforementioned issues, this application provides a data processing method for HDF5 data files. First, a YML configuration file is used, defining a first type of top-level object, a second type of top-level object, and a method for generating the target HDF5 data file. Based on this configuration file, the HDF5 data file to be processed is located. The source metadata and source table data within the HDF5 data file are processed to obtain the target table data. Then, according to the target HDF5 data file generation method, the target table data and empty target metadata are used to generate the target HDF5 data file, which is then stored. This target HDF5 data file can be used for plotting experimental results and for experimental analysis. This method enables low-cost data processing of HDF5 data files through configuration file settings.

[0056] The HDF5 file structure contains two types of objects: Group and Dataset. A Group is similar to a folder or directory; each HDF5 file itself is like a root directory, and the Group is like a subdirectory. A Dataset is similar to an array in NumPy, or simply a table. The main data is presented in tabular form, which we call table data. Besides table data, a Dataset also has additional attributes, each consisting of a name and a corresponding value; these additional attributes are called metadata. The data within a Dataset (metadata and table data) will be processed in the following text; for ease of description, the Dataset can also be referred to as a dataset.

[0057] See Figure 1 , Figure 1This is a schematic diagram of the internal hierarchical structure of an HDF5 data file in the prior art. Figure 1 The left-hand directory tree shows the internal hierarchical nodes of an HDF5 data file. The root node, `sample`, is the HDF5 filename. `group1` and `group2` are the top-level groups under the root directory. Under `group1`, there are four Datasets named `sheet1`, `sheet2`, `sheet3`, and `sheet4`. The dashed box in the lower left corner shows the metadata corresponding to `sheet1`, hiding basic metadata such as rank, dimension, and data type, and only showing the custom attribute part, which is the metadata to be processed in this embodiment. The solid box on the right represents the data part of `sheet1`, that is, the table data.

[0058] In this embodiment, the Groups in the HDF5 data file are not considered; only the datasets under the Groups are processed. That is, the table data and metadata in the datasets are the data that need to be processed. To distinguish them from the final generated target table data in this embodiment, the metadata in the existing HDF5 data file will be referred to as source metadata, and the table data as source table data.

[0059] YML files are written in YAML (YAML Ain't Markup Language). YAML is an intuitive data serialization format that can be recognized by computers and is easily readable by humans. YML files are data-centric and more concise than traditional XML. YML files can use the extension .yml or .yaml.

[0060] YML uses the basic form "key: value," where the name and value are separated by a colon, to represent object values. For complex objects, the value is expanded into sub-objects, which can be nested indefinitely. Sub-objects are preceded by spaces to distinguish hierarchical relationships. Besides simple types (numbers, strings) and sub-objects, values ​​can also be arrays, represented by "-".

[0061] In this embodiment of the application, a YML configuration configuration file is used to process metadata and table data in the HDF5 data file. The specific configuration process and content are as follows:

[0062] This configuration file defines the first and second types of top-level objects, as well as the method for generating the target HDF5 data file. For the first type of top-level object, it defines reuse information and entry variables for the HDF5 data file to be processed. It also configures the attribute information for the second type of top-level object, including data extraction information, inheritance information, and the data calculation process. Each configuration file is explained below:

[0063] For the first type of top-level object, the reusable information includes: global variables (vars) and global alias variables (alias);

[0064] Global variables are used to define variable values ​​used in objects other than global variables in YML; each variable is defined as a sub-object.

[0065] The variable value type can be a number or a string.

[0066] This global variable will be used in other objects, such as to check whether the value of a certain source metadata variable in the source metadata corresponding to the source table data is equal to the value of the specified variable.

[0067] Defining global variables facilitates the reuse of configuration files.

[0068] Global alias variables are used to define aliases for long strings in reusable variables. Assuming the alias is named "cf", its value is... In other variable values, aliases are used in the form of "${cf}", such as "${cf}('vg', vb=vbs)". When processing this value, the alias will be replaced first, and the value will be replaced with... The main purpose of defining global alias variables is to avoid repeating long strings, making the configuration file content look more concise.

[0069] The entry variable (settings) for the HDF5 data files to be processed mainly contains HDF5 sub-objects. The content of the entry variable is an array, and each element in the array corresponds to the entry of an HDF5 data file to be processed. That is, each configuration file needs to process one or more HDF5 data files.

[0070] See Figure 2 , Figure 2 This is a schematic diagram of the contents of the entry variables for the HDF5 data file to be processed as defined in the embodiments of this application. Figure 2 In this context, `name` represents the name of the HDF5 data file to be processed, for example, `001.HDF5`. For the HDF5 data file configured in the configuration file, the system will search the array of `HDF5` under `settings`. Only when a matching entry is found will further processing be performed.

[0071] The `frame` directive specifies the target table data settings for generating HDF5 data files. The value of a `frame` is also an array, indicating that an HDF5 data file can generate multiple datasets. Within each sub-object of the `frame`, `name` represents the name of the dataset to be generated. Figure 2 Taking "targets" as an example, `metas` represents the source metadata to be extracted from the target table data in the HDF5 data file to be processed. For example, `prefix` under `metas` means to extract only metadata starting with "device_", and `exclude` means to exclude metadata. In addition to `prefix` and `exclude`, `postfix` is also supported, which filters source metadata by suffix. The extracted source metadata will be placed in the target table data.

[0072] Top-level objects other than the first type of top-level objects are classified as second-type top-level objects. The three main categories of properties for second-type top-level objects are configured as follows:

[0073] The data extraction information for the second type of top-level object includes:

[0074] Whether the second type of top-level object can be extracted, and whether the extraction conditions are met;

[0075] The second type of top-level object is defined by an array to determine whether it meets the extraction conditions: cc in the array indicates whether the source table data contains the specified column, and cm is used to check the source metadata. cm has two parameters: the first parameter is the metadata name, and the second parameter is the value to be matched. cc and cm are built-in functions of the configuration file handler.

[0076] See Figure 3 , Figure 3 This is a schematic diagram illustrating the data extraction information of the second type of top-level object defined in this application embodiment. Figure 3 In this context, `itarget` represents the name of the second type of top-level object. `calculable` indicates whether this second type of top-level object can be extracted; it can be represented by values ​​such as `true` and `false`, `1` and `0`, etc. This embodiment does not impose any restrictions on this. Figure 3 In this example, the value is either true or false. False means that it cannot be extracted, and true means that it can be extracted.

[0077] In specific configurations, the reason for defining a second type of top-level object that cannot be extracted is to use it to define the base second type of top-level object. If other second type of top-level objects set their parent to it, they will inherit the settings of all its child objects. The base second type of top-level object is to simplify the configuration file and avoid repeatedly defining similar second type of top-level objects, for example, except for some different variable values.

[0078] `cond` indicates whether the table data corresponding to the second type of top-level object in the HDF5 data file meets the extraction conditions. `cond` is the first four letters of `condition`. `cond` defines a series of conditions in array form. `cc` and `cm` are built-in functions of the configuration file handler. `cc` indicates whether the source table data contains the specified columns; `cc` is an abbreviation for `containscolumns`. `cm` is used to check source metadata. `cm` has two parameters: the first is the source metadata name, and the second is the value to be matched. Figure 3 In the code "cm('cond_vd', ${meta_vd})", the value is defined as an alias. Here, the global alias does not define the alias "meta_vd"; its definition comes from other second-class top-level objects that have the underlying second-class top-level object as their parent.

[0079] The second type of inheritance information for top-level objects includes: the base object (parent) and local alias variables (alias);

[0080] Among them, the base object is used to specify all second-class top-level objects that will inherit from the base object;

[0081] Local alias variables are used to define aliases for long strings in inherited variables;

[0082] If a global alias variable and a local alias variable have the same name, the local alias variable will be used.

[0083] The data computation process for the second type of top-level object includes:

[0084] The values ​​of the variables to be calculated are obtained by extracting information;

[0085] Perform the calculations according to the defined calculation method;

[0086] The calculation result is used as the calculation result of the second type of top-level object.

[0087] If a defined calculation method contains multiple calculation steps, the result of the last calculation step will be used as the calculation result of the second type of top-level object.

[0088] The configuration for the data calculation process is still based on Figure 3 As shown in the example, Figure 3 middle

[0089] column_vgs: ${cf}('vg', vb=vbs)

[0090] column_ids: ${cf}('id', vb=vbs)

[0091] Here, both column_vgs and column_ids are calculation processes. Taking column_vgs as an example, first, the alias is replaced, and the value is... The dataset contains two built-in functions, `m` and `c`. `m` reads the values ​​of source metadata from the dataset to be extracted. It takes two arguments: the first is the metadata name, and the second is a default value. If the source metadata does not contain this metadata name, the default value is used. `c` reads the values ​​of a specified column from the dataset. It also takes two arguments: the first is the column name, and the second is the filter condition. `c('vg', vb=vbs)` means reading all rows where the value of column `vb` in column `vg` equals the value of `vbs`. The result of `c` can be a single value or an array. The value of `vbs` comes from the loop variable defined in the `loop` object.

[0092] column_ids means reading all the rows in which the value of another column (vb) is equal to the value of vbs.

[0093] target: closest(column_vgs, column_ids, ${vgg})

[0094] In this calculation process, the values ​​of column_vgs and column_ids obtained earlier are used and substituted into the closest built-in function to find the value of ids corresponding to the row position of column_vgs that is closest to ${vgg}, and use it as the value of itarget.

[0095] The value of the final calculation is the final value of the target. Combined with the extracted metadata values ​​defined in the settings, this yields a set of name-value objects. By integrating all the target values, the resulting target table data can be obtained.

[0096] Figure 3 In the `target` directive, `column_vgs`, `column_ids`, and `itarget` are not directly written under `target`, but rather placed under `loop`. `loop` is a higher-order computation process that indicates a recurring computation. The `loop` header contains a `vars` definition, indicating that the values ​​in `vars` are iterated over, and all permutations of the values ​​in `vars` are performed, with the computation process under `loop` calculating the value once for each combination. This allows `loop` to obtain multiple `target` values. If there is no need for `loop`, the computation process can be directly placed under `target`.

[0097] The method for generating the destination HDF5 data file is to use the destination table data as the table data in the destination HDF5 data file, and use the empty destination metadata as the metadata of the destination HDF5 data file to generate the destination HDF5 data file; while the original HDF5 data file is not changed.

[0098] This completes the configuration of the configuration file. The configuration file can then be used to process HDF5 data files to obtain the target data table, and then used to generate new HDF5 data files for plotting and analysis. The specific process is as follows:

[0099] See Figure 4 , Figure 4 This is a schematic diagram of the data processing method for HDF5 data files in an embodiment of this application. The specific steps are as follows:

[0100] Step 401: Obtain the configuration file.

[0101] This configuration file is used to configure the extraction and calculation of data from the HDF5 data file to be processed, as well as the generation of the target HDF5 data file.

[0102] Step 402: Locate the HDF5 data file to be processed based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein, the HDF5 data file to be processed is the experimental data stored directly in HDF5 format during material experiments.

[0103] The application scenario of this application can be experimental data recording, that is, storing experimental results data as HDF5 data files to be processed.

[0104] An experiment may be conducted once or multiple times, and the results of each experiment can be stored in one or more HDF5 data files. Multiple HDF5 data files can be searched in an array. In the embodiment of the application, the entry variable for the HDF5 data file to be processed can find one or more HDF5 data files, which is determined according to actual needs.

[0105] Step 403: Based on the reuse information, data extraction information, inheritance information and data calculation process in the configuration file, process the source metadata and source table data in the HDF5 data file to be processed accordingly, and generate the destination table data based on the processed data results.

[0106] For the configured HDF5 data files to be processed, the configuration reuse information is used to reuse the relevant parameters, and the configuration inheritance information is used to inherit the relevant parameters, which simplifies the configuration file and enables quick location of the relevant parameters;

[0107] Extract data corresponding to the relevant objects in the configuration based on the data extraction information.

[0108] For the configured data calculation process, the specified object is identified, and the value corresponding to the object is calculated according to the calculation method specified in the data calculation process to obtain the calculation result;

[0109] For each second-class top-level object, the calculation results are compared with the data extracted through information extraction to obtain a set of names and values;

[0110] Combine all group names and values ​​corresponding to all top-level objects of the second type to generate the target table data.

[0111] See Figure 5 , Figure 5 This is a schematic diagram of the target table data generated in the embodiments of this application. Figure 5 The target metadata section is empty. The target table data includes the content corresponding to the dashed and solid boxes on the right. The data in the dashed boxes of the target table data comes from the source metadata, and the data in the solid boxes comes from the calculation results. Each column in the solid box is an extractable second-type top-level object. The data in each row of the dashed boxes corresponds one-to-one with the data in the implementation boxes. The data in the dashed boxes comes from the source metadata of the source table data, and the data in the implementation boxes comes from the second-type top-level objects calculated based on the same source table data. For example, Figure 5 The 8th line in the text comes from Figure 1 Display source metadata and selected source table data.

[0112] See Figure 6 , Figure 6 This is a schematic diagram illustrating the configuration information for calculating id_vgg in the configuration file in this embodiment of the application. Figure 6 The second type of top-level object is set with inheritance information, specifically including: the base object (parent) and local alias variables (alias). Taking a voltage and current related experiment as an example, id_vgg refers to the current value when vg reaches its maximum value, which is also the maximum current value; meta_vd: vd_threshold; this alias is used to check the source metadata to determine the voltage condition of Vd during the test; vgg: vg_max, this alias is used to confirm that the value of the vg column is the maximum value of Vg; Threshold (threshold voltage) refers to the voltage value at which the state of an electronic device or circuit changes when the voltage reaches a certain specific value; for example, the calculation result id_vgg in line 8 is 7.59304e-7, which is the calculation result corresponding to the following calculation: find the value of ids corresponding to the row position of column_vgs that is closest to vg_max, and use it as the value of itarget, where vg_max is set to 3.3.

[0113] In practical applications, the calculation method can be configured according to the actual application, such as calculating the average value, variance, etc.

[0114] Step 404: Generate a target HDF5 data file using the target table data and empty target metadata according to the target HDF5 data file generation method in the configuration file, and store it.

[0115] When generating the target HDF5 data file, this application uses the target table data as the table data in the target HDF5 data file and uses the empty target metadata as the metadata in the target HDF5 data file. In other words, it does not change the format of the HDF5 data file, but only uses the target table data and target metadata to generate the target HDF5 data file.

[0116] When you need to use the HDF5 data file for this purpose, you can obtain it directly, such as for plotting experimental results or conducting experimental analysis.

[0117] In this embodiment, a YML configuration file is first used to define a first type of top-level object and a second type of top-level object, as well as a method for generating the target HDF5 data file. Based on this configuration file, the HDF5 data file to be processed is located, and the source metadata and source table data in the HDF5 data file are processed to obtain the target table data. Then, according to the method for generating the target HDF5 data file, the target table data and empty target metadata are used to generate the target HDF5 data file, which is then stored. This target HDF5 data file can be used for plotting experimental results and for experimental analysis. This method can achieve low-cost processing of data in HDF5 data files by configuring a configuration file.

[0118] In this embodiment, the experimental results data can be plotted directly using the target table data, or a target HDF5 data file can be generated and stored first, and then the stored target HDF5 data file can be used directly to plot the experimental results data when needed.

[0119] In this embodiment, the HDF5 data file that directly stores experimental data is processed through a configuration file to obtain a target HDF5 data file. This target HDF5 data file can be compressed for storage, reducing storage space and facilitating transmission. Moreover, when in use, the required data can be retrieved more quickly.

[0120] In this application embodiment, the experimental results data can be plotted as a curve or other graph. The specific graph can be determined according to the needs of the specific application scenario. The following are application scenarios for material experiments:

[0121] The resistivity characteristics of the measured material, i.e., the relationship between current and voltage, are stored in the existing HDF5 format to save the experimental data obtained directly from the experiment, i.e., the HDF5 data file to be processed in this embodiment. The table data (source table data) of the HDF5 data file to be processed is used to store the series values ​​of current and voltage, and the metadata (source metadata) of the HDF5 data file to be processed is used to record different experimental conditions, such as temperature. Finally, multiple HDF5 data files to be processed are stored based on the experimental results. Then, the method provided in this application is used to process the metadata and table data in the experimental results, i.e., the HDF5 data file to be processed, according to the configuration file in the configuration file to generate the target table data. The target table data is used to replace the source table data and the source-source data in the target HDF5 data file. Finally, based on the target HDF5 data file, the resistance change curves at different temperatures can be plotted.

[0122] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0123] Based on the same inventive concept, this application also provides a data processing device for HDF5 data files. See also Figure 7 , Figure 7 This is a schematic diagram of the data processing device for HDF5 data files in an embodiment of this application. The device includes:

[0124] Configuration unit 701 is used to configure a configuration file based on YML. The configuration file defines a first type of top-level object and a second type of top-level object, as well as the method for generating the target HDF5 data file. For the first type of top-level object, it defines reuse information and entry variables for the HDF5 data file to be processed. It configures the attribute information of the second type of top-level object, including data extraction information, inheritance information, and data calculation process.

[0125] Acquisition unit 702 is used to acquire configuration files;

[0126] The lookup unit 703 is used to find the HDF5 data file to be processed based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein, the HDF5 data file to be processed is the experimental data directly stored in HDF5 format when conducting material experiments;

[0127] The processing unit 704 is used to process the source metadata and source table data in the HDF5 data file to be processed according to the reuse information, data extraction information, inheritance information and data calculation process in the configuration file, and generate the destination table data based on the processed data results; it generates the destination HDF5 data file using the destination table data and empty metadata according to the generation method of the destination HDF5 data file in the configuration file, and stores it; the destination HDF5 data file is used for experimental result data plotting and experimental analysis.

[0128] In another example, the reused information includes: global variables and global alias variables;

[0129] Global variables are used to define variable values ​​used in objects other than global variables in YML.

[0130] Global alias variables are used to define aliases for long strings in reusable variables.

[0131] In another example, the data extraction information includes:

[0132] Whether the second type of top-level object can be extracted, and whether the extraction conditions are met;

[0133] The second type of top-level object is defined by an array to determine whether it meets the extraction conditions: cc in the array indicates whether the source table data contains the specified column, and cm is used to check the source metadata. cm has two parameters: the first parameter is the metadata name, and the second parameter is the value to be matched. cc and cm are built-in functions of the configuration file handler.

[0134] In another example, inheritance information includes: the base object and local alias variables;

[0135] Among them, the base object is used to specify all second-class top-level objects that will inherit from the base object;

[0136] Local alias variables are used to define aliases for long strings in inherited variables;

[0137] If a global alias variable and a local alias variable have the same name, the local alias variable will be used.

[0138] In another example, the data calculation process includes:

[0139] The values ​​of the variables to be calculated are obtained by extracting information;

[0140] Perform the calculations according to the defined calculation method;

[0141] The calculation result is used as the calculation result of the second type of top-level object.

[0142] In another example,

[0143] If a defined calculation method contains multiple calculation steps, the result of the last calculation step will be used as the calculation result of the second type of top-level object.

[0144] In another example,

[0145] The processing unit 704 is specifically used to generate target table data based on the processed data results. For each second-class top-level object, the calculation results are combined with the data extracted by extracting information to obtain a set of names and values. All the group names and values ​​corresponding to all second-class top-level objects are integrated together to generate target table data.

[0146] The units in the above embodiments can be integrated into one unit or deployed separately; they can be merged into one unit or further divided into multiple sub-units.

[0147] In another embodiment, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a data processing method for HDF5 data files.

[0148] In another embodiment, a computer-readable storage medium is also provided, on which computer instructions are stored, which, when executed by a processor, implement a data processing method for an HDF5 data file.

[0149] Figure 8 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention. Figure 8 As shown, the electronic device may include: a processor 810, a communications interface 820, a memory 830, and a communication bus 840, wherein the processor 810, the communications interface 820, and the memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute the following methods:

[0150] The configuration file is based on YML; it defines a first type of top-level object and a second type of top-level object, as well as the method for generating the target HDF5 data file; it defines reuse information and entry variables for the HDF5 data file to be processed for the first type of top-level object; and it configures the attribute information of the second type of top-level object, including data extraction information, inheritance information, and data calculation process, which includes:

[0151] Get the configuration file;

[0152] The HDF5 data file to be processed is located based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein, the HDF5 data file to be processed is the experimental data stored directly in HDF5 format during material experiments;

[0153] Based on the reuse information in the configuration file, as well as the data extraction information, inheritance information and data calculation process, the source metadata and source table data in the HDF5 data file to be processed are processed accordingly, and the target table data is generated based on the processed data results.

[0154] The target HDF5 data file is generated using the target table data and empty metadata according to the generation method in the configuration file, and then stored. This target HDF5 data file is used for plotting experimental results and for experimental analysis.

[0155] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0157] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0158] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments disclosed in this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those shown in the drawings. For example, two blocks shown connectedly may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0159] Those skilled in the art will understand that the features described in the various embodiments and / or claims disclosed in this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments and / or claims of this application can be combined and / or combined in various ways, and all such combinations and / or combinations fall within the scope of this application.

[0160] This document uses specific embodiments to illustrate the principles and implementation methods of the present invention. The descriptions of these embodiments are merely illustrative of the method and core concepts of the present invention and are not intended to limit this application. Those skilled in the art can make changes to the specific implementation methods and application scope based on the ideas, spirit, and principles of the present invention. Any modifications, equivalent substitutions, or improvements made should be included within the scope of protection of this application.

Claims

1. A data processing method for HDF5 data files, characterized in that, Based on a YML configuration file; wherein, the configuration file defines a first type of top-level object and a second type of top-level object, as well as a method for generating the target HDF5 data file; for the first type of top-level object, it defines reuse information and entry variables for the HDF5 data file to be processed; and it configures the attribute information of the second type of top-level object, the attribute information including: data extraction information, inheritance information, and data calculation process, the method including: Obtain the configuration file; The HDF5 data file to be processed is located based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein, the HDF5 data file to be processed is the experimental data directly stored in HDF5 format during material experiments; According to the reuse information in the configuration file, as well as the data extraction information, the inheritance information and the data calculation process, the source metadata and source table data in the HDF5 data file to be processed are processed accordingly, and the target table data is generated based on the processed data results. The target HDF5 data file is generated using the target table data and empty target metadata according to the method for generating the target HDF5 data file described in the configuration file, and then stored. The target HDF5 data file is used for experimental data plotting and experimental analysis.

2. The method according to claim 1, characterized in that, The reused information includes: global variables and global alias variables; The global variable is used to define the variable values ​​used in objects other than the global variable in YML; The global alias variable is used to define aliases for long strings in reusable variables.

3. The method according to claim 1, characterized in that, The data extraction information includes: Whether the second type of top-level object can be extracted, and whether the extraction conditions are met; The second type of top-level object is defined by an array to determine whether it meets the extraction conditions: cc in the array indicates whether the source table data contains the specified column, and cm is used to check the source metadata. cm has two parameters: the first parameter is the metadata name, and the second parameter is the value to be matched. cc and cm are built-in functions of the configuration file handler.

4. The method according to claim 1, characterized in that, The inherited information includes: the base object and local alias variables; The base object is used to specify all second-class top-level objects that will inherit from the base object; The local alias variable is used to define aliases for long strings in inherited variables; If a global alias variable and a local alias variable have the same name, the local alias variable will be used.

5. The method according to claim 1, characterized in that, The data calculation process includes: The value of the variable to be calculated is obtained through the extracted information; Perform the calculations according to the defined calculation method; The calculation result is used as the calculation result of the second type of top-level object.

6. The method according to claim 5, characterized in that, The method further includes: If a defined calculation method contains multiple calculation steps, the result of the last calculation step will be used as the calculation result of the second type of top-level object.

7. The method according to claim 1, characterized in that, The process of generating the target table data based on the processed data results includes: The calculation results obtained for each of the second type of top-level objects are compared with the data extracted through the extraction information to obtain a set of names and values; Combine all group names and values ​​corresponding to all top-level objects of the second type to generate the target table data.

8. A data processing device for HDF5 data files, characterized in that, The device includes: A configuration unit is used to configure a configuration file based on YML; wherein the configuration file defines a first type of top-level object and a second type of top-level object, as well as a method for generating the target HDF5 data file; for the first type of top-level object, it defines reuse information and entry variables for the HDF5 data file to be processed; and it configures the attribute information of the second type of top-level object, the attribute information including: data extraction information, inheritance information, and data calculation process. The acquisition unit is used to acquire the configuration file; The search unit is used to find the HDF5 data file to be processed based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein, the HDF5 data file to be processed is experimental data directly stored in HDF5 format during material experiments; The processing unit is configured to process the source metadata and source table data in the HDF5 data file to be processed according to the reuse information, data extraction information, inheritance information and data calculation process in the configuration file, and generate destination table data based on the processed data results; generate a destination HDF5 data file using the destination table data and empty destination metadata according to the method for generating the destination HDF5 data file in the configuration file, and store it; the destination HDF5 data file is used for experimental data plotting and experimental analysis.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the method described in any one of claims 1-7.

Images