Data processing method and device for HDF5 data file

Abstract

The invention provides a data processing method and device for an HDF5 data file. Acquiring a configuration file based on YML configuration; searching an HDF5 data file to be processed based on the configuration file; performing corresponding processing on source metadata and source table data in the to-be-processed HDF5 data file according to multiplexing information in the configuration file, data extraction information, inheritance information and a data calculation process, and generating target table data based on a processed data result; generating a target HDF5 data file by using the target table data and the target metadata with empty content, and storing the target HDF5 data file; the target HDF5 data file is used for experimental data drawing and experimental analysis. According to the method, the data in the HDF5 data file can be processed at low cost by configuring the configuration file.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method and device for processing HDF5 data files. Background Art

[0002] HDF5 (Hierarchical Data Format 5) is a file format used to store and organize large amounts of data. HDF5 files use a hierarchical data structure, can store rich metadata, support parallel I / O and multiple compression technologies, can handle large-scale data sets, and provide high-performance data reading and writing capabilities.

[0003] In the material experiment scenario, the relevant parameter characteristics of the measured material are directly saved in the HDF5 format as the original experimental data, where the table data is used to store the parameter characteristic values, the metadata is used to record different experimental conditions, and each experimental result is stored as an HDF5 data file. Such direct data storage leads to scattered data storage, and the stored data is raw data, which is not conducive to data drawing and experimental result analysis. The data processing of related HDF5 data files is all implemented through code, which requires professional execution and has relatively high labor costs. Summary of the invention

[0004] In view of this, the present application provides a data processing method and device for an HDF5 data file, which can process the data in the HDF5 data file at a low cost by configuring a configuration file.

[0005] To solve the above technical problems, the technical solution of this application is implemented as follows: In one embodiment, a method for processing data of an HDF5 data file is provided, which is based on a YML configuration configuration file; wherein the configuration file defines a first type of top-level object and a second type of top-level object, and a method for generating a target HDF5 data file; defines reuse information and an entry variable of a to-be-processed HDF5 data file for the first type of top-level object; configures attribute information of the second type of top-level object, wherein the attribute information includes: data extraction information, inheritance information, and a data calculation process, and the method includes: Obtaining the configuration file; Finding the HDF5 data file to be processed based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein the HDF5 data file to be processed is experimental data directly stored in the HDF5 format when conducting material experiments; Processing the source metadata and source table data in the HDF5 data file to be processed accordingly according to the reuse information in the configuration file, the data extraction information, the inheritance information and the data calculation process, and generating the destination table data based on the processed data results; The target HDF5 data file is generated and stored using the target table data and the target metadata with empty content according to the target HDF5 data file generation method in the configuration file; the target HDF5 data file is used for experimental data drawing and experimental analysis.

[0006] Wherein, the reuse information includes: global variables and global alias variables; The global variables are used to define variable values ​​used in objects other than the global variables in YML; The global alias variable is used to define an alias for a long string in a reused variable.

[0007] The data extraction information includes: Whether the second type of top-level object can be extracted and whether it meets the extraction conditions; Among them, whether the second type of top-level object meets the extraction conditions is defined in an array manner: cc in the array indicates whether the source table data contains the specified column, and cm is used to check the source metadata, where cm has two parameters, the first parameter is the metadata name, and the second parameter is the value to be matched; cc and cm are built-in functions of the configuration file handler.

[0008] The inheritance information includes: basic objects and local alias variables; The base object is used to specify all second-class top-level objects that will inherit the base object; The local alias variable is used to define an alias for a long string in an inherited variable; If the global alias variable and the local alias variable have the same name, the local alias variable will be used.

[0009] The data calculation process includes: Obtaining the value of the variable to be calculated through the extracted information; Calculate according to the defined calculation method; The calculation result is used as the calculation result of the second-type top-level object.

[0010] Wherein, the method further comprises: If there are multiple calculation steps in the defined calculation method, the calculation result of the last calculation step is used as the calculation result of the second-type top-level object.

[0011] The step of generating target table data based on the processed data results includes: The calculation results obtained for each of the second-class top-level objects are respectively combined with the data extracted by the extraction information to obtain a set of names and values; All group names and values ​​corresponding to all the second-category top-level objects are integrated together to generate destination table data.

[0012] In another embodiment, a data processing device for HDF5 data files is provided, the device comprising: A configuration unit is used to configure a configuration file based on YML; wherein the configuration file defines a first type of top-level object and a second type of top-level object, and a method for generating a target HDF5 data file; defines multiplexing information and an entry variable of a to-be-processed HDF5 data file for the first type of top-level object; configures attribute information of the second type of top-level object, wherein the attribute information includes: data extraction information, inheritance information, and data calculation process, An acquisition unit, used for acquiring the configuration file; A search unit, used for searching the HDF5 data file to be processed based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein the HDF5 data file to be processed is experimental data directly stored in the HDF5 format when performing material experiments; A processing unit is used to process the source metadata and source table data in the HDF5 data file to be processed according to the reuse information in the configuration file, the data extraction information, the inheritance information and the data calculation process, and generate the destination table data based on the processed data results; generate the destination HDF5 data file using the destination table data and the destination metadata with empty content according to the generation method of the destination HDF5 data file in the configuration file, and store it; the destination HDF5 data file is used for experimental data drawing and experimental analysis.

[0013] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements a data processing method for an HDF5 data file when executing the program.

[0014] In another embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, a data processing method for an HDF5 data file is implemented.

[0015] As can be seen from the above technical solution, the above embodiment provides a method for generating a target HDF5 data file by first configuring a configuration file through YML, in which the first type of top-level objects and the second type of top-level objects are defined, and the target HDF5 data file is generated; based on the configuration file, the HDF5 data file to be processed is found, and the source metadata and source table data in the HDF5 data file are processed to obtain the target table data; and the target HDF5 data file is generated and stored using the target table data and the target metadata with empty content according to the target HDF5 data file generation method; the target HDF5 data file can be used for experimental result data drawing and experimental analysis. This method can realize the processing of data in the HDF5 data file at a low cost by configuring the configuration file. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 It is a schematic diagram of the internal hierarchical structure of HDF5 data files in the prior art; Figure 2 A schematic diagram of the contents of the entry variables of the HDF5 data file to be processed as defined in the embodiments of the present application; Figure 3 A schematic diagram of the content of data extraction information defining the second type of top-level object in an embodiment of the present application; Figure 4 This is a flow chart of a method for processing HDF5 data files in an embodiment of the present application; Figure 5 A schematic diagram of the target table data generated in an embodiment of the present application; Figure 6 This is a schematic diagram of configuration information for calculating id_vgg in a configuration file in an embodiment of the present application; Figure 7 This is a schematic diagram of the structure of a data processing device for HDF5 data files in an embodiment of the present application; Figure 8 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe the order or precedence of the objects. It should be understood that the data used in this way can be interchanged where appropriate, so that the implementation of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0020] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0021] HDF5 (Hierarchical Data Format 5) is a file format used to store and organize large amounts of data. HDF5 files use a hierarchical data structure, can store rich metadata, support parallel I / O and multiple compression technologies, can handle large data sets, and provide high-performance data reading and writing capabilities. Although HDF5 format data files have been increasingly widely used. However, after the data is stored in HDF5 format, data processing is not so intuitive and convenient. For example, to extract the maximum, minimum, average, etc., there is no general and simple tool, and it can only be done through special methods such as programming.

[0022] For example, in the material experiment scenario, when conducting material experiments, the relevant parameter characteristics of the measured materials are directly saved in HDF5 format to save the original experimental data, where the table data is used to store the parameter characteristic values, the metadata is used to record different experimental conditions, and each experimental result is stored as an HDF5 data file. For example, when measuring the resistance characteristics of the material, that is, the relationship between current and voltage, the original experimental data is saved in HDF5 format, where the table data is used to store the current and voltage series values, and the metadata is used to record different experimental conditions, such as temperature. Finally, the results of multiple experiments are stored in different HDF5 data files.

[0023] Such direct data storage results in scattered data storage, and all data stored is raw data, which is not conducive to data drawing and experimental results analysis.

[0024] The data processing of related HDF5 data files is all implemented through code, which requires professional execution and has relatively high labor costs.

[0025] Based on the above problems, a data processing method for HDF5 data files is provided in an embodiment of the present application. First, a configuration file is configured through YML, in which a first-class top-level object and a second-class top-level object, as well as a method for generating a target HDF5 data file are defined; based on the configuration file, the HDF5 data file to be processed is found, and the source metadata and source table data in the HDF5 data file are processed to obtain the target table data; and according to the target HDF5 data file generation method, the target table data and the target metadata with empty content are used to generate the target HDF5 data file, and store it; the target HDF5 data file can be used for experimental result data drawing and experimental analysis. This method can realize the processing of data in HDF5 data files at a low cost by configuring the configuration file.

[0026] There are two types of objects in the HDF5 file structure: Group and Dataset. Group is similar to a folder or directory. Each HDF5 file itself is equivalent to a root directory, and Group is a subdirectory. Dataset is similar to the array in NumPy, and can also be simply understood as a table. The main data part exists in the form of a table, which we call table data. In addition to table data, Dataset also has additional attributes. An attribute is a name and a corresponding value. The additional attributes are called metadata. What we will deal with in the following is the data in Dataset (metadata and table data). For the convenience of the following description, Dataset can also be called a data set.

[0027] See also Figure 1 , Figure 1 Schematic diagram of the internal hierarchical structure of HDF5 data files in the prior art. Figure 1The directory tree on the left shows the internal hierarchical nodes of an HDF5 data file. The root node sample is the HDF5 file name, group1 and group2 are the top-level groups under the root directory, and group1 has four Datasets named sheet1, sheet2, sheet3, and sheet4. The dotted box in the lower left corner shows the content corresponding to the metadata of sheet1, hiding the basic metadata such as rank, dimension, data type, etc., and only showing the custom attribute part, which is the metadata to be processed in the embodiment of this application; the solid box on the right is the data part of sheet1, that is, the table data.

[0028] In the embodiment of the present application, the Group in the HDF5 data file is not concerned, and only the data set under the Group is processed. That is, the table data and metadata in the data set are the data that need to be processed. In order to distinguish from the target table data finally generated in the embodiment of the present application, the metadata in the existing HDF5 data file is referred to as source metadata, and the table data is referred to as source table data in the embodiment of the present application.

[0029] The YML file format is written in YAML (YAML Ain't Markup Language), which is an intuitive data serialization format that can be recognized by computers and is easy for humans to read. YML files are data-centric and more concise than traditional XML. The extension of the YML file can be .yml or .yaml.

[0030] YML uses "key: value", that is, the name and value are separated by a colon, to represent the object value. For complex objects, the value is expanded and represented by sub-objects, which can be nested continuously. Spaces are used before sub-objects to distinguish the hierarchical relationship. In addition to simple types (numbers, strings) and sub-objects, values ​​can also be in array form, represented by "-".

[0031] In the embodiment of the present application, based on the YML configuration file, the configuration file is used to process the metadata and table data in the HDF5 data file. The specific configuration process and content are as follows: The configuration file defines the first type of top-level objects and the second type of top-level objects, as well as the method for generating the target HDF5 data file; defines the reuse information and the entry variables of the HDF5 data file to be processed for the first type of top-level objects; and configures the attribute information of the second type of top-level objects, which includes: data extraction information, inheritance information and data calculation process. The following describes each configuration file separately: The reuse information defined for the first type of top-level objects includes: global variables (vars) and global alias variables (alias); Among them, global variables are used to define variable values ​​used in objects other than global variables in YML; each variable is defined in the form of a sub-object.

[0032] Variable value types can be numbers or strings.

[0033] This global variable will be used in other objects, for example, to check whether the value of a source metadata variable in the source metadata corresponding to a source table data is equal to the value of a specified variable.

[0034] The definition of global variables facilitates the reuse of configuration files.

[0035] Global alias variables are used to define aliases for long strings in reused variables. Assuming the alias name is "cf", its value is , the alias is used in the value of other variables in the form of "${cf}", such as "${cf}('vg', vb=vbs)". When processing this value, the alias will be replaced first, and the value will be replaced The definition of global alias variables is mainly to avoid repeating some long strings and make the content of the configuration file look more concise.

[0036] The entry variable (settings) of the HDF5 data file to be processed, whose main content is the HDF5 sub-object. The content corresponding to the entry variable is an array, and each element in the array corresponds to the entry of an HDF5 data file to be processed, that is, each configuration file is to process one or more HDF5 data files.

[0037] See also Figure 2 , Figure 2 This is a schematic diagram of the contents of the entry variables of the HDF5 data file to be processed defined in the embodiment of the present application. Figure 2 In the above code, name indicates the name of the HDF5 data file to be processed, for example, 001.HDF5. For the HDF5 data file to be processed configured in the configuration file, it will be searched from the HDF5 array under settings, and only the entry with the matching name will be further processed.

[0038] Frame represents the destination table data setting for processing HDF5 data files. The value of frame is also an array, indicating that one HDF5 data file can generate multiple datasets. In each sub-object of frame, name represents the name of the dataset to be generated. Figure 2Take "targets" as an example. metas indicates the source metadata to be extracted from the target table data in the HDF5 data file to be processed. For example, prefix under metas indicates that only metadata starting with "device_" is extracted, and exclude indicates metadata to be excluded. In addition to prefix and exclude, postfix is ​​also supported to filter source metadata by suffix. The extracted source metadata will be placed in the target table data.

[0039] The top-level objects other than the first-level top-level objects are the second-level top-level objects. The three major attributes of the second-level top-level objects are configured as follows: The data extraction information of the second type of top-level objects includes: Whether the second-class top-level object can be extracted and whether it meets the extraction conditions; Among them, whether the second type of top-level object meets the extraction conditions is defined in an array manner: cc in the array indicates whether the source table data contains the specified column, and cm is used to check the source metadata, where cm has two parameters, the first parameter is the metadata name, and the second parameter is the value to be matched; cc and cm are built-in functions of the configuration file handler.

[0040] See also Figure 3 , Figure 3 This is a schematic diagram of the content of data extraction information defining the second type of top-level objects in an embodiment of the present application. Figure 3 In the above, itarget indicates the name of the second-class top-level object. calculable indicates whether the second-class top-level object can be extracted, which can be represented by values ​​such as true and false, 1 and 0, etc. This embodiment does not limit this. Figure 3 In the example, the value is true or false, false means it cannot be extracted, and true means it can be extracted.

[0041] In the specific configuration, the reason why there is a second-class top-level object that cannot be extracted is to use it to define the basic second-class top-level object. If other second-class top-level objects set their parent to it, they will inherit all the child object settings under it. The basic second-class top-level object is to simplify the configuration file and avoid repeated definition of similar second-class top-level objects, such as different values ​​of some variables.

[0042] cond indicates whether the table data corresponding to the second-class top-level object in the HDF5 data file to be processed meets the extraction condition. cond is the first four letters of condition. cond defines a series of conditions in the form of an array. cc and cm are built-in functions of the configuration file processing program. cc indicates whether the source table data contains the specified column. cc is the abbreviation of containscolumns. cm is used to check the source metadata. cm has two parameters. The first parameter is the source metadata name, and the second parameter is the value to be matched. Figure 3 In "cm('cond_vd', ${meta_vd})", the value is defined in the form of alias. Here, the global alias does not define the alias meta_vd, and its definition comes from other second-class top-level objects that have the basic second-class top-level object as parent.

[0043] The second type of inheritance information of top-level objects includes: base objects (parent) and local alias variables (alias); The base object is used to specify all second-class top-level objects that will inherit the base object; Local alias variables, used to define aliases for long strings in inherited variables; If the global alias variable and the local alias variable have the same name, the local alias variable is used.

[0044] The data calculation process of the second type of top-level objects includes: Obtain the value of the variable to be calculated by extracting information; Calculate according to the defined calculation method; The calculation result is used as the calculation result of the second-class top-level object.

[0045] If there are multiple calculation steps in the defined calculation method, the calculation result of the last calculation step is used as the calculation result of the second-class top-level object.

[0046] The configuration for the data calculation process is still based on Figure 3 As shown in the example, Figure 3 middle column_vgs: ${cf}('vg', vb=vbs) column_ids: ${cf}('id', vb=vbs) Among them, column_vgs and column_ids are both a calculation process. Taking column_vgs as an example, first replace the alias in it, and the value is , which has two built-in functions m and c. m means reading the value of the source metadata in the dataset to be extracted. It has two parameters. The first parameter is the metadata name and the second parameter is the default value. If the source metadata does not contain this metadata name, the default value is used. c means reading the value of the specified column in the dataset. It also has two parameters. The first parameter is the column name and the second parameter is the filter condition. "c ('vg', vb=vbs)" means reading all the values ​​of the rows in which the value of another column vb in all rows of column vg is equal to vbs. The result of c can be a single point value or an array. The value of vbs comes from the loop variable defined by the loop object.

[0047] column_ids means read all the values ​​of the columns id in all the rows where the value of another column vb is equal to vbs.

[0048] target: closest(column_vgs, column_ids, ${vgg}) During this calculation process, the previously obtained values ​​of column_vgs and column_ids are used and substituted into the closest built-in function to find the value of ids corresponding to the row position of column_vgs closest to ${vgg} as the value of itarget.

[0049] The value of the last calculation process is the final value of the target. Combined with the value of the metadata extracted in settings, we can get a set of name and value objects. By integrating the values ​​of all targets together, we can get the result target table data.

[0050] Figure 3 In the code, column_vgs, column_ids, and itarget are not written directly under target, but under loop. Loop is a high-level calculation process, which means loop calculation. There is a vars definition at the head of loop, which means looping over the values ​​in vars, and fully permuting all the values ​​of vars. Under each combination, the calculation process under loop will calculate the value once. In this way, loop can get multiple target values. If there is no need for loop, the calculation process can be directly placed under target.

[0051] The method for generating the target HDF5 data file is to use the target table data as the table data in the target HDF5 data file and use the target metadata with empty content as the target HDF5 data file metadata to generate the target HDF5 data file; while the original HDF5 data file is not changed.

[0052] At this point, the configuration of the configuration file is completed. You can use the configuration file to process the HDF5 data file to obtain the target data table, and use the obtained target data table to generate a new HDF5 data file for drawing and analysis. The specific process is as follows: See also Figure 4 , Figure 4 This is a flow chart of the data processing method of the HDF5 data file in the embodiment of this application. The specific steps are: Step 401, obtain the configuration file.

[0053] This configuration file is used to configure the extraction and calculation of data in the HDF5 data file to be processed, and to generate the target HDF5 data file.

[0054] Step 402, based on the entry variable of the HDF5 data file to be processed in the configuration file, the HDF5 data file to be processed is found; wherein the HDF5 data file to be processed is experimental data directly stored in the HDF5 format when performing material experiments.

[0055] The application scenario of the present application may be an experimental data recording scenario, that is, storing the experimental result data in the form of an HDF5 data file to be processed.

[0056] One or more experiments may be conducted for an experiment, and the results of each experiment may be stored in one or more HDF5 data files. Multiple HDF5 data files may be searched in an array manner; and the entry variable of the HDF5 data file to be processed configured in the application embodiment may search for one or more HDF5 data files, which is determined according to actual needs.

[0057] Step 403, processing the source metadata and source table data in the HDF5 data file to be processed accordingly according to the reuse information, data extraction information, inheritance information and data calculation process in the configuration file, and generating the destination table data based on the processed data results.

[0058] For the configured HDF5 data file to be processed, the configured reuse information is used to reuse the relevant parameters, and the configured inheritance information is used to inherit the relevant parameters, so that the configuration file is simplified and the relevant parameters can be quickly located; Extract data corresponding to related objects of related configurations based on the configured data extraction information; According to the configured data calculation process, the specified object is determined, and the numerical value corresponding to the corresponding object is calculated according to the calculation method specified in the data calculation process to obtain the calculation result; The calculation results obtained for each second-class top-level object are respectively combined with the data extracted by extracting information to obtain a set of names and values; All group names and values ​​corresponding to all second-category top-level objects are integrated together to generate destination table data.

[0059] See also Figure 5 , Figure 5 A schematic diagram of the target table data generated in an embodiment of the present application. Figure 5 The content of the target metadata part is empty, and the target table data includes the content corresponding to the dashed box and the solid box on the right; the data in the dashed box in the target table data comes from the source metadata, and the data in the solid box comes from the calculation result; each column in the solid box is an extractable second-class top-level object; the data in each row of dashed boxes corresponds to the data in the implementation box one by one, the data in the dashed box comes from the source metadata of the source table data, and the data in the dashed box comes from the second-class top-level object calculated based on the same source table data. For example, Figure 5 Line 8 in comes from Figure 1 Displays source metadata and selected source table data.

[0060] See also Figure 6 , Figure 6 This is a schematic diagram of configuration information for calculating id_vgg in a configuration file in an embodiment of the present application. Figure 6 The inheritance information corresponding to the second-class top-level object is set in , including: the base object (parent) and the local alias variable (alias); taking the test voltage and current related experiments as an example, id_vgg refers to the current value when vg takes the maximum value, which is also the maximum value of the current; meta_vd: vd_threshold; this alias is used to check the source metadata to determine the voltage condition of Vd during the test; vgg:vg_max, this alias is used to confirm that the value of the vg column is the maximum value of Vg; Threshold (threshold voltage) refers to the voltage value at which the state of an electronic device or circuit changes when the voltage reaches a certain value in an electronic device or circuit; For example, the calculation result id_vgg in the 8th row is 7.59304e-7, which is the calculation result corresponding to the following calculation: find the value of ids corresponding to the row position of column_vgs closest to vg_max, as the value of itarget, where vg_max is set to 3.3.

[0061] In specific applications, the calculation method can be configured according to the actual application, such as finding the average value, variance, etc.

[0062] Step 404: Generate a destination HDF5 data file using the destination table data and empty destination metadata according to the method for generating the destination HDF5 data file in the configuration file, and store the file.

[0063] When this application generates a destination HDF5 data file, the destination table data is used as the table data in the destination HDF5 data file, and the destination metadata with empty content is used as the metadata in the destination HDF5 data file. In other words, the format of the HDF5 data file is not changed, but the destination table data and the destination metadata are used to generate the destination HDF5 data file.

[0064] When the HDF5 data file is needed for this purpose, it can be directly obtained, such as for plotting experimental result data, conducting experimental analysis, etc.

[0065] In the embodiment of the present application, a configuration file is first configured through YML, in which the first type of top-level objects and the second type of top-level objects, as well as a method for generating a target HDF5 data file are defined; based on the configuration file, the HDF5 data file to be processed is found, and the source metadata and source table data in the HDF5 data file are processed to obtain the target table data; and according to the target HDF5 data file generation method, the target table data and the target metadata with empty content are used to generate the target HDF5 data file, and store it; the target HDF5 data file can be used for experimental result data drawing and experimental analysis. This method can realize the processing of data in the HDF5 data file at a low cost by configuring the configuration file.

[0066] In the embodiment of the present application, the experimental result data can be directly plotted using the destination table data, or a destination HDF5 data file can be generated first for storage, and when the experimental result data needs to be plotted, the stored destination HDF5 data file can be directly used for plotting the experimental result data.

[0067] In the embodiment of the present application, the HDF5 data file that directly stores the experimental data is processed by means of a configuration file to obtain a target HDF5 data file. The target HDF5 data file can be compressed and stored, which reduces the storage space and is easy to transmit; and when used, the required data can be retrieved more quickly.

[0068] The experimental result data plotted in the embodiment of the present application may be a curve graph, etc. The specific graph may be determined according to the needs of the specific application scenario. The following application scenario of material experiment is now given: The resistance characteristics of the material are measured, that is, the relationship characteristics between current and voltage, and the existing HDF5 format is used to save the experimental data directly obtained through the experiment, that is, the HDF5 data file to be processed in the embodiment of the present application, wherein the table data (source table data) of the HDF5 data file to be processed is used to store the current and voltage series values, and the metadata (source metadata) of the HDF5 data file to be processed is used to record different experimental conditions, such as temperature, and finally the results of multiple experiments are stored in multiple HDF5 data files to be processed; then the method provided by the present application is used to process the experimental results, that is, the metadata and table data in the HDF5 data file to be processed according to the configuration file in the configuration file, to generate the target table data, and use the target table data instead of the source table data and the source source data to use the target HDF5 data file, and finally, based on the target HDF5 data file, a curve of resistance change at different temperatures can be drawn.

[0069] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0070] Based on the same inventive concept, the present application also provides a data processing device for HDF5 data files. Figure 7 , Figure 7 This is a schematic diagram of the structure of a data processing device for HDF5 data files in an embodiment of the present application. The device includes: The configuration unit 701 is used to configure the configuration file based on YML; wherein the configuration file defines the first type of top-level objects and the second type of top-level objects, and a method for generating a target HDF5 data file; defines multiplexing information and an entry variable of the HDF5 data file to be processed for the first type of top-level objects; configures the attribute information of the second type of top-level objects, the attribute information including: data extraction information, inheritance information and data calculation process, An acquisition unit 702 is used to acquire a configuration file; A search unit 703 is used to search for the HDF5 data file to be processed based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein the HDF5 data file to be processed is the experimental data directly stored in the HDF5 format when performing material experiments; The processing unit 704 is used to process the source metadata and source table data in the HDF5 data file to be processed according to the reuse information in the configuration file, as well as the data extraction information, inheritance information and data calculation process, and generate the destination table data based on the processed data results; generate the destination HDF5 data file using the destination table data and metadata with empty content according to the generation method of the destination HDF5 data file in the configuration file, and store it; the destination HDF5 data file is used for plotting the experimental result data and performing experimental analysis.

[0071] In another example, the reuse information includes: global variables and global alias variables; Among them, global variables are used to define variable values ​​used in objects other than global variables in YML; Global alias variables are used to define aliases for long strings in reused variables.

[0072] In another example, data extraction information includes: Whether the second-class top-level object can be extracted and whether it meets the extraction conditions; Among them, whether the second type of top-level object meets the extraction conditions is defined in an array manner: cc in the array indicates whether the source table data contains the specified column, and cm is used to check the source metadata, where cm has two parameters, the first parameter is the metadata name, and the second parameter is the value to be matched; cc and cm are built-in functions of the configuration file handler.

[0073] In another example, inheritance information includes: base objects and local alias variables; The base object is used to specify all second-class top-level objects that will inherit the base object; Local alias variables, used to define aliases for long strings in inherited variables; If the global alias variable and the local alias variable have the same name, the local alias variable is used.

[0074] In another example, a data calculation process includes: Obtain the value of the variable to be calculated by extracting information; Calculate according to the defined calculation method; The calculation result is used as the calculation result of the second-class top-level object.

[0075] In another example, If there are multiple calculation steps in the defined calculation method, the calculation result of the last calculation step is used as the calculation result of the second-class top-level object.

[0076] In another example, Processing unit 704 is specifically used to generate target table data based on the processed data results, and obtain a set of names and values ​​by respectively combining the calculation results obtained for each second-category top-level object with the data extracted by extracting information; and integrate all group names and values ​​corresponding to all second-category top-level objects to generate target table data.

[0077] The units in the above-mentioned embodiments may be integrated into one body or deployed separately; they may be combined into one unit or further divided into multiple sub-units.

[0078] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the data processing method of the HDF5 data file when executing the program.

[0079] In another embodiment, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, a data processing method for an HDF5 data file is implemented.

[0080] Figure 8 Schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. Figure 8 As shown, the electronic device may include: a processor (Processor) 810, a communication interface (Communications Interface) 820, a memory (Memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute the following method: The configuration file is configured based on YML; wherein the configuration file defines a first-class top-level object and a second-class top-level object, and a method for generating a target HDF5 data file; for the first-class top-level object, multiplexing information and an entry variable of the HDF5 data file to be processed are defined; attribute information of the second-class top-level object is configured, and the attribute information includes: data extraction information, inheritance information and data calculation process. The method includes: Get the configuration file; Find the HDF5 data file to be processed based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein the HDF5 data file to be processed is the experimental data directly stored in the HDF5 format when conducting material experiments; According to the reuse information in the configuration file, as well as the data extraction information, inheritance information and data calculation process, the source metadata and source table data in the HDF5 data file to be processed are processed accordingly, and the destination table data is generated based on the processed data results; The target HDF5 data file is generated and stored using the target table data and metadata with empty content according to the target HDF5 data file generation method in the configuration file; the target HDF5 data file is used for experimental result data drawing and experimental analysis.

[0081] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0082] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0084] The flow chart and block diagram in the accompanying drawings of the present application show the possible architecture, function and operation of the system, method and computer program product according to the various embodiments disclosed in the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in the order of the standards in different figures. For example, the boxes represented by two connections can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0085] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims disclosed in this application may be combined and / or combined in a variety of ways, even if such combinations and / or combinations are not explicitly described in this application. In particular, without departing from the spirit and teachings of this application, the features described in the various embodiments and / or claims of this application may be combined and / or combined in a variety of ways, and all of these combinations and / or combinations fall within the scope disclosed in this application.

[0086] Specific embodiments are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core ideas, and is not used to limit the present application. For those skilled in the art, changes can be made in the specific implementation methods and application scopes according to the ideas, spirits and principles of the present invention, and any modifications, equivalent substitutions, improvements, etc. made therein should be included in the scope of protection of this application.

Claims

1. A data processing method for HDF5 data files, characterized in that: A configuration file is configured based on YML; wherein the configuration file defines a first type of top-level object and a second type of top-level object, and a method for generating a target HDF5 data file; multiplexing information and an entry variable of a to-be-processed HDF5 data file are defined for the first type of top-level object; attribute information of the second type of top-level object is configured, the attribute information including: data extraction information, inheritance information and data calculation process, and the method includes: Obtaining the configuration file; Finding the HDF5 data file to be processed based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein the HDF5 data file to be processed is experimental data directly stored in the HDF5 format when conducting material experiments; Processing the source metadata and source table data in the HDF5 data file to be processed accordingly according to the reuse information in the configuration file, the data extraction information, the inheritance information and the data calculation process, and generating the destination table data based on the processed data results; The target HDF5 data file is generated and stored using the target table data and the target metadata with empty content according to the target HDF5 data file generation method in the configuration file; the target HDF5 data file is used for experimental data drawing and experimental analysis.

2. The method according to claim 1, characterized in that The reuse information includes: global variables and global alias variables; The global variables are used to define variable values ​​used in objects other than the global variables in YML; The global alias variable is used to define an alias for a long string in a reused variable.

3. The method according to claim 1, characterized in that The data extraction information includes: Whether the second type of top-level object can be extracted and whether it meets the extraction conditions; Among them, whether the second type of top-level object meets the extraction conditions is defined in an array manner: cc in the array indicates whether the source table data contains the specified column, and cm is used to check the source metadata, where cm has two parameters, the first parameter is the metadata name, and the second parameter is the value to be matched; cc and cm are built-in functions of the configuration file handler.

4. The method according to claim 1, characterized in that The inheritance information includes: basic objects and local alias variables; The base object is used to specify all second-class top-level objects that will inherit the base object; The local alias variable is used to define an alias for a long string in an inherited variable; If the global alias variable and the local alias variable have the same name, the local alias variable will be used.

5. The method according to claim 1, characterized in that The data calculation process includes: Obtaining the value of the variable to be calculated through the extracted information; Calculate according to the defined calculation method; The calculation result is used as the calculation result of the second-type top-level object.

6. The method according to claim 5, characterized in that The method further comprises: If there are multiple calculation steps in the defined calculation method, the calculation result of the last calculation step is used as the calculation result of the second-type top-level object.

7. The method according to claim 1, characterized in that The generating of the target table data based on the processed data result includes: The calculation results obtained for each of the second-class top-level objects are respectively combined with the data extracted by the extraction information to obtain a set of names and values; All group names and values ​​corresponding to all the second-category top-level objects are integrated together to generate destination table data.

8. A data processing device for HDF5 data files, characterized in that: The device comprises: A configuration unit is used to configure a configuration file based on YML; wherein the configuration file defines a first type of top-level object and a second type of top-level object, and a method for generating a target HDF5 data file; defines multiplexing information and an entry variable of a to-be-processed HDF5 data file for the first type of top-level object; configures attribute information of the second type of top-level object, wherein the attribute information includes: data extraction information, inheritance information, and data calculation process, An acquisition unit, used to acquire the configuration file; A search unit, used for searching the HDF5 data file to be processed based on the entry variable of the HDF5 data file to be processed in the configuration file; wherein the HDF5 data file to be processed is experimental data directly stored in the HDF5 format when performing material experiments; A processing unit is used to process the source metadata and source table data in the HDF5 data file to be processed according to the reuse information in the configuration file, the data extraction information, the inheritance information and the data calculation process, and generate the destination table data based on the processed data results; generate the destination HDF5 data file using the destination table data and the destination metadata with empty content according to the generation method of the destination HDF5 data file in the configuration file, and store it; the destination HDF5 data file is used for experimental data drawing and experimental analysis.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method described in any one of claims 1 to 7 is implemented.

Images